On May 6, 2015, Apple confirmed a new way which specific Apple products will provide search functionality via a unique crawler called Applebot. Applebot has been confirmed to work with Spotlight Suggestions and Siri, ultimately forgoing the use of Google or Bing to return search queries to users. While Applebot is crawling and indexing web content, Bing remains its search partner of choice since Apple’s iOS7 update in 2013, as well as Wolfram Alpha for Siri answers.
Apple has provided a help document, both confirming the new crawler and providing basic functionality overview. They have confirmed that all robots.txt directives will be followed in the normal Robots Exclusion Protocol. They advise webmasters that the only changes they may see is in referrers, where “Applebot” will be appended to a traditional referrer. For example, a Mozilla referrer coming from Applebot would look like:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
What does this mean?
As of now, the Applebot traffic will be relatively small as Apple grows into search. Other evidence points to a greater shift in the way Apple handles search on its devices; the company has acquired social analytics company Topsy in December 2013, enfolding Bing in Siri and Spotlight as the default search engine, only to drop Bing from Siri and Spotlight in mid-2014. Sources like Apple insider are pointing to “Apple's rapidly-expanding internal search group” looking to take search engine giant Google head-on using the very Topsy team acquired in 2013.
Where Apple’s moxie comes from is simple: this past January, Apple reveled in their quarterly earnings of $74.5 million in iPhones, a 41% increase year-over-year. That is a lot of devices powered by Apple search. The Apple Watch was estimated to have around 2.3 million units on pre-order. Even though iPad sales decreased 17.71% in Q1 2015, Apple still sold 21.42 million devices powered by Apple search engine as default.
Apple’s move to power Siri and Spotlight with Applebot before their default browser (Safari) signals that Apple is predicting more reliance on voice searches, especially in light of the Apple Watch launch. As Siri is more integrated in Apple products along with Search Spotlight, Apple will try to keep end users within the Apple digital ecosystem – similar to how Google Now and knowledge graph reference Google products before external ones. For search, Applebot crawling sites would give Apple additional content to keep building Siri up as more relevant tool for diehard Apple fans.
While Applebot could be a small piece for search marketing professionals today, it signals a shift in the way Apple treats search – no longer as a necessary evil of UX, but an opportunity to push semantic search beyond its competitors.
SEO actions to take:
Since Applebot follows directives for disallows in robots.txt rules and robots meta tags, current robots instructions don't need to be adjusted immediately. AppleBot will follow Googlebot instructions, which means there is no immediate impact to websites - unless webmasters want to block AppleBot from crawling specific content.
iProspect recommends webmasters update existing robots.txt files through the regularly scheduled site maintenance processes. Include AppleBot disallows in robots.txt while help to solve for any future instruction changes from Apple.
We expect to see continued diversification of the search landscape not only across different search engines, but also into non-traditional “search” environments including Amazon, eBay, Yelp, and other specialized and niche properties. Driven by high consumer expectations and empowered by the ever-increasing palette of technologies available, search performance will continue to evolve and deliver more relevant results.