Apple’s New Web Crawler: AppleBot: What You Need to Know

Apple’s New Web Crawler: AppleBot, What You Need to Know

May 25, 2015 by Tim Dugan

The existence of an Apple web crawler has been speculated for months now. Apple has finally confirmed these rumors with the creation of a new support page for the web crawler, Applebot. We’ve put together the basics on what you need to know about Applebot.

What is Applebot?

According to Apple, Applebot is “used for products including Siri and Spotlight Suggestions.” Prior to the creation of Applebot, Apple would have needed to rely on Google or Bing powered web crawlers to index the web and provide results for Spotlight or Siri search queries. Apple states that Applebot “respects customary robots.txt and robots meta tags.” It functions in the same way as Googlebot and if a website lacks Applebot instructions, Applebot will follow the instructions for Googlebot.

Future Uses of Applebot?

Although Bing has been attracting people, Google still dominates the majority of the search engine market for computer based searching. When it comes to mobile search engine usage however, there is not a clear forerunner. With the creation of Applebot, Apple seems to be clearly exploring the possibility of competing directly with Google for search engine users.

Apple’s success with the iPhone, the company’s bestselling product, could help Apple leverage its success in the mobile market and potentially become a leader in the mobile search market. Applebot is very likely a foundational piece of a future Apple-owned search engine hoping to pull a large portion of mobile and traditional search traffic away from Google.

What Should Web Developers & SEOs Do?

Since Applebot will be following any instructions you’ve created for Googlebot, you don’t necessarily need to do anything right now. For example, if you have blocked Googlebot, Applebot will be blocked because it is following Googlebot instructions. That may change in the future. If you do block Googlebot, it couldn’t hurt to take the extra step to block Applebot as well.

As of right now, the search results from Applebot do not seem to vary from Googlebot that much. It is important to realize that as Apple continues to expand the functionality of Applebot, it will likely present distinguishing factors to attract users away from Google. Those distinguishing factors may very well require several SEO adjustments to maintain your search traffic and online visibility.