Crawler A crawler (bot) is used by search engines to access, download and archive your website's data to be included in its index.