Crawler
The Searchscope crawler
SearchscopeBot reads websites that their owners have connected to Searchscope. This page explains what it does, how to recognize it, and how to allow or block it.
What it does
SearchscopeBot visits a website when the site owner connects it to Searchscope. It fetches the public HTML pages, follows internal links and reads the sitemap. It records titles, headings, links, images, structured data (schema.org) and the page text, so the analysis can run. It does not log in, submit forms or fetch anything behind a password.
How to recognize it
- Name
- SearchscopeBot
- User agent
- SearchscopeBot/1.0 (+https://searchscope.io/bot)
- IP address
- 161.35.167.82
Every request comes from that one IP address.
How it behaves
- It checks robots.txt before it fetches a URL. It follows the rules for its user agent, including a Crawl-delay line if you set one.
- It makes at most 6 requests at a time, with a 0.3 second pause between requests to the same site. A customer can slow it down further in their settings.
- A crawl reads between about 500 and 10,000 pages, depending on the customer's plan.
- After a crawl it checks up to 100 outbound links from the site to see that they still work. It sends a HEAD request for each one, which asks for the headers only, not the page.
Allow it
If a firewall or bot protection blocks unknown crawlers, allow the IP address 161.35.167.82 or the user agent string above. This applies to Cloudflare, Sucuri, Wordfence and similar tools.
Block it
Add these two lines to your robots.txt:
User-agent: SearchscopeBot
Disallow: /Or block the IP address 161.35.167.82.
Blocking the crawler stops the analysis for that site. The site owner then sees no findings from Searchscope. If someone else set up Searchscope for the site, check with them before you block it.
Content Studio
When a customer uses Content Studio, the same user agent fetches single public pages they point it at: their own page, or a competitor's page they are studying. That is one request per page, not a crawl.
Questions
Questions about the crawler? Contact Growth Minded Marketing, the team that builds and runs Searchscope.