What it is
ScoutBot reads the public pages of Australian business websites to note which booking and job management software a business shows on its site (for example a booking link to its job management system), together with the business details the site itself publishes: business name, ABN, licence number, business phone number, suburb and postcode.
Its User-Agent is ScoutBot/<version> (+https://illetro.com/scoutbot).
What it does not collect
ScoutBot does not collect email addresses, personal social media profiles, street addresses or personal details. Mobile phone numbers are not stored; only the fact that a site publishes one is noted. It never logs in, fills in or submits forms, solves challenges or tries to get past a block.
How it behaves
- It reads your
robots.txtbefore anything else and follows it, using the product tokenScoutBot. - It sends one request at a time to a site, at least 3 seconds apart, or further apart if your
Crawl-delayasks for it. - It reads at most a handful of pages per site, usually the home, contact, about and booking pages.
- It slows down on HTTP 429 or 503, honours
Retry-After, and stops visiting a site that blocks it. - If your robots.txt disallows Anthropic's agents or sets
Content-Signal: ai-input=no, your pages are never passed to an AI model, and your site is never the subject of an AI web search.
How to block it
Add this to your site's robots.txt:
User-agent: ScoutBot Disallow: /
Removal requests
To have everything ScoutBot holds about your business deleted, and to keep it from being collected again, email [email protected] with your business website or ABN. Requests are actioned within 30 days.