LatSouthBot
LatSouthBot reads company homepages, once, politely.
The LatSouth crawler that identifies which technologies a company's public homepage uses. What it does, what it never does, and how to opt out.
LatSouthBot
What the crawler does
User-agent: LatSouthBot/1.0 (+https://latsouth.com/bot/; hello@latsouth.com). robots.txt token: LatSouthBot.
- It visits only the homepage of a company website that is already listed for that company in the official registry, and reads
/robots.txtfirst. If robots.txt disallows LatSouthBot (or all user-agents), the homepage is not requested. If robots.txt cannot be read because the server errors or times out, the site is skipped. - One request for robots.txt and one for the homepage, at least a second apart, never more than one at a time per host and three per server IP; redirects to another host are followed only after that host's robots.txt.
- It does not log in, fill forms, follow links beyond the homepage, run JavaScript, keep cookies or try to get past a bot wall; a 403, 429 or challenge page is recorded as blocked and left alone.
- It keeps the technology names it recognised and how (a header, a script URL, a meta tag), never the page text, and no personal data.
To opt out, add User-agent: LatSouthBot / Disallow: / to your robots.txt, or write to hello@latsouth.com with the domain: it is removed from the next scan and from the published files.