LatSouthBot

LatSouthBot reads company homepages, once, politely.

The LatSouth crawler that identifies which technologies a company's public homepage uses. What it does, what it never does, and how to opt out.

LatSouthBot

What the crawler does

User-agent: LatSouthBot/1.0 (+https://latsouth.com/bot/; hello@latsouth.com). robots.txt token: LatSouthBot.

  • It visits only the homepage of a company website that is already listed for that company in the official registry, and reads /robots.txt first. If robots.txt disallows LatSouthBot (or all user-agents), the homepage is not requested. If robots.txt cannot be read because the server errors or times out, the site is skipped.
  • One request for robots.txt and one for the homepage, at least a second apart, never more than one at a time per host and three per server IP; redirects to another host are followed only after that host's robots.txt.
  • It does not log in, fill forms, follow links beyond the homepage, run JavaScript, keep cookies or try to get past a bot wall; a 403, 429 or challenge page is recorded as blocked and left alone.
  • It keeps the technology names it recognised and how (a header, a script URL, a meta tag), never the page text, and no personal data.

To opt out, add User-agent: LatSouthBot / Disallow: / to your robots.txt, or write to hello@latsouth.com with the domain: it is removed from the next scan and from the published files.