Fetch and parse a site's robots.txt, including whether GPTBot, ClaudeBot, and PerplexityBot are allowed in.
Fetches /robots.txt, lists every rule group, flags syntax problems, and checks whether major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot) are allowed or blocked.