From Dev.to
Should you block AI crawlers (GPTBot, ClaudeBot, CCBot) in robots.txt? Let's argue about it
Automated technical SEO checks that catch indexing blocks and deploy regressions — seotto.com A couple of years ago, robots.txt was mostly about search engines. Now a typical site's logs include a who...