Technology · Culture · Change
Sign inJoin us
TechTrends NowFollow the next shiftTechnology · Culture · Change
← Latest news

From Dev.to

User-agent: * doesn't apply to crawlers you named in robots.txt

A Disallow under User-agent: * does nothing for a crawler that has its own group in robots.txt. Each crawler reads exactly one group, the one matching its name, and * is only the fallback for crawlers...

Read the full report at Dev.to ↗