AI Crawler Access Study: robots.txt and llms.txt of 261 Major Websites (Oct 2026)
Description
This dataset records whether 285 major websites allow or block 15 user agents in robots.txt (14 AI crawlers plus Googlebot as a control), and whether each site serves a valid llms.txt. Data was collected in October 2026 by AI Ranked. One row per domain. 261 of the 285 robots.txt files could be read; the other 24 are kept in the file with fetched_ok = no. Headline results (from the report): of the 261 readable sites, 91 (35%) block at least one AI training crawler and 56 (21%) block at least one AI search crawler. CCBot and Bytespider are blocked most (75 sites each, 28.7%) and OAI-SearchBot least (30 sites, 11.5%). 81 of 241 sites with a definite answer (34%) serve a valid llms.txt. Categories: B2B SaaS, News & media, Reference & education, E-commerce & marketplaces, Tech & developer, Social & community, Government & public sector, Search & portals, Video & streaming, Travel & local. Method: HTTPS fetch of /robots.txt and /llms.txt for each domain. robots.txt was parsed per RFC 9309 (groups merged, fallback to the * group, longest match wins, Allow wins ties, * and $ wildcards) and each user agent was evaluated against the path /. Limitations: a single October 2026 snapshot, homepage path only, and a hand-built list of well-known sites rather than a random sample of the web. Full report and method: https://airanked.ai/research/ai-crawler-access-report Source CSV: https://airanked.ai/research/ai-crawler-access-data.csv
Files
Steps to reproduce
For each domain, fetch https://DOMAIN/robots.txt and https://DOMAIN/llms.txt over HTTPS. Parse robots.txt per RFC 9309 (groups merged, fallback to the * group, longest match wins, Allow wins ties, * and $ wildcards) and evaluate each of the 15 user agents against the path /. Record allowed or blocked per agent, plus whether a valid llms.txt is served. Sites whose robots.txt could not be read are kept with fetched_ok = no. Full method: https://airanked.ai/research/ai-crawler-access-report