Blog

What US news outlets do with AI bots: we analyzed 86 robots.txt files

By Esteve Castells

What US news outlets do with AI bots: 86 robots.txt files analyzed
Table of contents
  1. Key findings
  2. Which bots come knocking
  3. The four strategies of US news outlets
  4. Publisher by publisher
  5. The back door: sponsored content gets in
  6. What about images?
  7. Blocking doesn't take an outlet out of AI answers
  8. Lawsuits: where the real fight is
  9. Beyond robots.txt
  10. What publishers can take from this
  11. Methodology
  12. About LLM Pulse
  13. Frequently asked questions

In the US, a major publisher's robots.txt is no longer a technical file someone wrote ten years ago. It reads almost like a public record of who the publisher has signed with and who it is suing. Outlets with an OpenAI deal let OpenAI in and keep everyone else out; outlets in court block nearly everything.

On November 16, 2026, we reviewed the robots.txt files of 86 US news outlets, from The New York Times and The Wall Street Journal to TV networks, local papers and tech sites, and checked how each one treats 15 bots from the major AI companies. We then compared the results with the known licensing deals and lawsuits, and with our data on which outlets AI assistants actually cite in the US.

Key findings

  • 90% of US news outlets block at least one bot from the major AI companies. That is 77 of the 86. Only 9 (10%) block none of them.
  • The most common strategy is picking partners. Half of the outlets let some AI companies in and keep others out, often in line with their licensing deals.
  • The most blocked bots are the bulk collectors. CCBot, from Common Crawl, and Bytespider, from ByteDance, are blocked by 83% of outlets; ClaudeBot by 76%.
  • ChatGPT's search crawler gets in most often. Only 36% block OAI-SearchBot, against 58% that block GPTBot, the same company's training bot.
  • Gemini is mostly shut out. 63% block Google-Extended. None block Googlebot, which Google's AI Overviews and AI Mode rely on.
  • Sponsored content still gets in. 28 of the 77 outlets that block AI bots leave specific sections open, and at least 16 do it for their sponsored content.
  • Blocking doesn't take an outlet out of AI answers. The New York Times blocks 13 of the 15 bots and is still among the ten outlets AI cites most in the US.

Which bots come knocking

Each AI company runs several bots with different jobs, and the difference matters:

  • Training bots (GPTBot, ClaudeBot, meta-externalagent, Applebot-Extended, Bytespider and Common Crawl's CCBot) collect text to train future models.
  • Search bots (OAI-SearchBot, Claude-SearchBot and PerplexityBot) build the index assistants draw on to answer questions and cite sources.
  • User bots (ChatGPT-User, Claude-User, Perplexity-User and MistralAI-User) visit a page when someone asks for it in a chat. OpenAI warns in its bot documentation that robots.txt rules may not apply to ChatGPT-User, and Perplexity says in its own that Perplexity-User "generally ignores" them.
  • Google-Extended is a token, not a crawler: it decides whether Gemini can train on a site's content and use it in answers. According to Google's documentation, it does not affect Search, so it does not affect AI Overviews or AI Mode either.
Horizontal bar chart: share of the 86 US news outlets analyzed whose robots.txt blocks each AI bot, November 16, 2026. CCBot and Bytespider lead at 83%; OAI-SearchBot is the least blocked at 36%.
Bot Company What it does Outlets that block it
CCBot Common Crawl Open archive used for training 83%
Bytespider ByteDance Training 83%
ClaudeBot Anthropic Training 76%
Applebot-Extended Apple Training 71%
PerplexityBot Perplexity Search index 69%
Google-Extended Google Gemini training and answers 63%
meta-externalagent Meta Training 62%
GPTBot OpenAI Training 58%
Amazonbot Amazon Training and Alexa 57%
Perplexity-User Perplexity Visits on a user's request 53%
Claude-SearchBot Anthropic Search index 50%
Claude-User Anthropic Visits on a user's request 50%
ChatGPT-User OpenAI Visits on a user's request 48%
MistralAI-User Mistral Visits on a user's request 41%
OAI-SearchBot OpenAI ChatGPT search index 36%
Googlebot and Bingbot Google and Microsoft Search engines (and their AI features) 0%

The files also show how well they are kept up to date. Nineteen outlets block GPTBot but not OAI-SearchBot, and 22 block ClaudeBot but not Claude-SearchBot. Sometimes that is a deliberate choice to separate training from search; sometimes nobody has added the newer bots. Six outlets, including Esquire, Cosmopolitan, ABC News and ESPN, still block only anthropic-ai, an old Anthropic name the company itself lists as deprecated, and leave ClaudeBot alone. And NPR writes "PerplexityUser" without a hyphen, so its block never reaches Perplexity-User.

The four strategies of US news outlets

Bar chart: the 86 US news outlets analyzed split into four strategies toward AI. Picks partners, 50%; shuts out the assistants, 22%; blocks training and allows search, 17%; open doors, 10%.
Strategy What it does Outlets Examples
Picks partners Lets some AI companies in and keeps others out 43 (50%) The Wall Street Journal, The Washington Post, The Atlantic, AP, Forbes
Shuts out the assistants Blocks the search and user bots of OpenAI, Anthropic and Perplexity 19 (22%) The New York Times, USA Today, Politico, Chicago Tribune, CNN, CNET
Blocks training, allows search Bars the bots that train models and lets in those that search and cite 15 (17%) Los Angeles Times, The Boston Globe, Business Insider, Axios
Open doors Blocks none of the 15 bots 9 (10%) Fox News, TIME, Slate, ProPublica, The Texas Tribune

Publisher by publisher

In the US, robots.txt has turned into a commercial document: if you want to know who a publisher has signed with, look at who it lets in. What surprised us is how little blocking keeps an outlet out of the answers. Outlets that block almost everything are still among the ones AI cites most.

Head-and-shoulders portrait of a smiling man in a dark zip-up sweater against a plain gray background.Esteve Castells Co-founder, LLM Pulse
Dot matrix: what the robots.txt of 21 major US news outlets lets each AI company read. The Wall Street Journal, The New Yorker, People and The Atlantic only let OpenAI in; USA Today and the Chicago Tribune block every company; Fox News and TIME block none.
Publisher What its robots.txt does Context
The New York Times Blocks 13 of the 15 bots, Gemini included. Lets Amazonbot in (except on Wirecutter) and does not mention Mistral's bot Deal with Amazon (May 2025); lawsuits against OpenAI and Microsoft (2023) and against Perplexity (2025)
News Corp (WSJ, Barron's, MarketWatch) Allowlist: everything closed by default and, among AI bots, only OpenAI's three get in Deal with OpenAI (May 2024); Dow Jones and New York Post lawsuit against Perplexity (2024)
Condé Nast (The New Yorker, Vogue, Wired, Vanity Fair) Only lets OpenAI in Deal with OpenAI (August 2024)
People Inc. (People, Investopedia, Allrecipes) Lets OpenAI in, except for images; shuts out Gemini, Anthropic, Perplexity and Meta except on sponsored content Deals with OpenAI (2024), Microsoft (2025) and Meta (2025)
Vox Media (Vox, The Verge, New York Magazine) Allows GPTBot, OAI-SearchBot and Amazonbot, blocks ChatGPT-User and shuts out the rest except sponsored content and The Strategist Deal with OpenAI (May 2024)
The Atlantic Lets OpenAI in, shuts out the rest except /sponsored/ and states its preferences with Content Signals Deal with OpenAI (May 2024)
The Washington Post Lets OpenAI and Gemini in; blocks ClaudeBot, PerplexityBot, Meta and Amazon, and opens its branded content studio to almost all of them Deal with OpenAI (April 2025)
Hearst (Esquire, Cosmopolitan, Good Housekeeping) Lets GPTBot into its US content but not its international editions; shuts out Gemini, Amazon, ByteDance and Common Crawl Deal with OpenAI (October 2024)
Associated Press Blocks GPTBot but lets in ChatGPT's search crawler and Gemini; shuts out Anthropic and Perplexity Deals with OpenAI (2023) and Google (January 2025)
Reuters Allowlist: of OpenAI's bots, only the search and user bots get in Deal with Meta (October 2024)
Politico (Axel Springer) Blocks all 15 bots, OpenAI included, except on its sponsored content Axel Springer deal with OpenAI (December 2023)
Axios Blocks only CCBot, Bytespider and Amazonbot, and explicitly allows Google-Extended Deal with OpenAI (January 2025)
Star Tribune Allowlist: only search engines, social networks, ad bots and Google-Extended get in -
McClatchy (Miami Herald, The Kansas City Star) Bars AI bots from article pages; the Miami Herald opens its sponsored articles to them -
CNBC A single group for AI and trading bots that only lets them read /select/, its shopping section -
NPR Blocks 12 of the 15 bots -
USA Today Co. Blocks all 15 bots; opens its sponsored and shopping stories Deals with Perplexity (2025) and Meta (2025); lawsuit against OpenAI (October 2026)
Alden (Chicago Tribune, New York Daily News, Denver Post) Blocks all 15 bots and adds a text and data mining reservation (TDMRep) Lawsuits against OpenAI and Microsoft (2024) and, by the Chicago Tribune, against Perplexity (2025)
CNN Blocks everything except Gemini and Meta Deal with Meta (December 2025); lawsuit against Perplexity (May 2026)
Ziff Davis (CNET, PCMag, ZDNET, Mashable) Blocks 14 of the 15 bots but opens sponsored articles one by one Lawsuit against OpenAI (April 2025)
Penske Media (Variety, Rolling Stone, The Hollywood Reporter) Shuts out OpenAI, Gemini, ClaudeBot and PerplexityBot; opens its shopping sections Lawsuit against Google over AI Overviews, dismissed in September 2026
Forbes Blocks GPTBot, ClaudeBot and PerplexityBot; explicitly allows OpenAI's and Anthropic's search crawlers Threatened Perplexity with legal action over plagiarism (June 2024)
Fox News, TIME, Slate, ProPublica Block no bots Fox News has a deal with Meta; TIME with OpenAI and Perplexity

Sources for the deals and lawsuits are in Press Gazette's tracker and our map of OpenAI's publisher deals.

OpenAI's partners: open to one, closed to the rest

It is the most common pattern: News Corp, Condé Nast, People Inc. and The Atlantic all signed with OpenAI and let only OpenAI in. News Corp goes furthest, with an allowlist. The Wall Street Journal, Barron's and MarketWatch open their robots.txt with a "Disallow: /" for everyone, then list who may enter: the traditional search engines, some advertising tools and, among AI bots, only GPTBot, OAI-SearchBot and ChatGPT-User. Reuters uses the same system but admits only OpenAI's search and user bots, not its training bot. The Star Tribune also runs an allowlist, and the only AI bot on it is Google-Extended.

Within that pattern, there are variations. Vox Media lets GPTBot train and blocks ChatGPT-User. Hearst opens the door to OpenAI on its US content and closes it on its editions in other countries, including the Spanish one (/es/). The Washington Post lets in Gemini as well as OpenAI. And Politico goes against its own group: Axel Springer signed with OpenAI in December 2023, yet Politico blocks all three OpenAI bots and only opens its sponsored content to them, while Business Insider, from the same group, does not block OpenAI.

The litigants: block everything

The New York Times, the Alden papers and USA Today Co., all suing AI companies, block all or nearly all of the 15 bots. What the Times leaves open is telling: Amazonbot can go everywhere except Wirecutter, an exception that lines up with its May 2025 licensing deal with Amazon.

Perplexity gets singled out

PerplexityBot is blocked by 69% of outlets, more than any other search bot. CNN, which had been in talks with Perplexity, sued it in May 2026; according to NPR, CNN alleges Perplexity ignored its robots.txt block. USA Today Co. signed with Perplexity in 2025 and still blocks both of its bots: the licensed content reaches Perplexity through other channels.

The back door: sponsored content gets in

As in Spain, many outlets that block AI bots site-wide add an explicit rule opening the sections where they publish branded content and shopping articles.

Table: US news outlets that block AI bots across their site but let them read their sponsored and shopping sections: Vox Media, People Inc., The Atlantic, The Washington Post, USA Today, Politico, Miami Herald, U.S. News, CNBC and Ziff Davis.

Of the 77 outlets that block at least one AI bot, 28 leave specific paths open, and at least 16 do it for sponsored content: /sp/ at Vox Media, */presented/ and */integrated/ at People Inc., /sponsored/ at The Atlantic and Politico, /paid/sponsored-articles/ at the Miami Herald, /creativegroup/ at The Washington Post, /branded-story/ and /sponsor-story/ at USA Today, /brandfuse/ at U.S. News. Ziff Davis takes the idea furthest: on Mashable, PCMag, ZDNET and CNET it allows specific articles one by one, many of them sponsored, such as the Samsung pieces Mashable publishes under the /ad/ path. Others open only their shopping sections, like Penske Media, NBC News and CNBC with Select, or NJ.com, which also opens its betting and casino pages.

There is another back door, built for one specific partner: custom feeds. Ars Technica, from Condé Nast, shuts Amazonbot out of the whole site and gives it its own feed at /feed/amazon-rss. BuzzFeed does the same with a feed called "amazon-ai-grounding".

What about images?

People, Investopedia and Allrecipes let OpenAI's bots read their articles but bar them from the /thmb/ path, where they serve their photos. On top of that, 31 of the 86 outlets block ImagesiftBot, a crawler that specializes in images, and several add img2dataset, a tool for building image datasets to train models.

Blocking doesn't take an outlet out of AI answers

We took the ten US outlets AI cites most, from our study of the most-cited US news outlets, and checked their robots.txt. The value is the share of answers generated for the US that cite the outlet at least once, across ChatGPT, Perplexity, Gemini, Google's AI Mode and AI Overviews, in a 7-day window updated on November 15, 2026.

Outlet OpenAI bots Gemini Perplexity bots Answers that cite it
Forbes Only training blocked Open Partial 1.34%
TechRadar Open Open Open 1.26%
U.S. News Blocked Blocked Blocked 1.04%
Yahoo Partial Blocked Partial 1.01%
Tom's Guide Open Open Open 0.81%
PCMag Blocked Open Blocked 0.77%
The New York Times Blocked Blocked Blocked 0.41%
Digital Trends Only training blocked Blocked Open 0.38%
Good Housekeeping Open Blocked Open 0.37%
CNET Blocked Open Blocked 0.37%

Four of the ten (U.S. News, PCMag, The New York Times and CNET) block every OpenAI and Perplexity bot and still made the list. Part of the explanation is that the ranking includes Google's AI Overviews and AI Mode, which rely on Googlebot, and nobody blocks Googlebot. The other part is that robots.txt is a request, not a barrier: in December 2025, while dismissing part of Ziff Davis's lawsuit against OpenAI, a federal judge in New York compared it to a "keep off the grass" sign, according to Eric Goldman's analysis of the case.

The other side shows up too: TechRadar and Tom's Guide, which keep the door open to OpenAI, Google and Perplexity, rank second and fifth. The sample consists of questions brands choose to track and leans toward commercial topics, which helps explain why tech and shopping outlets rank so high. In July we published a broader analysis of the US news outlets AI cites most, based on 5.3 million citations.

Lawsuits: where the real fight is

In the US, litigation is much further along than in Europe:

  • The New York Times sued OpenAI and Microsoft in December 2023 and Perplexity in December 2025.
  • Eight Alden papers, including the New York Daily News and the Chicago Tribune, sued OpenAI and Microsoft in April 2024.
  • Dow Jones and the New York Post, both owned by News Corp, sued Perplexity in October 2024, the same year they signed with OpenAI.
  • Ziff Davis sued OpenAI in April 2025.
  • CNN sued Perplexity in May 2026.
  • USA Today Co. sued OpenAI in October 2026, seeking more than $250 million.
  • Penske Media lost the first round of its lawsuit against Google over AI Overviews: it was dismissed on September 30, 2026, according to Press Gazette, because the judge saw no formal bargain obliging Google to send traffic in exchange for the content.

Beyond robots.txt

US outlets use more tools than Spanish ones to spell out their preferences, though none is widespread:

  • Content Signals. The syntax Cloudflare proposed in September 2025 to separate three uses: search, AI answers and training (Cloudflare's explanation). The Atlantic uses it: "search=yes, ai-input=no, ai-train=no".
  • TDMRep. A protocol for reserving text and data mining rights. The four Alden papers we analyzed publish it.
  • RSL (Really Simple Licensing). A standard for stating, in robots.txt, the licensing terms for AI use of the content. The Seattle Times applies it.
  • llms.txt. A file that summarizes a site for language models. Only Fox News, Business Insider and Good Housekeeping publish one.

The next step is planned for 2027. On October 30, 2026, Google announced on its developer blog that it aims to support, along with other AI companies, a common robots.txt signal for opting out of model training. Scientific American has already added it to its file, with the comment "Draft spec".

What publishers can take from this

  1. robots.txt is a negotiating tool. US outlets use it to reward their partners and keep everyone else out, and News Corp's allowlist is the clearest example.
  2. Blocking is not enough to disappear. Outlets that block almost everything are still among the most cited. Publishers that want to get paid for their content end up negotiating or in court.
  3. Sponsored content gets its own rule. Opening only the sponsored sections to AI is already standard practice, in the US and in Spain.
  4. Measure it. The only way to check whether the strategy works is to know which bots actually visit your site (something Agent Analytics shows from your server logs) and which answers your outlet appears in.

For comparison, we ran the same analysis on 88 Spanish news outlets (in Spanish). The gap is large: in Spain, 35% of outlets block no AI bot at all, and only 12% shut out Gemini.

Methodology

  • Date. We reviewed the files on November 16, 2026.
  • Sample. We analyzed the robots.txt of 86 US news outlets, chosen to cover the major newspaper groups, local papers, TV networks, news agencies, magazines and tech sites. We downloaded 78 with an automated client. The other 8 (Politico, CNBC, NPR, Axios, Miami Herald, The Kansas City Star, Star Tribune and Newsmax) reject that kind of request, so we checked them in a browser the same day; in some cases we worked from an excerpt that keeps the AI bot rules intact. PBS was left out: its site returns an error in a browser too.
  • Reading. The automated client identifies itself as an analysis tool, without posing as any browser or bot. We read each file according to the robots.txt standard (RFC 9309): we look for the group of rules that names the bot and, if there is none, apply the general group. We count an outlet as "blocking" a bot when its rules stop the bot from reading article paths; for the McClatchy papers, which bar AI bots from pages ending in .html, we tested those pages, which are their articles.
  • The 15 bots. GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); Google-Extended (Google); ClaudeBot, Claude-SearchBot and Claude-User (Anthropic); PerplexityBot and Perplexity-User (Perplexity); meta-externalagent (Meta); Applebot-Extended (Apple); Amazonbot (Amazon); Bytespider (ByteDance); CCBot (Common Crawl); and MistralAI-User (Mistral). We also looked at smaller bots, but they do not count toward the strategies.
  • Strategies. "Open doors": no block among the 15 bots. "Blocks training": some block, but none on the six search and user bots of OpenAI, Anthropic and Perplexity. "Shuts out the assistants": blocks all six. "Picks partners": the rest.
  • Limits. robots.txt states an intention and does not prevent access: a firewall or content delivery network can block or allow a bot regardless of what the file says. Licensing deals can deliver content through other channels. This is a snapshot of one day, and these files change often.
  • Citation data. It comes from our public study of US news outlets cited by AI: answers to unbranded questions generated for the US, with each outlet counted once per answer.

About LLM Pulse

LLM Pulse measures how brands and publishers show up in the answers of ChatGPT, Gemini, Perplexity, Google's AI Mode, AI Overviews and other assistants: mentions, citations, Share of Voice and sentiment, tracked weekly or daily across markets. For a publisher, that means knowing which answers it appears in, which competitors get cited instead, and how those numbers move when it changes its robots.txt or signs a deal.

Measure your outlet's visibility in AI answers

See which ChatGPT, Gemini, Perplexity and Google answers mention and cite you, in the US and in other markets.

View plans

Frequently asked questions

Discover your brand's visibility in AI search effortlessly

Start free trial