Cloudflare and OpenAI Share First Results of Their AI Discovery Research Pilot
Cloudflare's official blog says the joint research pilot launched in July 2026 has brought hundreds of millions of previously undiscovered pages into OpenAI's search index, and shares what it learned about dead pages, change signals and page-level risk flags
Tool Overview
Details, capabilities and usage guidance
Features
- ✓ Scale results already: Cloudflare says the joint pilot used a participating cohort of about 1% of its total network and has enabled OpenAI to discover and index hundreds of millions of new web pages, giving ChatGPT more sources to draw on so people find more relevant, timely information - with links back to the original content and the publishers behind it
- ✓ Filtering out pages that look useful but are not: Cloudflare says automated traffic can easily make a nonexistent page look interesting - bots repeatedly ask for files that are not there, some sites redirect almost any address to their homepage, and other addresses lead to copies of the same content. Cloudflare says recognising these patterns cuts wasted work before a crawler fetches a page
- ✓ Change signals cut repeat crawling: Cloudflare says that for a website owner a crawler repeatedly downloading an unchanged page adds load without benefit. Cloudflare says the pilot uses change signals to tell crawlers when a page is worth another visit and when it is not, aiming to keep discovering and stay current while reducing the work that takes
- ✓ Page-level risk flagging: Cloudflare says it is testing signals that identify and flag potentially harmful pages. Cloudflare says a legitimate site can have a single compromised page, such as a fake login form planted by an attacker, and that flagging that one page rather than the whole domain keeps people safe without penalising the site owner; early results suggest these signals help keep pages out of an index
- ✓ Rights and boundaries for participants: Cloudflare states clearly that a signal about a page does not grant permission to access it, and that this research uses metadata about the web for search and discovery rather than sharing page content for model training. Cloudflare says site owners can use its AI Crawl Control to understand and manage how AI systems access their website
How to Use
- Read the official Cloudflare blog post to understand the pilot's scope, participation scale and the early results shared so far
- If you own a site, first decide where you stand on AI using your content: Cloudflare says content that chooses to be available to AI systems becomes easier to discover
- Use Cloudflare's AI Crawl Control to see how AI systems access your website and to manage that access according to your own rules
- Watch the official next phase: Cloudflare says it will improve page selection, test faster and more reliable change signals, and measure what happens after content is discovered
FAQ
What is this pilot, and who ran it?
Cloudflare's official blog says Cloudflare and OpenAI launched the joint research pilot in July 2026 and shared the first results in an official blog post on 15 October 2026. Cloudflare says the goal was to find out whether signals from its network could help ChatGPT users discover new content more efficiently.
How large is the pilot, and what were the results?
Cloudflare says the pilot used a participating cohort of about 1% of its total network and has enabled OpenAI to discover and index hundreds of millions of new web pages. Cloudflare says the answer so far is a resounding yes: the pilot brought previously unknown pages into OpenAI's search index, giving ChatGPT more sources to help people find relevant, timely information.
Is site content being used to train models?
Cloudflare states that this research uses metadata about the web for search and discovery, rather than sharing page content for model training. Cloudflare also stresses that a signal about a page does not grant permission to access it.
How can a site owner manage AI access?
Cloudflare says site owners can use Cloudflare's AI Crawl Control to understand and manage how AI systems access their website. Cloudflare adds that for those who choose to make their content available to AI systems, these improvements make discovery easier and more efficient.
Why flag a single page instead of a whole domain?
Cloudflare says a legitimate site can have a single compromised page, such as a fake login form planted by an attacker. Cloudflare says flagging that one page rather than the whole domain keeps people safe without penalising the site owner, and that early results suggest these signals can help identify pages that should be kept out of an index.
What comes next?
Cloudflare says the next phase will build on what the pilot learned: improving the selection of pages, testing faster and more reliable change signals and measuring what happens after content is discovered. It says it will keep working with OpenAI to understand how better site discovery translates into real business outcomes.