OpenAI is building its own search index. Here's what it means for you.

Between May and July 2026, researchers reverse-engineering ChatGPT's server-side events found four active retrieval sources working simultaneously: OpenAI's own "Labrador" index, two third-party scraping services (Bright Data, and Oxylabs), and a live SERP fallback, plus an internal test named "prefer-index-over-serp-v3" that's actively shifting query routing toward OpenAI's own index.

This was the first concrete, publicly available proof that OpenAI is routing real traffic through infrastructure it built, not just testing it in a lab.

What does this mean? If your website content isn't reachable by whichever of these four paths a given query hits, you have no visibility into a growth channel expanding independently of Google, and Bing. Every month you're invisible there, a competitor that isn't pulls further ahead.

What follows sorts confirmed from inferred from contested, so your crawl configuration, robots.txt, and content structure decisions are built on evidence.

What we actually know: the evidence chain for openAI's retrieval index

The evidence for OpenAI's independent index didn't arrive all at once. It accumulated across five distinct sources over roughly 18 months.

April 22, 2025, sworn testimony. In the Google antitrust remedies trial, OpenAI's head of product Nick Turley confirmed under oath that OpenAI is building its own search index. He also confirmed that Google rejected OpenAI's request to use Google's search index to power ChatGPT. He confirmed OpenAI had "significant quality issues" with non-Google search API partners, widely understood to mean Microsoft Bing.

Those issues were significant enough to accelerate the decision to build in-house. Critically, Turley revised the coverage target downward: the goal of using the index for 80% of ChatGPT searches, originally set for end of 2025, was pushed to "years down the line" (courthousenews.com, April 22, 2025; pymnts.com, April 22, 2025).

Before OpenAI made any public announcement, llmrefs.com documented an external_web_access parameter in the Responses API Web Search feature. When set to false, ChatGPT answers queries purely from cached indexed content, no live page fetch, no third-party SERP call. This is API-level proof that a functional independent index exists, separate from live search, and that it was operational months before OpenAI acknowledged it publicly.

April 2026, Help Center confirmation. OpenAI's official Help Center publicly confirmed the index for the first time, stating that eligible ChatGPT workspace accounts can enable "offline web search" using "OpenAI's indexed, and cached web content" instead of a live external search provider (help.openai.com, April 2026). This is the first first-party disclosure, but it describes a specific workspace feature, not the full production architecture.

May 22, 2026, crawl volume data. Botify and Nectiv analyzed approximately 7 billion server log entries covering November 2024 through March 2026. OAI-SearchBot activity tripled after August 2025, a 3.5x increase, and now slightly exceeds GPTBot requests by count. The absolute gap against established indexes, however, remains large: OpenAI's total crawl volume represents approximately 4% of Google's, and roughly 14% of Bing's based on the most recent month measured (887 million OpenAI events versus 18.2 billion Google events).

September 4, 2026, server-side event analysis. Peec AI researchers reverse-engineered ChatGPT's server-side events between May 21 and July 21, 2026, and identified a result_source field with four values: Labrador (OpenAI's own index), Bright (likely Bright Data scraping Google), Oxylabs, and SERP. They also identified the A/B test named "prefer-index-over-serp-v3" and references to specialized sub-indexes for news, shopping, PDFs, YouTube, and academic papers (searchenginewatch.com citing Peec AI, September 4, 2026).

The four result_source values identified describe meaningfully different retrieval mechanisms, not four labels for different data providers. This is the infrastructure that external_web_access: false exposes at the API level, a genuine independent index.

Bright and Oxylabs are commercial scraping and proxy services. When a query routes to either of these, ChatGPT is likely fetching Google SERP results through an intermediary rather than pulling from its own cached content. This is architecturally closer to the Bing-dependency model than to a true independent index; the source content is Google's, not OpenAI's.

SERP as a distinct value likely represents direct SERP retrieval, though the precise mechanism hasn't been confirmed by OpenAI.

The operational implication is that "ChatGPT search" is not a single retrieval path, it's a routing layer over multiple backends. A page could appear in Labrador's cache, in Bright Data's scrape of Google, in Oxylabs' data, and in a live SERP result simultaneously, or not in any of them, depending entirely on how a given query gets routed.

What openAI's crawl growth actually signals, and what it doesn't

The Botify and Nectiv data is the most concrete infrastructure signal available, and it rewards careful reading rather than headline-level interpretation.

OAI-SearchBot tripling after August 2025 is significant. The timing corresponds with the internal push toward index completeness that Turley's testimony described, and the fact that OAI-SearchBot (the search indexing crawler) now outpaces GPTBot (the training crawler) by request count suggests OpenAI is prioritizing search index coverage over training data collection, at least as measured by crawl activity.

The 3.5x growth rate is also significant, but the absolute starting point matters. At approximately 4% of Google's crawl volume and 14% of Bing's, even aggressive growth leaves OpenAI's index materially incomplete relative to what teams expect from a primary search index. Google's index has decades of crawl accumulation behind it. OAI-SearchBot's current trajectory, even sustained, closes that gap over years, not months.

What this means practically: OAI-SearchBot is actively crawling and the index is growing. Content that was not crawlable before may be indexed now. But assuming your content is already in Labrador's index, or that Labrador returns comprehensive results across your topic area, is not yet warranted by the crawl volume evidence.

Peec AI also reported references to an internal crawl experiment of up to one billion pages, though OpenAI hasn't confirmed it. Treat it as a signal to watch, not a number to cite: if real, it would materially expand the index beyond current estimates.

The robots.txt and crawler distinction: what's settled, what's contested

There are three OpenAI crawlers teams need to treat as distinct, because they behave differently, and may respect different directives.

GPTBot is OpenAI's training crawler. The consensus, established clearly enough to treat as settled, is that blocking GPTBot affects training data collection, not search indexing. If your goal is managing AI search visibility rather than training opt-out, GPTBot directives are the wrong lever.

OAI-SearchBot is OpenAI's search indexing crawler, and the one that has tripled in activity since August 2025. This crawler populates the Labrador index. Allowing OAI-SearchBot while blocking GPTBot is the current recommended approach for sites that want search visibility without training opt-in, but this is a policy distinction, not a technical guarantee, and OpenAI could change how these crawlers are labeled or what they actually do.

ChatGPT-User is the live browsing bot, the one that fetches pages in real time during conversations. As of December 2025, this crawler was reported to ignore robots.txt restrictions, with OpenAI's stated rationale being that live browsing during conversations serves user intent (this position is contested legally, and ethically as of September 8, 2026).

If you want inclusion in Labrador's index, you need to explicitly allow OAI-SearchBot. Many sites still run blanket blocks on AI crawlers that don't distinguish training bots from indexing bots. As of the Botify data (May 22, 2026), that means being invisible to OAI-SearchBot's active indexing pass while competitors are not.

Two questions remain open. Whether blocking OAI-SearchBot indirectly affects ChatGPT's routing decisions for a domain at all, and what happens to sites that allow the crawler but have structural issues preventing effective indexing. Neither has an answer yet. Closing them requires architecture disclosure from OpenAI, or independent replication of findings that goes further than what's been published.

What this means for content indexability decisions in the next 12 months

Given the multi-source routing model and the current state of disclosure, here is what you can act on with reasonable confidence, and what you cannot.

Act on: OAI-SearchBot access. If a site is blocking OAI-SearchBot via robots.txt, whether through a blanket AI-crawler block, or a specific directive, that site is opting out of Labrador index consideration. With OAI-SearchBot now the most active OpenAI crawler by request count and the index actively expanding, this is no longer a low-stakes oversight. Audit existing robots.txt configurations for OAI-SearchBot explicitly. You can run an AI search audit to identify whether crawlers are currently reaching your key pages.

Act on: content structure for sub-indexes. For the specialized sub-indexes identified for news, shopping, PDFs, YouTube, and academic papers: OpenAI has not confirmed the sub-index names or their operational status. This is reverse-engineered inference, not disclosed architecture. But the practical logic holds regardless: if retrieval for PDFs, news, and academic content follows different logic than general web content, structuring your content to signal its type clearly is not a speculative optimization. It's the same principle that governs structured data for Google's specialized search surfaces, and it's worth prioritizing now, before it's table stakes, if your content naturally falls into these categories: research reports, product listings, news publishing.

Act on: parallel-path assumptions. The multi-source routing model means optimizing for ChatGPT visibility is not equivalent to optimizing for any single index. A page that ranks well in Google SERP already has a retrieval path through Bright/Oxylabs if OpenAI is scraping Google for those queries. A page that is well-structured and crawlable by OAI-SearchBot has a Labrador path. A page that does neither well has no reliable path.

Do not act on: source weighting assumptions. Until OpenAI discloses how queries are routed across Labrador, Bright, Oxylabs, and SERP, or until an independent audit replicates and extends current research findings, there is no defensible basis for assuming that optimizing for Labrador is more valuable than optimizing for Google SERP, or vice versa. Don't bet the whole budget on one path when the honest answer is that both matter and the split is unknown.

Do not act on: index completeness claims. At 4% of Google's crawl volume, Labrador's index is materially incomplete. The index is growing, the trajectory is real, and the right posture is to ensure crawlability now so content is positioned for inclusion as coverage expands, not to assume current indexing is comprehensive. Set the expectation as a trajectory, not a milestone.

Where this leaves you

The infrastructure picture for ChatGPT's retrieval index is materially clearer than it was 18 months ago. The index exists, it's in production, it's being fed by an actively growing crawler, and OpenAI is running tests to shift more query routing toward it. That progression is real and the evidence base is solid.

What hasn't changed is the disclosure gap. OpenAI has confirmed the index exists. It has not confirmed how it's weighted against third-party sources, what the sub-index architecture actually is, or how ChatGPT-User's live browsing interacts with robots.txt directives going forward. The reverse-engineering work is the most detailed independent analysis available, and it's a reasonable basis for forming hypotheses. It's not a substitute for architectural disclosure.

The practical posture through the next 12 months: allow OAI-SearchBot explicitly, structure content to signal type clearly, treat parallel retrieval paths as the baseline assumption, and resist any framework that claims certainty about source weighting.

If you want to assess where your content actually stands against these retrieval paths rather than working from aggregate estimates, an AI search audit is the starting point, not because it replaces architectural disclosure, but because crawl access, and content structure are the variables you can actually control right now.