ChatGPT Named Most Final-Shortlist Medical Providers in Its First Search Queries

The answer in brief

Most final-shortlist providers were already present in the first observable query set

Across the 180 responses, 612 of 720 final provider positions, or 85.0%, were occupied by organizations already named in the query strings of the first returned web-search action.

At least one provider organization appeared at that stage in 179 of 180 responses. Each first action contained four query strings.

The relationship between that first observable candidate set and the final answer depended on the patient’s task. Broad comparison answers retained 93.0% of provider mentions from the first search action. Only 7.4% of their final provider positions contained an organization absent from that first action. For cost-and-insurance questions, retention fell to 72.4%, while 36.1% of final provider positions contained an organization absent from the first action.

The final shortlists also became less consistent across repeated runs. Mean provider-set similarity was 77.9% for broad comparisons, 61.2% for appointment-access questions and 50.4% for cost-and-insurance questions.

For a health system CMO, this creates two measurement questions: Did the organization enter the first observable candidate set? Did it remain in, or appear only in, the final answer? A single AI visibility score cannot answer both.

First action

Provider names appeared in 179 of 180 first search actions

The user prompts contained no provider brands. Even so, at least one provider organization appeared in the first returned search action in:

• 108 of 108 broad comparison responses;
• 36 of 36 appointment-access responses;
• 35 of 36 cost-and-insurance responses.

This is not a claim that 99.4% of providers were chosen before search. It is a response-level result: 179 of 180 first search actions contained at least one provider among their four query strings.

The query strings are observable API output. They show the names used at the first search step, but not where those names originated.

For health system CMOs

A measurement framework for health system CMOs

Track the stages separately:

1. First-action presence. Did the provider appear in any query string in the first returned search action?
2. Final inclusion. Did the provider appear in the final four-provider answer?
3. Retention. When the provider appeared in the first action, did it remain in the final answer?
4. Final-only presence. When the provider appeared in the final answer, was it absent from the first action?
5. Stability. How often did it appear across repeated runs of the same frozen prompt?

Build separate views for broad comparison, appointment access and cost or insurance. Then segment by service line and metro. This keeps a strong general-comparison result from masking a weak or unstable result for a high-intent patient task.

Use the result to choose the next audit:

• If a provider appears early and stays, document the commercial questions for which that pattern holds.
• If it appears early but drops out, compare the final answer’s retained providers and their supporting public information.
• If it appears only in the final answer, identify which public pages answer that specific patient task.
• If it rarely appears across repeats, treat the result as unstable rather than as a fixed rank.

These are measurement paths, not a universal optimization recipe. The study shows where shortlist changes occurred; the page and source review explains what to investigate next.

Methodology

A dated observational snapshot

The analytical sample contains 180 completed responses from August 18, 2026. Collection used the OpenAI Responses API with the requested and returned model alias chat-latest, required web_search, low search context, fixed approximate metro location, no prior conversation state, store=false and a 1,200-token output ceiling. OpenAI describes web search as a Responses API tool that lets models access current web information and return sourced answers.

The response was the observational unit. Provider names were normalized to organizations or consumer-facing provider brands. Rule-based extraction was followed by manual adjudication of every unmatched query string and all identified ambiguous positive matches. The analytical sample had complete coverage: four first-action query strings, four final providers and four citation URLs in every response.

Retention equals retained initial provider mentions divided by all normalized provider mentions in the first action. First-action-absent final presence equals final provider entities absent from the first action divided by all final provider entities. Stability is the mean within-cell pairwise Jaccard similarity of normalized provider sets across repeated responses. Ninety-five percent prompt-cell cluster bootstrap intervals used 10,000 replications with the exact prompt cell as the cluster unit; the six fixed scenario seeds are published in the downloadable methodology and scenario data. The resulting intervals were 89.8%–95.8%, 81.2%–92.2% and 66.2%–79.4% for retention; and 4.4%–10.9%, 9.7%–24.3% and 28.5%–44.4% for first-action-absent final presence, in broad comparison, appointment access and cost or insurance respectively.

The 180 analytical responses cost an estimated $16.98. Including the excluded feasibility pilot, calibration and stopped first attempt, total estimated API spend was $19.12.