# Methodology

## Research question

When a brand-free commercial prompt asks which local medical providers to compare, which provider organizations appear in the query strings of the first returned web-search action, and how often are those organizations present in the final four-provider answer?

## Design

- 180 completed responses across 72 exact prompt cells.
- Six service lines: fertility, orthopedics, ophthalmology, urgent care, primary care, and bariatric surgery.
- Three metros: New York, Chicago, and Houston.
- Every prompt had commercial provider-selection intent and named no provider brand.
- Broad-comparison cells used two frozen phrasings and three fresh responses per cell. Appointment-access and cost/insurance cells used one frozen phrasing and two fresh responses per cell.
- Collection used one `chat-latest` OpenAI Responses API configuration with required web search, low search-context size, `store=false`, a fresh conversation for every response, a metro-level approximate user location, and a 1,200-token output ceiling.

## Observable stages

The analysis uses two observable stages: provider organizations identified in the query strings of the first returned web-search action, and provider organizations named in the final four-provider answer. Returned source pages were not normalized into a separate provider-discovery stage.

## Metrics

- **Final positions already named in the first action:** final provider positions occupied by organizations already present in first-action query strings, divided by all final provider positions.
- **First-action-to-final retention:** first-action provider occurrences also present in the final answer, divided by all first-action provider occurrences.
- **Final providers absent from the first action:** final provider positions occupied by organizations absent from the first-action query set, divided by all final provider positions.
- **Stability:** pairwise Jaccard similarity between provider sets from fresh repetitions of the exact same prompt. Cell values are averaged without weighting when a scope-level stability value is reported. The public `stability-pairs.csv` exposes the intersection count, union count, and Jaccard value for every repeat pair without exposing provider names.
- **Intervals:** 95% percentile intervals from 10,000 bootstrap samples, with the exact prompt cell as the resampling unit. In scenario order (broad comparison, appointment access, cost and insurance), retention seeds were 2026081810, 2026081811, and 2026081812; final-absent seeds were 2026081820, 2026081821, and 2026081822.

The retention and final-absent metrics have different denominators and must not be treated as complements.

## Entity handling

Provider organizations were normalized using explicit alias rules, parent-child deduplication within responses, targeted manual review of unmatched query strings, and documented adjudication of an ambiguous location-versus-provider phrase. Automated rules were corrected before publication when a substring collision was identified.

## Interpretation

The study describes observable output from one API configuration in one collection window. The `chat-latest` alias can change over time. Repeats estimate stochastic variation for frozen prompts; they do not turn 72 prompt cells into a probability sample of all US medical searches. Service-line, metro, and prompt-phrasing cuts are exploratory. The study does not expose hidden chain of thought or establish that a provider name came from training data.