Rotgar Research · Medical

When ChatGPT Adds Caveats to Clinic Recommendations

A secondary analysis of 450 repeated New York City answers examined whether clinic-specific commercial and operational caveats were supported by current primary sources—and whether competing clinics were evaluated on comparable evidence.

By Evgeniy Yudin, Founder and Strategy Lead Methodology reviewer — Boris Teplyakov, SEO Lead Published September 3, 2026 Version 1.0 US English

Methodology review only; not clinical review, peer review, legal review, a formal external audit or an assessment of the providers named in the answers. The study does not assess treatment quality, safety, clinical appropriateness or which clinic a patient should choose.

Read the findings Download the data

Direct answer

Most caveats had a factual basis, but 48.8% could not be repeated as written without qualification

Of 43 deduplicated clinic-specific caveat families, 22 (51.2%) were fully supported by current or correctly dated primary sources. Another 16 (37.2%) had a factual basis but were overstated, applied asymmetrically or built on comparisons that were not like for like. Two (4.7%) were historically supported but stale as current claims, two (4.7%) were not supported by the checked primary sources, and one (2.3%) could not be verified.

In total, 21 of 43 claim families (48.8%) could not be repeated as written without qualification. This combines the partly supported or overstated, stale, not supported and unverifiable categories; it does not mean that 48.8% were false.

No serious clinic-specific allegation was identified under the study’s coding rules. Across the 450-answer corpus, no coded claim involved malpractice, lawsuits, sanctions, fraud, unsafe care or poor patient outcomes. The observed issue was narrower: ChatGPT sometimes converted a limited fact, missing field or old document into a broader commercial impression.

The central finding is evidence asymmetry. In some answers, one clinic received a specific, checkable caveat while another received broad positive language without an equivalent check. A competitor without a visible caveat was not necessarily better documented or more reliable; it was sometimes simply not evaluated on the same field.

This study does not show that publishing more information causes worse framing, that publishing less information causes worse framing, or that one named clinic is clinically better than another.

43deduplicated clinic-specific claim families fact-checked

51.2%fully supported by current or correctly dated primary sources

37.2%had a factual basis but were overstated, asymmetric or not like for like

0 of 450serious clinic-specific allegations identified under the study’s coding rules

Study at a glance

A claim-level audit of a frozen 450-answer corpus

Parent corpus: 450 answers
Claim families: 43
Same-answer contrasts: 18
Medical service lines: 3
Prompt geography: New York City
Source-check date: August 29, 2026
Core collection date: August 18, 2026
Unit of analysis: Deduplicated claim family
Clinical ranking performed: No

This is a companion analysis of the original 450-answer shortlist and source-stability study. The parent release is archived under version DOI 10.5281/zenodo.22162725.

Verdict distribution

Support was common, but evidence status was not binary

The five-part coding distinguishes a bounded supported fact from wording that widened the fact, used old evidence, lacked current support or could not be checked.

Fully supported22 · 51.2%
Partly supported or overstated16 · 37.2%
Stale2 · 4.7%
Not supported2 · 4.7%
Unverifiable1 · 2.3%

Figure 1. Verdicts for 43 deduplicated claim families. Exact counts and definitions are repeated in the table below.

Table 1. Claim-family verdict distribution
Verdict	Claim families	Share	Meaning
Fully supported	22	51.2%	Current or correctly dated primary evidence supported the bounded claim.
Partly supported or overstated	16	37.2%	A narrow fact existed, but the wording widened it, compared unlike fields or applied the check asymmetrically.
Stale	2	4.7%	The fact was historically documented but was not suitable as a current claim on the audit date.
Not supported	2	4.7%	The checked current primary source did not support the statement; in one case it directly contradicted the claimed ambiguity.
Unverifiable	1	2.3%	The relevant primary page was unavailable, so the claim could not be confirmed or rejected.

Displayed category shares sum to 100.1% because each share is rounded independently to one decimal place.

Denominator: 43 deduplicated claim families, not 450 independent answers and not 43 measures of clinic quality.

Findings by niche

The type of caveat differed across the three service lines

Table 2. Verdict distribution within each niche
Niche	Claim families	Fully supported	Partly supported / overstated	Stale	Not supported	Unverifiable
IVF	19	10 (52.6%)	8 (42.1%)	0 (0%)	1 (5.3%)	0 (0%)
Full-arch dental implants	18	9 (50%)	7 (38.9%)	0 (0%)	1 (5.6%)	1 (5.6%)
Bariatric surgery	6	3 (50%)	1 (16.7%)	2 (33.3%)	0 (0%)	0 (0%)

IVF caveats most often concerned package exclusions, prices, insurance participation and the age or precision of published outcome data. Dental answers more often turned a missing field or workflow description into an inference about price, timing, credentials or clinic maturity. Bariatric caveats included exact outcome figures and historical price or fee documents, making source date and metric scope particularly important.

The niche percentages are descriptive. Each niche contains a small number of deduplicated claim families, and the families differ in commercial importance and recurrence.

Concrete fact checks

Seven examples show where the distinction changed the commercial impression

Each finding below evaluates one bounded statement against the cited primary source as checked on August 29, 2026. It does not score the clinic, treatment or clinical outcome.

Extend Fertility