What Happened
On June 23, 2026, Nature Medicine published a head-to-head benchmark that pitted three general-purpose large language models — OpenAI's GPT-5.2, Google's Gemini 3.1 Pro, and Anthropic's Claude Opus 4.6 — against two purpose-built clinical decision-support tools, OpenEvidence and Wolters Kluwer's UpToDate Expert AI. The questions were not tidy multiple-choice vignettes. They were the messy, unstructured queries that practicing physicians actually type at the point of care, and across every benchmark the researchers tested, the general-purpose models came out ahead.
The detail that matters is the kind of question being asked. Board-exam items and curated case studies have long flattered medical AI, because they reward pattern matching over judgment. This study leaned on genuine clinician questions instead, and on that terrain neither specialist tool could claim an edge. The result quietly reframes an assumption that has shaped a decade of health-tech procurement: that a system built specifically for medicine, and reviewed as a medical device, should naturally outperform a generalist chatbot at the bedside. According to the paper, that intuition simply did not hold.
Why It Matters
The finding lands squarely on a regulatory fault line. The two specialist tools reached the market through defined medical-device pathways; the generalist models that beat them did not pass through any process calibrated to clinical-decision-support performance.
.jpg){kind=link}
The U.S. Food and Drug Administration's most current statement on AI in medical devices — its December 2024 final guidance on Predetermined Change Control Plans — governs how a cleared algorithm may change after approval, not whether its baseline performance is competitive with the unregulated alternatives doing the same job. The agency's public list of AI-enabled devices likewise documents that a product completed a process; it says nothing about how that product stacks up against a general model a clinician could open in a browser tab. When a cleared tool quietly underperforms an uncleared one, the clearance record carries no signal of that gap. Two distinct questions are in play — does the device meet its own specification, and does it beat the free alternative — and only the first has an answer on file.
Reaction
The study did not arrive in a vacuum. Physicians are already reaching for these tools, cleared or not, and the data on that shift is striking.
The American Medical Association's most recent physician survey found that roughly two-thirds of doctors now use AI in practice, up from just over a third a year earlier, and a comparable majority report a definite advantage from it. That adoption curve means the comparison the paper draws is not academic. Clinicians are making treatment-adjacent decisions with AI assistance today, and for the user the regulatory status of a tool has become a secondary concern behind whether it actually answers the question well. Health systems and trial sponsors that have treated clearance as a proxy for fitness now have peer-reviewed reason to ask whether that proxy holds in the specific domains where they deploy these systems.
What's Next
Regulators are not blind to the problem, but their frameworks were built to answer a different question than the one this study poses.
The European Medicines Agency's 2024 reflection paper on AI takes a risk-tiered view, reserving its heaviest scrutiny — including prospective performance monitoring — for systems that could sway trial endpoints or safety reporting. Its logic implies that a high-stakes tool should at least match the available alternatives, not merely stay internally consistent against a sponsor-defined benchmark. In the United States, the FDA's Digital Health Center of Excellence has signaled updated guidance on how clinical-decision-support software is categorized, expected during 2026. How that guidance treats a general-purpose model performing the same function as a cleared tool will decide whether this performance gap becomes a formal classification question or lingers as unresolved risk that sponsors and hospitals quietly absorb on their own.
Closing Thoughts
There is a temptation to read the study as a victory lap for the large model labs, but the more durable lesson is about measurement, not marketing.
For years the medical-AI conversation has equated a regulatory stamp with quality, and a specialist label with suitability. This benchmark gently pulls those two ideas apart. A tool can clear every procedural hurdle and still trail a generalist on the questions clinicians care about most; conversely, a model never designed for medicine can excel at it without anyone having formally checked. None of that makes clearance worthless — provenance, accountability, and post-market surveillance still matter enormously in a field where errors carry real harm. What it means is that the questions we validate against have to catch up with the questions actually being asked. Until comparative, real-world performance becomes part of how these tools are judged, the most telling number about a clinical AI system may be the one that no filing currently requires it to report.
Sources: Nature Medicine · Becker's Hospital Review · FDA AI-Enabled Medical Devices List · American Medical Association
한글 요약
2026년 6월 23일 네이처 메디신은 범용 대규모언어모델(OpenAI GPT-5.2, 구글 Gemini 3.1 Pro, 앤트로픽 Claude Opus 4.6) 세 종을 의료 전용 의사결정 지원 도구인 OpenEvidence와 월터스클루어 UpToDate Expert AI와 정면 비교한 벤치마크 연구를 발표했다. 정돈된 객관식이 아니라 임상 현장에서 의사가 실제로 입력하는 비정형 질문을 사용했고, 모든 항목에서 범용 모델이 전용 도구를 앞섰다. 의료용으로 설계되고 의료기기로 심사받은 도구가 일반 챗봇보다 당연히 나을 것이라는 오랜 전제가 흔들린 셈이다.
이 결과는 규제의 빈틈을 드러낸다. 전용 도구들은 정해진 의료기기 인허가 경로를 통과했지만, 그들을 앞선 범용 모델은 임상 의사결정 성능을 기준으로 한 어떤 절차도 거치지 않았다. FDA의 2024년 12월 사전변경관리계획(PCCP) 지침은 승인된 알고리즘이 어떻게 바뀔 수 있는지를 규율할 뿐, 그 도구의 기본 성능이 규제받지 않는 대안과 견줄 만한지는 묻지 않는다. 인허가 기록은 '절차를 마쳤다'는 사실만 담을 뿐, 무료로 열 수 있는 범용 모델과의 비교 우위는 담지 못한다. 미국의사협회 조사에서 이미 의사 약 3분의 2가 진료에 AI를 쓴다고 답한 만큼, 이 비교는 더 이상 학술적 호기심에 그치지 않는다.
유럽의약품청(EMA)의 2024년 AI 성찰 보고서는 위험 등급제 접근을, FDA 디지털헬스 우수센터는 2026년 중 임상의사결정지원 소프트웨어 분류 지침 갱신을 예고했다. 다만 핵심은 모델 경쟁의 승패가 아니라 '무엇을 기준으로 검증하느냐'에 있다. 인증 도장이 곧 품질이고 전용 라벨이 곧 적합성이라는 등식은 이번 연구로 분리됐다. 인증이 무의미하다는 뜻은 아니며 출처·책임·시판 후 감시는 여전히 중요하다. 다만 실제로 던져지는 질문을 기준으로 한 비교 성능이 평가 체계에 들어오기 전까지는, 임상 AI에 관한 가장 중요한 숫자가 정작 어떤 서류에도 요구되지 않는 숫자일 수 있다.