PatientAgentBench — Amazon Science's Benchmark for Patient-Facing Health AI Agents
— by Vivax
Amazon Science has released PatientAgentBench, a reproducible, clinician-vetted benchmark built specifically for patient-facing health AI agents.
On July 29, 2026, Amazon Science released PatientAgentBench, a reproducible, clinician-vetted evaluation standard built specifically for patient-facing health AI agents — with an accompanying paper on arXiv and the full framework open-sourced on GitHub. The premise is simple but consequential: healthcare AI is shifting from answering questions to completing tasks on a patient's behalf — scheduling visits, managing prescriptions, triaging symptoms — and the benchmarks the field grades itself on were never designed to measure that. PatientAgentBench is an attempt to close that gap, and its findings are a useful map of where the whole field, including us, still has work to do.
The core argument is one we have made ourselves: static medical-exam benchmarks miss what a patient-facing agent actually has to do. Most healthcare AI evaluations fall into two camps — those that quiz medical knowledge in single-turn, exam-style questions, and those that test tool-using agents on technical tasks done for clinicians rather than conversations with patients. Neither captures the real job: reasoning over a patient's record across many turns, deciding when to gather more information, when to act, and when to escalate, all while holding a clinical-safety line. Worse, once a fixed benchmark is published it leaks into training data, so a model can learn the answer key and inflate its score through memorization rather than reasoning.
PatientAgentBench sidesteps both problems with a three-phase pipeline. It generates a synthetic patient chart and health record, derives a realistic clinical vignette from it, and spins up a patient agent that converses with the health-AI system under evaluation. The system under test is itself an agent — a base model wrapped in a harness that governs how it reasons over context and uses the benchmark's stateful, simulated healthcare tools. Then an LLM-as-a-jury panel scores each conversation against more than 100 clinician-vetted criteria across six dimensions: clinical safety, triage quality, workflow accuracy, task completion, clinical helpfulness, and conversational quality. Because the criteria are reusable and every scenario is freshly generated from a seed distribution, the benchmark extends to new domains and populations without new physician annotation — and there is no fixed answer key to memorize. Licensed clinicians validated the automated jury, whose scores aligned with human annotators and, reassuringly, err toward flagging a potential problem rather than missing a real one. Every profile, narrative, and conversation is fully synthetic; no real patient data is used.
The most striking finding is a severity paradox. The hardest cases were not emergencies — an obvious crisis triggers strong safety and triage behavior — but routine administrative requests from clinically complex patients. A simple pharmacy update from someone with multiple active medications and a documented mental-health concern is exactly the kind of request most agents processed transactionally, without pausing to screen for risk. That mirrors human medicine, where diagnostic errors cluster in the mundane cases that quietly hide danger, not the dramatic ones. It is a pointed reminder that an agent scoring well on clearly severe scenarios can still be unsafe where it matters most.
The safety failures also concentrated in specific, nameable patterns. The dominant one was crisis-resource omission — recognizing suicidal ideation, for instance, but failing to surface hotline or escalation information. A second was clinical fabrication: invented provider credentials, fake citations, and claimed tool executions that never actually ran. Crucially, more-capable models narrowed these gaps but did not close them; the strongest systems still left clinically important cases unhandled. Amazon frames this not as an indictment of any one model but as a roadmap — the per-dimension scores and written explanations show precisely where model choice helps and where agent design has to compensate for what raw capability alone does not deliver.
We read PatientAgentBench as strong outside validation of an evaluation philosophy Vivax has held from the start: clinical AI has to be judged the way care is actually delivered — multiturn, record-grounded, and safety-first, not on exam trivia. That the hardest cases are the routine-looking ones, that capability alone does not guarantee safety, and that the failures that matter only surface in sustained, tool-using conversations over a real patient record — these are the exact assumptions behind how we build and evaluate our clinical world model and AcuDx. A world model that represents how a patient's state evolves, an agent scoped to a specialty rather than the open web, cryptographically safe handoffs, and evaluation that scores the whole conversation against clinician-vetted criteria rather than a single answer: that is the same standard PatientAgentBench is pushing the field toward. We welcome an open, reproducible benchmark for patient-facing agents — read the paper on arXiv, the framework on GitHub, and the announcement on Amazon Science.