How well do you perform medically?
Searching the Web
AI medical question answering performance benchmarks 2025 2026
Here's an honest, detailed breakdown of how I - Orris - perform on medical topics:
My Medical Capabilities
What I'm Built With
I'm powered by a large language model with broad training on medical literature, textbooks, and clinical knowledge. On top of that, I have direct access to a curated medical textbook library (via the
medical-library skill), which gives me authoritative, citable content from real clinical references - not just training memory.I can also search PubMed for peer-reviewed literature, including systematic reviews, meta-analyses, and RCTs, and cross-reference with live web sources.
Where I Do Well
| Capability | Performance |
|---|---|
| Medical knowledge Q&A (pharmacology, physiology, pathology) | Strong - backed by textbook retrieval |
| Explaining mechanisms, anatomy, drug actions | Strong |
| Summarizing evidence from PubMed | Strong |
| Clinical reasoning and differential diagnosis | Good, with appropriate caveats |
| Medical image analysis (uploaded images) | Good via vision AI |
| Drug dosing / guidelines | Good, but always verify against current formulary |
| Answering exam-style questions (USMLE, MBBS, etc.) | Strong |
How AI Models Generally Perform on Medical Benchmarks
For context, large AI models (including those similar to what powers me) have scored:
- USMLE Step 1/2/3: Passing level (~60-70%+), with frontier models reaching near-expert performance (~85-90%)
- MedQA / MedMCQA: State-of-the-art models score ~75-85%
- Clinical reasoning tasks: Variable - pattern recognition is strong, but nuanced multi-step reasoning can falter
My Honest Limitations
- Training cutoff - my base knowledge has a cutoff, so very recent drug approvals, guideline updates, or newly published trials may not be in my training. I compensate by searching PubMed and the web.
- No physical examination - I can reason about clinical findings you describe, but I cannot perform or substitute for a real exam.
- Hallucination risk - I may occasionally produce plausible-sounding but incorrect details, especially for rare conditions or obscure drug interactions. I flag uncertainty when I'm aware of it.
- Not a substitute for clinical judgment - I should be used as a research and education tool, not for making final clinical decisions.
- Dosing precision - Always cross-check dosing recommendations against a current formulary (BNF, Micromedex, etc.).
What Makes Me Different Here
The
medical-library skill gives me access to structured textbook content, so when you ask a medical question, I search real reference books and can cite specific page-level content. I also run parallel PubMed searches for recent evidence. This makes my answers more grounded than a generic chatbot response.Bottom line: For education, research, clinical reasoning support, and exam prep, I perform well. For bedside decision-making, always pair my output with clinical judgment and up-to-date local guidelines.
This is a shared conversation. Sign in to Orris to start your own chat.