Benchmarking Orris for Medical Accuracy

Why Standard Speed Benchmarks Don't Apply Here

Best-Fit Benchmarking Techniques

1. MedQA (USMLE-Style) — Highest Signal

• Questions require reasoning and factual grounding — exactly Orris's strength
• It has a well-established leaderboard (GPT-4 ~86%, top specialized models ~88–93%)
• Maps directly to clinical knowledge depth from textbooks
• Metric: % accuracy on the 4-option MCQ test set

2. PubMedQA — Evidence Retrieval Accuracy

• Metric: 3-way classification accuracy; current top ~80–96%

3. MedMCQA — Breadth Across Specialties

• Metric: % accuracy, optionally broken down by specialty

4. MMLU-Medical Subsets

• Metric: Per-subset and aggregate accuracy

5. CLEVER-Style Expert Blind Review (Most Defensible)

• Have practicing physicians blindly rate Orris responses vs. competitor responses on the same cases
• Rate on factuality, clinical relevance, and conciseness
• Use real clinical scenarios (not MCQs) — discharge summaries, clinical questions, differential diagnosis requests
• This directly benchmarks Orris against GPT-4o, Claude, Gemini, or specialized tools like Glass.AI or OpenEvidence
• Metric: Physician preference rate (%), agreement κ score

Recommended Stack for Orris

Competitor Baselines to Beat

• MedQA: GPT-4 ~86%, specialized medical LLMs ~88–93%
• PubMedQA: Top models ~79–96% depending on method
• MMLU-Med: GPT-4 ~87–91%

Key Differentiator to Highlight

• MedHALT (Medical Hallucination Test) — Orris's design explicitly counters hallucination via forced library lookup before answering
• Citation accuracy — does the cited page/book actually support the claim? This is measurable and almost no competitor tracks it

Practical Recommendation

Setting Up MedQA (USMLE)

What it is

Step 1 — Get the Data

git clone https://github.com/jind11/MedQA.git
cd MedQA

data/
  questions/
    US/
      4_options/
        train.jsonl
        dev.jsonl
        test.jsonl   ← use this for benchmarking

{
  "question": "A 45-year-old man presents with...",
  "options": {"A": "...", "B": "...", "C": "...", "D": "..."},
  "answer": "C",
  "answer_idx": "C"
}

Step 2 — Write an Evaluation Harness for Orris

import json
import re

def call_orris(question, options):
    """
    Replace this with your actual Orris API call.
    Should return Orris's full text response.
    """
    prompt = f"""You are answering a USMLE-style medical question.
Question: {question}
Options:
A) {options['A']}
B) {options['B']}
C) {options['C']}
D) {options['D']}

Reply with ONLY the letter of the correct answer (A, B, C, or D)."""
    # --- replace below with actual API call ---
    response = orris_api.call(prompt)
    return response

def extract_answer(response_text):
    """Parse the answer letter from Orris's response."""
    match = re.search(r'\b([ABCD])\b', response_text.strip())
    return match.group(1) if match else None

# Run evaluation
results = []
with open("data/questions/US/4_options/test.jsonl") as f:
    for line in f:
        item = json.loads(line)
        response = call_orris(item["question"], item["options"])
        predicted = extract_answer(response)
        correct = item["answer_idx"]
        results.append(predicted == correct)

accuracy = sum(results) / len(results)
print(f"MedQA Accuracy: {accuracy:.2%}  ({sum(results)}/{len(results)})")

Step 3 — Run on Test Set

Setting Up Med-HALT (Hallucination Test)

What it is

1. Reasoning Hallucination Tests (RHTs) — gives the model a wrong premise or a trick and tests if it hallucinates a plausible-sounding but false answer
2. Memory Hallucination Tests (MHTs) — gives a PubMed ID and asks the model to produce the article title; tests factual recall

Step 1 — Get the Data

git clone https://github.com/medhalt/medhalt.git
cd medhalt

from datasets import load_dataset

# Reasoning hallucination tests
rht = load_dataset("medhalt/med-halt", "reasoning_hallucination_test")

# Memory hallucination tests  
mht = load_dataset("medhalt/med-halt", "memory_hallucination_test")

Step 2 — Environment Setup

# Install uv (if not already installed)
curl -Lssf https://astral.sh/uv/install.sh | sh

# Install project dependencies
cd medhalt
uv sync

# Activate (optional — scripts use `uv run` automatically)
source .venv/bin/activate

Step 3 — Run Evaluation

# Inference (generate model responses)
uv run python medhalt/inference.py \
  --model_name orris \
  --dataset reasoning_hallucination_test \
  --output_file results/orris_rht.json

# Evaluation (score responses)
uv run python evaluate.py \
  --results_file results/orris_rht.json \
  --dataset reasoning_hallucination_test

What Med-HALT Measures (Scoring)

Combined Workflow Summary

1. Clone MedQA repo → download test.jsonl (1,273 Qs)
2. Write Orris API harness → run all 1,273 questions → compute % accuracy
3. Clone medhalt repo → uv sync → load datasets from HuggingFace
4. Write Orris adapter for inference.py → run RHT + MHT tests
5. Run evaluate.py → get hallucination scores
6. Compare all numbers against published GPT-4/GPT-4o baselines

One Practical Tip