ClinVerse: A Realistic Multi-Agent Diagnostic Environment for Training and Evaluating Medical Agents
Interaction Flowchart
The controller routes a question to the patient and returns the response.
One Prompt for Your Code Agent to Use ClinVerse
Use ClinVerse to evaluate or train my medical agent. Work through the following steps:
- Environment: Follow README.md and environment.yml to install dependencies and configure model endpoints. Check CPU/GPU requirements. Download ClinLab-2B and connect it using clinlab.md.
- Data: Use data.md for dataset access and preprocessing. Reuse prepared cases and create patient-disjoint train/test splits.
- Evaluate: Read evaluation.md, implement interfaces.py using openai_agent.py, and run the documented evaluation commands with the generated local configuration.
- Train: Follow training.md and memory_trainer.py. Integrate custom learning loops through clinical_env.py and runner.py. Evaluate frozen state on held-out cases.
- Verify: Start with a small run, then scale up. Save configurations and trajectories; report ICR, TOC, ATC, DCA and DSA with reproducible commands. Ask me for missing access, paths or credentials.
Build with ClinVerse
Prepare cases. Connect your agent. Train and evaluate.
From clinical records to an agent environment.
One command normalizes records, builds examination catalogs, generates patient backgrounds, and extracts structured information units.
- Normalize admissions and measurements
- Build examination catalogs
- Generate conditioned patient backgrounds
- Extract information units with a configured model
- Assemble cases and patient-disjoint splits
python scripts/prepare_data.py \
--mimic-dir ../dataset/mimic-iv-3.1 \
--note-dir ../dataset/mimic-iv-note/2.2 \
--synthea-root ../synthea \
--setup-synthea \
--extractor-provider extractor \
--resume
python scripts/split_cases.py \
--cases-dir data/cases --output data/splitsOutputs: data/cases/ and data/splits/
Use --limit 100 for a smaller run or --plan to inspect the stages.
Your agent. A shared clinical interface.
Receive the initial presentation and tool schemas, choose an action, and observe the response. Scores are returned after the encounter. Use ClinLab-2B with the checkpoint setup.
- reset(context)
- Initial presentation, available tools and turn budget.
- act() → Action
- Action name and structured arguments.
- observe(text)
- Patient reply, examination result or report.
Reference agent: examples/openai_agent.py
Custom agent: --agent my_package:MyAgent
python scripts/run_agent.py \
--config config/local.yaml \
--agent examples.openai_agent:OpenAIAgent \
--case-ids-file data/splits/test.txt \
--n 100 \
--output outputs/evaluation/referenceclass MyAgent:
def __init__(self, provider, language="en"): ...
def reset(self, context): ...
def act(self) -> Action: ...
def observe(self, observation: str): ...Per-case trajectories and scores, plus an aggregate summary.
ICR ↑ · TOC ↑ · ATC ↑ · DCA ↑ · DSA ↑
Learn from trajectories and feedback.
Retrieve memories by semantic similarity and utility. Update the trainer from completed trajectories and scores, then save the learned state. Connect ClinLab-2B using the checkpoint setup.
- context(episode) → str
- Retrieve memory or guidance for the next encounter.
- update(record)
- Learn from the trajectory and evaluation feedback.
- save(path)
- Persist a memory, skill bank or checkpoint.
MemRLSemantic recall followed by utility-based selection.
OfflineBuild a memory bank from training trajectories.
OnlineUpdate memory utilities and add completed experiences.
python scripts/run_agent.py \
--config config/local.yaml \
--agent examples.openai_agent:OpenAIAgent \
--trainer examples.memory_trainer:MemoryTrainer \
--trainer-state outputs/training/memory.json \
--case-ids-file data/splits/train.txt \
--n 100 \
--output outputs/training/episodesEvaluate the saved state
python scripts/run_agent.py \
--config config/local.yaml \
--agent examples.openai_agent:OpenAIAgent \
--memory-state outputs/training/memory.json \
--case-ids-file data/splits/test.txt \
--n 100 --output outputs/evaluation/trainedCustom trainer interface
class MyTrainer:
def __init__(self, state_path=None, *, config): ...
def context(self, episode) -> str: ...
def update(self, record): ...
def save(self, path): ...Environment and setup Conda · Python 3.11 · JDK 17 · GPU configuration
conda env create -f environment.yml
conda activate clinverse
cp .env.example .env
set -a; source .env; set +apip install -r requirements-gpu.txt
hf download SII-Sirry/ClinLab-2B --local-dir weights/ClinLab-2B
python scripts/configure_local_lab.py \
--model weights/ClinLab-2B \
--mapping data/normalized/mimic-iv-ext/lab_test_mapping.csv \
--output config/local.yamlConfigure the patient, doctor, ClinLab, extractor and embedding endpoints in .env. Pretrained weights: ClinLab-2B.
- Data preparation
- Python 3.11 · JDK 17 · Git · extractor endpoint
64 GB+ host RAM recommended for the full dataset. - Evaluation & memory / skill training
- CPU client with configured model endpoints.
- Local model serving
- Reference: 2 × 80 GB NVIDIA GPUs for Qwen3-32B. MemRL uses a configured embedding endpoint.
- ClinLab LoRA training
- Reference: 1 × 48 GB GPU · bfloat16 · 4,096 tokens · batch size 1.
GPU memory use depends on precision, context length and concurrency.
Clinical Simulation
Follow the agent’s decisions as clinical evidence comes into view.
Diagnostic trajectory
Loading…Loading saved interactions…
Saved ClinVerse agent interactions on MIMIC-IV-derived cases. Dialogue is translated into English; actions, results and scores are preserved. Prediction sources are verified from saved markers or ClinLab metadata.
Agent Benchmark
Compare evidence acquisition, examination coverage, and diagnostic agreement.
Model leaderboard
Paper · Table 2 · L0 static agents
| Evidence acquisition | Diagnostic accuracy | |||||
|---|---|---|---|---|---|---|
| # | Doctor agent | |||||
Collecting more information does not necessarily lead to a better diagnosis.
Evidence selection and integration matter alongside coverage.How are the five metrics defined?
- ICR · Information Collection Rate
- Weighted coverage of initially undisclosed patient-reportable information; key, supporting, and background units receive weights of 3, 2, and 1.
- TOC · Test Order Coverage
- Coverage of distinct, initially hidden case-supported laboratory, microbiology, and radiology findings. Repeated requests do not increase coverage.
- ATC · Abnormal Test Coverage
- Coverage of initially hidden abnormal laboratory and microbiology findings.
- DCA / DSA · Diagnostic agreement
- ICD-10 block match / three-character category match. Single-case scores are binary; the table reports aggregate accuracy.
Agent Evolution Benchmark
Compare adaptation strategies on the same Qwen3-32B backbone.
Evolution leaderboard
Qwen3-32B · Paper, Table 2
| # | Method | Level |
|---|
Evolution across backbones
Information acquisition vs. diagnosis
More evidence does not always mean a more accurate diagnosis.
Information collection capability averages ICR, TOC, and ATC; diagnosis accuracy averages DCA and DSA. Full Information is a reference condition.Cross-model adaptation
MemRL · Effect of trajectory source
| Trajectories | Doctor model | ICR | TOC | ATC | DCA | DSA |
|---|---|---|---|---|---|---|
Trajectory source | ||||||
| 5,000 | GPT-4o-mini | 0.30 | 6.16 | 11.41 | 8.20 | 4.15 |
| 5,000 | GPT-5.6-Luna | 1.86 | 1.94 | 0.65 | 1.31 | 1.49 |
| 5,000 | Qwen3-8B | 0.86 | 2.55 | 2.65 | 0.97 | 3.77 |
| 5,000 | Qwen3-32B | 0.40 | 2.18 | 6.10 | 5.36 | 4.37 |
| 500 | Qwen3-32B | 0.55 | 1.60 | 3.79 | 0.52 | 4.44 |
Trajectory source | ||||||
| 500 | GPT-4o-mini | 4.24 | 0.86 | 1.32 | 1.80 | 11.80 |
| 500 | GPT-5.6-Luna | 0.56 | 2.94 | 1.41 | 4.73 | 4.36 |
| 500 | Qwen3-8B | 8.19 | 8.71 | 14.07 | 0.80 | 8.31 |
| 500 | Qwen3-32B | 6.72 | 0.87 | 5.27 | 5.79 | 6.37 |