ClinVerseGitHub

ClinVerse: A Realistic Multi-Agent Diagnostic Environment for Training and Evaluating Medical Agents

Fudan University&Shanghai Innovation Institute
150,667 clinical cases6 specialized roles5 evaluation metrics

Interaction Flowchart

Six roles, one coordinated clinical encounterThe doctor interacts through the environment controller with the observable patient simulator and laboratory physician. The laboratory physician retrieves evidence from the latent patient simulator or ClinLab. A diagnostic judge scores the trajectory. ActionInquiryReportCase dataTrajectoryScores Doctor agent Environmentcontroller Patient simulatorObservable Laboratoryphysician Latent patientsimulator Diagnostic judgeRule-based scores

The controller routes a question to the patient and returns the response.

One Prompt for Your Code Agent to Use ClinVerse

Use ClinVerse to evaluate or train my medical agent. Work through the following steps:

  • Environment: Follow README.md and environment.yml to install dependencies and configure model endpoints. Check CPU/GPU requirements. Download ClinLab-2B and connect it using clinlab.md.
  • Data: Use data.md for dataset access and preprocessing. Reuse prepared cases and create patient-disjoint train/test splits.
  • Evaluate: Read evaluation.md, implement interfaces.py using openai_agent.py, and run the documented evaluation commands with the generated local configuration.
  • Train: Follow training.md and memory_trainer.py. Integrate custom learning loops through clinical_env.py and runner.py. Evaluate frozen state on held-out cases.
  • Verify: Start with a small run, then scale up. Save configurations and trajectories; report ICR, TOC, ATC, DCA and DSA with reproducible commands. Ask me for missing access, paths or credentials.

Build with ClinVerse

Prepare cases. Connect your agent. Train and evaluate.

Download source

From clinical records to an agent environment.

One command normalizes records, builds examination catalogs, generates patient backgrounds, and extracts structured information units.

  1. Normalize admissions and measurements
  2. Build examination catalogs
  3. Generate conditioned patient backgrounds
  4. Extract information units with a configured model
  5. Assemble cases and patient-disjoint splits
Data processing guide
Prepare and split cases
python scripts/prepare_data.py \
  --mimic-dir ../dataset/mimic-iv-3.1 \
  --note-dir ../dataset/mimic-iv-note/2.2 \
  --synthea-root ../synthea \
  --setup-synthea \
  --extractor-provider extractor \
  --resume

python scripts/split_cases.py \
  --cases-dir data/cases --output data/splits

Outputs: data/cases/ and data/splits/
Use --limit 100 for a smaller run or --plan to inspect the stages.

Environment and setup Conda · Python 3.11 · JDK 17 · GPU configuration
Create the environment
conda env create -f environment.yml
conda activate clinverse
cp .env.example .env
set -a; source .env; set +a
Download and connect ClinLab
pip install -r requirements-gpu.txt
hf download SII-Sirry/ClinLab-2B --local-dir weights/ClinLab-2B
python scripts/configure_local_lab.py \
  --model weights/ClinLab-2B \
  --mapping data/normalized/mimic-iv-ext/lab_test_mapping.csv \
  --output config/local.yaml

Configure the patient, doctor, ClinLab, extractor and embedding endpoints in .env. Pretrained weights: ClinLab-2B.

Data preparation
Python 3.11 · JDK 17 · Git · extractor endpoint
64 GB+ host RAM recommended for the full dataset.
Evaluation & memory / skill training
CPU client with configured model endpoints.
Local model serving
Reference: 2 × 80 GB NVIDIA GPUs for Qwen3-32B. MemRL uses a configured embedding endpoint.
ClinLab LoRA training
Reference: 1 × 48 GB GPU · bfloat16 · 4,096 tokens · batch size 1.

GPU memory use depends on precision, context length and concurrency.

Clinical Simulation

Follow the agent’s decisions as clinical evidence comes into view.

Saved evaluation
Loading saved predictions…

Diagnostic trajectory

Loading…

Loading saved interactions…

Scroll to review played rounds
Saved trajectory

Saved ClinVerse agent interactions on MIMIC-IV-derived cases. Dialogue is translated into English; actions, results and scores are preserved. Prediction sources are verified from saved markers or ClinLab metadata.

Agent Benchmark

Compare evidence acquisition, examination coverage, and diagnostic agreement.

9 models100 cases · 5 runs

Model leaderboard

Paper · Table 2 · L0 static agents

9 modelsSorted by DSA
Evidence acquisitionDiagnostic accuracy
#Doctor agent
Mean ± standard deviation (%) · ↑ Higher is better100 held-out cases · 5 independent runs

Collecting more information does not necessarily lead to a better diagnosis.

Evidence selection and integration matter alongside coverage.
How are the five metrics defined?
ICR · Information Collection Rate
Weighted coverage of initially undisclosed patient-reportable information; key, supporting, and background units receive weights of 3, 2, and 1.
TOC · Test Order Coverage
Coverage of distinct, initially hidden case-supported laboratory, microbiology, and radiology findings. Repeated requests do not increase coverage.
ATC · Abnormal Test Coverage
Coverage of initially hidden abnormal laboratory and microbiology findings.
DCA / DSA · Diagnostic agreement
ICD-10 block match / three-character category match. Single-case scores are binary; the table reports aggregate accuracy.

Agent Evolution Benchmark

Compare adaptation strategies on the same Qwen3-32B backbone.

10 methods3 evolution levels

Evolution leaderboard

Qwen3-32B · Paper, Table 2

#MethodLevel
Mean ± standard deviation (%) · ↑ Higher is betterSame backbone · same five metrics
L1 Offline adaptationL2 Online self-evolutionL3 Recursive self-improvement

Evolution across backbones

Information acquisition vs. diagnosis

The table compares Qwen3-32B methods. The figure shows representative strategies across three backbones.
Open original figure (PDF)

More evidence does not always mean a more accurate diagnosis.

Information collection capability averages ICR, TOC, and ATC; diagnosis accuracy averages DCA and DSA. Full Information is a reference condition.

Cross-model adaptation

MemRL · Effect of trajectory source

Δ vs Base
TrajectoriesDoctor modelICR TOC ATC DCA DSA
Trajectory sourceQwen3-32B
5,000GPT-4o-mini 0.30 6.16 11.41 8.20 4.15
5,000GPT-5.6-Luna 1.86 1.94 0.65 1.31 1.49
5,000Qwen3-8B 0.86 2.55 2.65 0.97 3.77
5,000Qwen3-32B 0.40 2.18 6.10 5.36 4.37
500Qwen3-32B 0.55 1.60 3.79 0.52 4.44
Trajectory sourceGPT-5.6-Luna
500GPT-4o-mini 4.24 0.86 1.32 1.80 11.80
500GPT-5.6-Luna 0.56 2.94 1.41 4.73 4.36
500Qwen3-8B 8.19 8.71 14.07 0.80 8.31
500Qwen3-32B 6.72 0.87 5.27 5.79 6.37
Change from each doctor model’s base · percentage points
↑ Improvement↓ Decline

Evolution across backbones

Full-size training benefit figureOpen original PDF