ClinicalEye: Finding the Right Views
for 3D CT Visual Question Answering

A training-free VLM agent that, like a clinician, uses anatomy to decide where to look in a CT volume — and how to arrange what it shows.

Evaluated on DeepTumorVQA (multiple choice; recognition, measurement, visual reasoning, medical reasoning; 42 subtypes)
Interactive trajectories Results Code: link to be added

Overview

Task and teaser figure
Figure 1. Task and idea. Given a 3D CT and a multiple-choice question (here: “Is there a pancreatic ductal adenocarcinoma? A: No, B: Yes”), fixed uniformly sampled slices rarely show what the question needs. Our agent asks for anatomy-specific views (“where is the pancreatic head? show it next to the duodenum”), which are located with an anatomy graph built from automatic segmentation and a cited anatomy knowledge base, and rendered as a question-specific mosaic of zoomed slices with a locator. The agent then answers (A: No, correct).

Vision-language models (VLMs) struggle with 3D CT: the volume has far more slices than fit into a prompt, and the relevant anatomy (a pancreatic head, a vessel next to a lesion) is often small and not marked. Clinicians solve this with anatomy: they know which structures to look at, where they are, and which views to put side by side. ClinicalEye gives a VLM the same two resources — an anatomy graph computed from automatic segmentation and a cited anatomy knowledge base — and splits the work between a reader that reasons about the question and a navigator that only locates anatomy and composes views.

Key message. It is not only about finding the right slice, but about returning the right mosaic of views: the right planes, slice range, zoom, window and neighbouring structures, arranged together and explained.

Benchmark protocol

  • No benchmark metadata reaches our agent: the reader starts with only the question, the answer options and the names of the segmented structures — no organ, lesion or question-type labels, and no images.
  • All compared methods run on the same 903 questions (lite903, stratified over the 42 subtypes) with the same base models.
  • One sample per condition; differences are tested with paired exact McNemar tests.

Method

Method overview
Figure 2. Architecture. (a) Reader VLM: starts from the question, options and the names of the segmented structures / tool lesions; each turn it writes a conclusion, an answer with confidence, an evidence ledger and tool calls (overview, view, measure, compute, box, knowledge); its memory is a step log plus the evidence ledger. (b) Navigator: receives the reader's free-text request, locates the anatomy with the anatomy graph (and the anatomy knowledge base for unsegmented parts, marked approximate), renders candidate views with deterministic rules, checks its own renderings and adjusts (≤2 per request), then explains its choice; it never judges findings. Its memory holds the image sets already shown and a coverage table. (c) Anatomy graph from CT + automatic segmentation, plus the anatomy knowledge base. Bottom: the image set returned to the reader, with the explanation and code-computed per-panel content.

resource Anatomy graph

Built automatically from TotalSegmentator output.
  • Nodes: segmented structures, tool lesions and liver segments, each with slice range, volume and HU.
  • Edges: contact / closest distance between structures, plus the slice where it occurs.

resource Anatomy knowledge

Not every clinically relevant structure has a sufficiently good tool. A cited knowledge base gives location priors for such structures (e.g. SMA, celiac trunk, pancreatic head/tail, CBD): vertebral level, relations, appearance on axial CT, and how to locate them from segmented neighbours.

agent Reader VLM

Starts with no images. Each turn: step conclusion, current answer + confidence, evidence ledger, tool calls:
  • overview — turn 0, named organs, fixed rendering
  • view — a free-text need, sent to the navigator
  • measure / compute / box — code on the segmentation (volume, diameter, HU, contacts; arithmetic; mark a finding)
  • knowledge — benchmark clinical criteria lookup
Memory: step log + evidence ledger.

Trajectory viewer

Real episodes from the evaluation run, step by step. All texts are quoted from the logged episode files (long texts truncated with “...”); images are the renderings that were actually produced. S# = image sets delivered to the reader; R# = navigator candidate renderings.

Tip: use the ← / → keys to step through; click an image to enlarge.

Results

Accuracy on lite903 (903 questions, stratified over 42 subtypes), single sample per condition. Every column uses the same base model in each row.

+7.8 pts
ClinicalEye vs 3DMedAgent, GLM-5.3-Flash
p = 1e-4
+6.3 pts
ClinicalEye vs 3DMedAgent, Qwen3-VL-32B
p = 0.003
+9.7 pts
anatomy-graph summary in a single call (GLM, InternVL)
p < 0.001
+10.7 pts
ClinicalEye vs direct answer, Claude Sonnet 5
p = 1e-7
Base modelDeepTumorVQA3DMedAgentSingle callClinicalEye (ours)
direct answertool agentw/o anatomy graph+ anatomy graph
Qwen3-VL-32B0.4160.3930.4620.4010.4780.525
GLM-5.3-Flash0.4270.4830.4970.4700.5670.575
InternVL3.5-38B (FP8)0.4320.3960.4510.3900.4870.483
MedGemma-27B0.4040.3090.3930.3770.4140.433
Claude Sonnet 50.445———0.5130.553

Bold = best per row. — = not run. Accuracy on 903 multiple-choice questions; one sample per condition.

Paired comparisons (exact McNemar)

ComparisonQwen3-VL-32BGLM-5.3-FlashInternVL3.5-38BMedGemma-27BClaude Sonnet 5
ClinicalEye vs 3DMedAgent+6.3 (p=0.003)+7.8 (p=1e-4)+3.1 (n.s.)+4.0 (p=0.07)—
Single call: + anatomy graph vs without+7.7 (p<0.001)+9.7 (p<0.001)+9.7 (p<0.001)+3.7 (p=0.05)—
ClinicalEye vs single call + anatomy graph+4.7 (p=0.012)n.s.n.s.n.s.+4.0 (p=0.043)
ClinicalEye vs direct answer————+10.7 (p=1e-7)

Differences in accuracy points. 3DMedAgent with benchmark metadata (organ / lesion / question type) removed, as in our protocol: Qwen 0.453, GLM 0.484, InternVL 0.464, MedGemma 0.391.

Accuracy by question type
Figure 3. Accuracy by question type (recognition, measurement, visual reasoning, medical reasoning) for four open base models: direct answer (grey), benchmark tool agent (orange), 3DMedAgent (reddish purple) and ours (blue). Ours is highest on recognition for all four models; on measurement and medical reasoning 3DMedAgent is on par or slightly higher for some models.

Calibration

The agent's expected calibration error is lower than that of the single call for 4 of 5 base models (e.g. Qwen3-VL-32B: 0.46 → 0.33).

Structure probes: can VLMs localise structures that have no tool?

Probe tasks
Figure 4. Probe tasks. (a) In-plane: which of four marked points lies on the structure (e.g. the superior mesenteric artery)? (b) Out-of-plane: the structure is not in the shown slice — is it superior or inferior (and how far)? Three tracks: image only; + anatomy graph facts computed by code (e.g. which segmented structure each point lies in or near, how far segmented neighbours extend above/below; never the target itself); + a cited anatomy knowledge entry.
Probe results
Figure 5. Probe accuracy for three open models. Dashed lines: chance (0.25 in-plane, 0.5 out-of-plane).
Modelimage only+ anatomy graph+ anatomy knowledge
In-plane: which of 4 points lies on the structure (125 items, chance 0.25)
Qwen3-VL-32B0.590.820.84
InternVL3.5-38B0.370.770.78
MedGemma-27B0.540.720.85
Out-of-plane: superior or inferior to this slice (100 items, structures without a tool, chance 0.5)
Qwen3-VL-32B0.620.870.90
InternVL3.5-38B0.660.890.90
MedGemma-27B0.580.710.82

Distance estimation (3 bins: ≤3 cm / 3–6 cm / >6 cm) did not improve for any model.

Ablations (lite903)

Navigator (Qwen3-VL-32B)

Settingwith navigatorwithout navigatorΔwins / lossesp
Pure views0.5260.492+3.4+72 / −410.005
Missing tool0.4040.381+2.3+65 / −440.055

(a) To isolate the navigator's role in finding and explaining views, both variants are given pure views: the images contain only the CT, and all anatomical information reaches the reader as text. Removing the navigator costs 3.4 points (52.6 → 49.2, +72/−41, p=0.005).

(b) When the tool for the question's organ is missing (missing-tool setting), the agent with the navigator keeps more accuracy than without it (40.4 vs 38.1, +65/−44, p=0.055).

Replication on InternVL3.5 and MedGemma is running.

Memory and anatomy graph (Qwen3-VL-32B)

ConditionAccuracyΔ
Full ClinicalEye0.525—
w/o reader memory0.444−8.1 (p=5e-7)
w/o navigator memory0.519−0.6 (n.s.)
Single call w/o anatomy graph (vs single call, 0.478)0.401−7.8 (p<1e-4)

Single call without the anatomy graph: −7.8 (47.8 → 40.1, p<1e-4).

Missing-tool simulation (single call)

The question's organ is removed from the segmentation (and from the graph, vocabulary, measurements and tool lesions), not from the images. Even the single call, which relies least on the tool, drops 5.4 points with Qwen3-VL (47.8 → 42.4, p=0.007) and 9.9 points with GLM (56.7 → 46.8, p<1e-6).

Takeaways