Artificial intelligence (AI) has rapidly transitioned from a futuristic concept to an integral part of everyday life, with AI-driven algorithms now embedded in a broad range of technologies.¹ Modern AI systems rely on machine learning trained on vast datasets, enabling specialized tasks with high efficiency.² Notable examples include AlphaFold for protein prediction and OpenAI’s GPT models for natural language processing.^3–5 In medicine, AI shows promise in enhancing diagnostics and workflow automation, with models like BioBERT and LYNA demonstrating expert-level performance in specific tasks.^6–8

Subsequently, numerous models have been developed with medical imaging as their central focus. For instance, CheXNet was designed to detect pneumonia, and BioViL took it a step further by allowing multiple diagnoses on chest X-rays.^9,10

Among these, vision language models (VLMs) show considerable promise in analyzing biomedical images.¹¹ They are AI systems designed to process both images and text together. They combine a visual encoder (like a CNN or Vision Transformer) that processes images with a language model (such as BERT or GPT) that interprets text. Trained on large datasets of image-text pairs, VLMs can perform tasks such as image captioning, visual question answering, and multimodal search. Despite their promise, they can struggle with tasks outside their training data and require careful validation before clinical use.¹²

One of these recent innovations is Google’s Med-PaLM family of models, which includes MedGemma-4B, adapted for use in medical contexts.^13–15 MedGemma builds upon the foundation of general-purpose AI models, refining them through fine-tuning on medical imaging datasets. Recent studies have already shown that, fine-tuned for chest X-ray analysis, it ranked above all its competitors in terms of diagnostic accuracy.¹⁶

While such models are marketed as being able to solve complex diagnostic tasks, performance variability may occur in tasks underrepresented in training data, regardless of perceived task simplicity, such as classification or multimodal comprehension.^12,17 Therefore, we hypothesized that even medically specialized AI models, such as MedGemma-4B, may exhibit significant limitations when applied to tasks that, despite being clinically routine, may be underrepresented in their training data, and therefore cannot be assumed to be simple for the model. To explore this hypothesis, we submitted a series of publicly accessible radiology images to the MedGemma-4B model. Our goal was not to evaluate its diagnostic capability, but rather to assess its performance on image interpretation tasks that clinicians consider fundamental, but whose simplicity for a VLM cannot be assumed, as model performance may be strongly influenced by training data coverage rather than clinical task complexity.^18,19 This study aims to provide insights into the generalization capabilities and limitations of emerging medical AI tools. It raises important questions about their future readiness for integration into everyday clinical practice.

MedGemma’s classification performance was evaluated across six radiological image datasets, including modality, body part, orientation, contrast, image mode, and organ. The VLM model demonstrated an overall classification accuracy of 62% (95% CI: 58.12%–65.88%), with important performance variation across different diagnostic tasks. Overall model performances are summarized in Figure 2 and Tables 2 and 3.

The model was the most accurate in correctly identifying the image modality, achieving a correct classification rate of 97%, with only 3 errors and no imprecise outputs (95% CI: 90.99%–99.25%, chance level: 20%). Among the three errors, one involved misclassifying an axial abdominal T2-weighted MRI slice as a CT scan, while the remaining two errors resulted from mistaking angiography images for conventional X-rays. All remaining categories (X-ray, CT, and ultrasound) were classified with perfect accuracy (100%).

Performance was also high in body part identification, with an accuracy of 80% (95% CI: 70.93%–86.59%, chance level: 20%). While most predictions were correct, 11% of outputs were deemed imprecise, typically involving ambiguous or overlapping anatomical regions (e.g, “wrist” was predicted instead of “hand”). Additionally, 9% were incorrect, with the majority arising from musculoskeletal misclassifications (e.g., knee instead of hand).

By contrast, performance dropped in the remaining tasks. In identifying image orientation, the model achieved only 41% accuracy (95% CI: 32.06%–51.10%, chance level: 12.5%), with 59% of images misclassified and no imprecise cases. Similarly, contrast usage yielded an accuracy of just 32% (95% CI: 23.62%–41.84%, chance level: 12.5%). This category also had the highest rate of imprecise answers (17%). Classification by image mode yielded an accuracy of 48% (95% CI: 38.41%–57.75%, chance level: 10%). Nearly as many predictions were incorrect (42%) as correct, with 9% of outputs classified as imprecise.

Finally, in the organ classification task, MedGemma achieved a moderate accuracy of 74% (95% CI: 64.81%–81.75%, chance level: 14.3%). The model demonstrated proficiency in distinguishing among common organs but occasionally struggled with images that featured multiple structures or had subtle anatomical boundaries. Detailed subgroup performance metrics for each dataset are provided in Figure 3 and Extended data 4. Furthermore, examples of typical misclassifications across the different datasets are illustrated in Figure 4.

Sensitivity and specificity varied markedly across tasks. Modality and body part classification achieved near-perfect sensitivity (97.0% and 80.0%) and specificity (99.2% and 98.7%), whereas orientation and contrast tasks showed low sensitivity (41.0% and 32.0%) despite high specificity (91.9% and 91.0%). MCC values confirmed these disparities, with strong correlation for modality (96.2%) and body part (78.7%), but moderate-to-poor correlation for orientation (33.1%) and contrast (23.5%).

A chi-squared test revealed a statistically significant difference in classification accuracy across the six tasks (p < 0.001). Pairwise post hoc comparisons with Holm-adjusted p-values confirmed that the modality group exhibited significantly higher accuracy than all other groups (p < 0.001 for all comparisons). The body part group also outperformed orientation, contrast and image-mode (p < 0.001), but did not differ significantly from the organ group (p = 0.79). The organ group, in turn, showed significantly higher accuracy than orientation, contrast and image-mode groups (p < 0.01), and was statistically comparable to the body part group. No significant differences were observed between orientation, contrast and image-mode groups (all p > 0.12).

The reproducibility of MedGemma’s predictions across repeated runs was high, with a mean reproducibility of 91.6% across all tasks. Perfect consistency (100%) was observed for both the modality and organ classifications. Body part, orientation, and contrast tasks each showed 90% reproducibility, reflecting minor variability. The lowest reproducibility was noted in the image mode task (80%). All accuracy results reported in this study reflect the model’s first-run predictions. McNemar’s test showed no statistically significant difference between initial and repeated predictions (χ² = 0.80, p = 0.37), indicating that observed discrepancies likely occurred by chance and do not reflect systematic variability.

Our evaluation of MedGemma-4B revealed significant variability in its performance across fundamental radiological tasks. The model demonstrated near-perfect accuracy in identifying imaging modalities (97%) and strong performance in anatomical localization (80% for body parts), suggesting robust capabilities in high-level broad categorizations. However, its proficiency dropped markedly for tasks requiring finer-grained distinctions: orientation (41%), contrast (32%), and image modes (48%) tasks were notably weaker, although they remained above chance. Organ identification achieved moderate accuracy (74%) but is more debatable for the definition of ground truth compared to the other tasks, since multiple other organs can sometimes be partially visible in the images. This disparity in performances underscores a critical gap in the model’s generalization abilities, excelling in broad categorizations possibly better represented in its training data, but faltering in tasks whose fine-grained labels may be less consistently or comprehensively captured in large-scale medical datasets.

These findings may appear counterintuitive to clinicians with limited familiarity with AI development, who may intuitively assume that tasks routinely mastered by radiology trainees should be straightforwardly solved by an advanced VLM. However, this assumption conflates human-defined task simplicity with model performance, overlooking a critical factor: training data dependency. The model’s performance is likely influenced by the scope and quality of its training dataset. While this study focuses on MedGemma-4B, similar evaluations on other VLMs are needed to confirm generalizability, as performance strongly depends on training data coverage and model design.

Thus, the observed patterns raise important questions about the origins of these limitations. For example, the model’s high accuracy for modalities such as X-ray (a frequently encountered imaging type in training datasets) contrasts with its poor performance in underrepresented tasks (e.g., contrast phase identification, or image mode, which may lack consistent annotation in large-scale, curated radiology datasets). This suggests that gaps in training data diversity may partially explain the results; however, inherent challenges in visual ambiguity (e.g., distinguishing between arterial and venous contrast phases on static images) likely compound the issue.

As a newly released model in the AI ecosystem, MedGemma has yet to be extensively evaluated in independent studies.²⁵ Prior to MedGemma, several medical VLMs were developed, but none combined its three key advantages: open accessibility, training dataset scale/diversity, and multimodal versatility. For example, Med3DVLM tackles 3D medical vision-language alignment, achieving 61% R@1 retrieval on volumetric data, but was trained on a single M3D dataset, lacking the multi-source, multi-domain training foundation of models like Med-Gemma.²⁶ Meanwhile, MedVLM-R1 showed efficient reasoning capabilities for radiology QA through reinforcement learning (78.22% accuracy with just 2B parameters), but its training data remains partially restricted, and its application is limited to closed-set questions.²⁷ While many of these models report strong performance across various standardized, closed-benchmark systems, only a limited number have undergone clinical validation to date.²⁸

Multiple benchmarks have been proposed for medical VLM in the literature.^12,29–32 However, while such evaluations are essential for standardization and reproducibility, they also present notable limitations. Models tend to over-optimize for benchmark tasks, potentially at the expense of real-world clinical applicability. Smaller-scale, clinically grounded evaluations, although more limited, are necessary to bridge this gap; yet, they remain markedly underrepresented in the current academic landscape.³³

The current work focused on fundamental radiological classification tasks, areas that have remained relatively underexplored in the existing literature. For instance, a 2019 study developed a classifier achieving 94.8% accuracy in identifying imaging modality and anatomical region.³⁴ Similarly, a 2020 study developed a modality classification system with ResNet-50, achieving 87.9% accuracy across 31 imaging classes, demonstrating the potential of such systems to support clinical case retrieval from large-scale archives.³⁵ Furthermore, a 2021 study developed a model capable of classifying both imaging modality and anatomical location with over 99.5% accuracy on converted JPEG images, demonstrating classification feasibility even when DICOM metadata is unavailable.³⁶ While these results are promising, they remain insufficient given the vast number of models yet to be evaluated and the immense diversity of potential clinical tasks in radiology.

The findings of this study highlight both the potential of MedGemma-4B and other possible future medical VLMs in real-world radiological workflows. The model’s high accuracy in modality and body part classification suggests its utility for automated pre-sorting of medical images, streamlining the organization of large-scale PACS (Picture Archiving and Communication Systems) databases. Future iterations or other models could achieve robust performance across finer-grained tasks. For instance, reliable contrast phase recognition (e.g., arterial vs. venous) could automate protocol verification, reducing human error in dynamic imaging studies. Similarly, precise MRI sequence identification might enable real-time quality control, flagging mislabeled sequences or suboptimal acquisitions before interpretation. Beyond classification, advanced models could standardize ambiguous annotations (e.g., “oblique” projections or “portal phase” timing) by learning consensus patterns from multi-institutional datasets, bridging variability in radiological lexicons. Such capabilities would unlock applications like intelligent PACS structuring, where AI pre-annotates studies with metadata (e.g., “contrast-enhanced CT, arterial phase, axial liver”), accelerating retrieval and triage. Moreover, the seamless integration of multimodal reasoning, which combines image features with clinical context from reports, could support the creation of automated preliminary summaries, thereby streamlining radiologist workflows. While current models remain imperfect, their trajectory points toward a future where AI handles routine perceptual tasks, allowing clinicians to focus on complex diagnostics and patient care.

This study has several limitations that should be considered when interpreting the results. First, the dataset size for specific subcategories was relatively small, which may limit the statistical power to detect subtle performance differences. While our image source provided diverse, multicentric data, the images may not fully represent the variability seen in single-institution datasets, where acquisition protocols and artifacts can differ. Additionally, model responses were manually categorized (correct, incorrect, or imprecise) by expert radiologists, introducing potential subjectivity in borderline cases. While the introduction of the imprecise category was intended to reduce forced binary judgments, it also creates an additional boundary where subjectivity may arise, and future studies could benefit from formal inter-rater reliability assessment. Furthermore, the one-vs-rest aggregation strategy used to compute task-level metrics inherently expands the true negative pool, which can inflate specificity and NPV. The MCC was included to counterbalance this, as it remains robust to such class imbalance. Furthermore, all evaluations were conducted on single static images, consistent with MedGemma’s design for image-level interpretation. However, modalities such as ultrasound or fluoroscopy are typically assessed dynamically, so this may not fully represent real-world conditions. Finally, the evaluation employed a fixed prompt structure without exploring variations in phrasing or context, which may have influenced MedGemma’s performance. Future studies could benefit from larger, prospectively collected datasets and systematic prompt optimization to better assess real-world applicability.

MedGemma-4B demonstrated strong performance on certain foundational radiological tasks, such as modality and anatomical region identification, while exhibiting variable or lower performance on other tasks considered clinically straightforward, such as image contrast or orientation classification. This pattern indicates that tasks perceived as simple by clinicians are not necessarily easy for AI models, whose performance may be strongly influenced by the coverage and quality of their training data rather than human-perceived task simplicity.

This study utilised publicly available anonymised medical images from Radiopaedia.org, which are distributed under the Creative Commons Non-Commercial Attribution CC-NC-BY-SA 3.0 license. No patient identifying information was included. As no human participants were directly involved and no identifiable data were used, formal ethical approval was not required.