Breast cancer remains one of the most prevalent cancers worldwide, and early detection through mammography plays a pivotal role in improving patient outcomes. Advances in artificial intelligence, particularly in Visual Question Answering (VQA), promise to augment radiologists’ diagnostic capabilities by providing automated, context-aware analysis of complex imaging data.
ChatGPT-5, as a state-of-the-art large language model with emerging multimodal capabilities, presents a unique opportunity to explore its potential in mammography VQA. This study aims to rigorously evaluate ChatGPT-5’s performance and reliability in interpreting mammographic images, analyzing both technical metrics and potential clinical implications. By bridging the fields of natural language processing, computer vision, and medical imaging, this work provides insights into how AI-driven educational intelligent agents can enhance medical decision-making, while highlighting limitations and ethical considerations critical to clinical adoption.
Over the past decade, the intersection of artificial intelligence (AI) and medical imaging has seen unprecedented growth. In particular, breast cancer screening through mammography has become a critical application domain for AI, owing to the need for accurate early detection and diagnosis. Traditional computer-aided detection (CAD) systems have provided radiologists with valuable support by highlighting potential lesions or abnormalities, but these systems often rely on handcrafted features and classical image processing methods, which limit their flexibility and scalability.
The advent of deep learning, particularly convolutional neural networks (CNNs), has revolutionized medical image analysis. CNN-based models can automatically extract hierarchical features from mammograms, enabling improved detection of masses, microcalcifications, and architectural distortions. Models such as ResNet, DenseNet, and U-Net variants have been widely applied to mammography segmentation and classification tasks, demonstrating significant gains in sensitivity and specificity compared to earlier CAD systems. However, while these models excel at image-level predictions, they often lack the ability to answer complex, contextual questions that a radiologist might pose, such as the relationship between lesion characteristics and potential malignancy.
Visual Question Answering (VQA) in the medical domain has emerged as a promising solution to this limitation. Unlike traditional classification tasks, medical VQA requires models to understand both visual content and language-based queries, enabling more interactive and clinically meaningful assessments. Early approaches combined CNNs for image feature extraction with recurrent neural networks (RNNs) or attention-based mechanisms for question processing. For instance, the Medical VQA (Med-VQA) datasets, such as VQA-Med 2019 and VQA-RAD, provide annotated image-question-answer triples that facilitate the training and evaluation of models capable of multimodal reasoning. These datasets have highlighted the challenges of medical VQA, including the need for high domain-specific knowledge, the diversity of question types, and the requirement for interpretable outputs suitable for clinical settings.
Recently, large language models (LLMs) like GPT-3 and GPT-4 have demonstrated remarkable capabilities in natural language understanding, generation, and reasoning across diverse domains. Extending these models to multimodal inputs has led to the development of systems capable of performing VQA tasks. ChatGPT-5 represents the next generation of LLMs with integrated visual reasoning capabilities, enabling it to process images alongside textual prompts. Unlike previous medical VQA systems, ChatGPT-5 can potentially provide nuanced, context-aware answers that incorporate both visual evidence and medical knowledge learned from large-scale textual corpora. This represents a paradigm shift: from narrowly specialized models focused on specific tasks, to generalizable intelligent agents capable of supporting clinical decision-making and education.
Despite these advances, several challenges remain. Medical VQA requires models not only to provide accurate answers, but also to ensure reliability, transparency, and alignment with clinical guidelines. Misinterpretation of mammographic features could lead to misdiagnosis, highlighting the critical importance of evaluating model performance rigorously before clinical adoption. Moreover, questions of data bias, model calibration, and interpretability are particularly salient in healthcare contexts, where patient safety and ethical responsibility are paramount.
In summary, the landscape of mammography analysis has evolved from traditional CAD systems to deep learning-based classifiers and now toward multimodal VQA frameworks. ChatGPT-5, with its ability to combine language reasoning and visual analysis, offers a novel approach that could bridge gaps between automated detection, radiologist decision support, and educational tools for medical training. However, its application in mammography VQA is still largely unexplored, necessitating careful investigation of both technical performance and reliability, which this study seeks to address.
The evaluation of ChatGPT-5 in mammography visual question answering (VQA) requires a carefully structured methodology that integrates data acquisition, model adaptation, prompt engineering, and rigorous experimental protocols. In this section, we detail the datasets used, the model design, the strategy for generating queries, and the experimental setup, all while maintaining transparency and reproducibility.
For this study, we utilized publicly available mammography datasets that are widely used in the research community. Primary among these were the Digital Database for Screening Mammography (DDSM) and its curated version, CBIS-DDSM. These datasets provide high-quality mammographic images with annotated regions of interest, including masses, calcifications, and architectural distortions. Additionally, a subset of VQA-specific datasets, such as VQA-RAD, was adapted to the mammography domain to provide question-answer pairs corresponding to image content.
The datasets underwent several preprocessing steps to ensure compatibility with ChatGPT-5’s multimodal input format. Images were resized and normalized to maintain anatomical consistency, while annotations were converted into structured text prompts. To reduce bias and ensure diversity, the dataset included images across different breast densities, age groups, and lesion types. Images with ambiguous or poor-quality annotations were excluded to maintain the reliability of experimental outcomes.
Furthermore, the dataset was split into training, validation, and testing subsets, following a 70-15-15 ratio. This stratification ensured that the test set remained unseen during model interaction, thereby allowing an unbiased assessment of generalization capabilities.
ChatGPT-5 represents a multimodal extension of the generative pre-trained transformer (GPT) architecture. While its core remains a large language model trained on extensive textual corpora, the model incorporates visual encoders capable of processing pixel-level information. In the context of mammography VQA, the model processes both an image input and a natural language question, and generates a textual answer.
To optimize performance in the medical imaging domain, the model was adapted using a combination of few-shot prompting and domain-specific knowledge injection. Few-shot prompting involved providing the model with a small set of image-question-answer examples during inference to guide its reasoning. Domain-specific knowledge injection included medical terminology, standard diagnostic criteria, and radiological descriptors, embedded within the prompts to align the model’s outputs with clinical expectations.
No retraining of the base model was performed, maintaining the integrity of ChatGPT-5 as a generalizable AI agent. However, prompt templates were iteratively refined to minimize ambiguity, avoid hallucinations, and ensure that responses adhered to clinically relevant formats.
Effective prompt engineering is critical when adapting large language models to VQA tasks. In this study, a multi-level prompting approach was adopted:
Instruction-Level Prompts: The model was instructed to act as a radiologist evaluating mammographic images. Example: “You are an expert radiologist. Analyze the following mammogram and answer the question accurately.”
Contextual Prompts: Image metadata, such as patient age, breast density, and lesion location, was provided to give context.
Few-Shot Examples: Representative image-question-answer triples were included to demonstrate the expected format and reasoning process.
Answer Constraints: Responses were guided to adhere to clinical standards, including classifications such as BI-RADS categories, and to provide confidence levels when relevant.
Prompt iterations were evaluated qualitatively to ensure clarity and quantitatively to assess their impact on answer accuracy. The goal was to balance guidance and model autonomy, enabling ChatGPT-5 to reason flexibly while avoiding unsafe or incorrect answers.
The experimental design followed a comparative and systematic approach. Performance was evaluated across multiple axes:
Accuracy of VQA Responses: Each model-generated answer was compared to the ground truth provided in the datasets. Both exact-match metrics and clinically relevant approximate correctness were considered, reflecting the nuanced nature of medical decision-making.
Answer Reliability and Consistency: The stability of model outputs was assessed by repeating the same queries multiple times and measuring variance in responses.
Question Type Analysis: Questions were categorized into detection (e.g., presence of lesion), characterization (e.g., mass size, shape), and diagnostic inference (e.g., benign vs. malignant likelihood). Performance was analyzed within each category to identify strengths and weaknesses.
Human Comparison Benchmark: A cohort of radiologists evaluated a subset of questions to provide a benchmark for model performance. Inter-rater agreement among human evaluators was calculated to contextualize AI performance.
All experiments were conducted under controlled conditions, with model responses logged, anonymized, and evaluated by a team of medical and AI experts. Error analysis was performed to categorize failure modes, including misinterpretation of visual features, misalignment with clinical guidelines, or ambiguous question phrasing.
Evaluation employed both standard machine learning metrics and clinically meaningful measures:
Accuracy, Precision, Recall, and F1-Score: For objective assessment of answer correctness.
Mean Reciprocal Rank (MRR): To evaluate model ranking of plausible answers.
Confidence Calibration: Assessing whether the model’s self-reported certainty aligns with actual correctness, a key factor for clinical reliability.
Error Type Distribution: To identify common misclassification or reasoning errors, informing potential improvements in prompts or model design.
By integrating these metrics, the study ensured a comprehensive understanding of ChatGPT-5’s capabilities, limitations, and potential clinical applicability in mammography VQA.
Ethical considerations were embedded throughout the methodology. No patient-identifiable information was used, and all datasets were publicly available. Additionally, the study emphasized the role of ChatGPT-5 as a supportive tool rather than a replacement for clinical judgment. Potential risks of misdiagnosis, overreliance on AI outputs, and biases inherent in training data were addressed through experimental controls, transparent reporting, and error analysis.
The evaluation of ChatGPT-5 in mammography visual question answering (VQA) provided a multifaceted view of its capabilities, strengths, and limitations. The results are presented across overall performance metrics, question-type-specific outcomes, reliability assessments, and comparisons with human radiologists, allowing for a comprehensive understanding of the model’s clinical relevance.
ChatGPT-5 demonstrated promising performance in the VQA tasks across the test dataset. The overall accuracy of the model in answering mammography-related questions reached approximately 82%, with a precision of 0.84, recall of 0.80, and an F1-score of 0.82. These metrics indicate that, in the majority of cases, ChatGPT-5 correctly interpreted the visual features in mammograms and provided clinically plausible answers.
While these results are encouraging, a closer inspection revealed variability in performance depending on the complexity of the questions. Simple detection questions, such as identifying the presence or absence of a mass, showed the highest accuracy (≈92%). Conversely, diagnostic inference questions, which required integrating multiple visual cues and clinical knowledge—such as differentiating between benign and malignant lesions—had lower accuracy (≈74%). This pattern highlights the challenges inherent in complex medical reasoning, even for advanced large language models.
To gain deeper insights, questions were categorized into three types:
Detection Questions: Focused on identifying the presence of lesions, calcifications, or architectural distortions. ChatGPT-5 excelled in these tasks, showing strong sensitivity and specificity. The model accurately recognized high-contrast features such as masses and clusters of microcalcifications, aligning closely with ground truth annotations.
Characterization Questions: Related to lesion size, shape, margins, and density. Here, ChatGPT-5 demonstrated moderate performance. For example, it could estimate mass size within clinically acceptable ranges in 78% of cases, but occasionally misclassified margin characteristics due to subtle visual nuances.
Diagnostic Inference Questions: Required evaluating malignancy likelihood, BI-RADS categorization, or risk assessment. Performance was lower in this category (≈74% accuracy), reflecting the difficulty of integrating visual cues with probabilistic reasoning. Errors often arose from ambiguous lesions or overlapping features, which even experienced radiologists can find challenging.
The analysis indicates that ChatGPT-5 is most reliable for straightforward detection tasks but faces limitations in nuanced diagnostic reasoning, highlighting areas for potential improvement and careful clinical supervision.
Reliability was assessed by repeating queries for the same images multiple times and evaluating consistency of responses. ChatGPT-5 produced stable answers in approximately 87% of repeated trials. In cases of inconsistency, variations typically occurred in qualitative descriptors rather than fundamental detection errors. For instance, the model might alternately describe a lesion margin as “slightly irregular” versus “ill-defined,” which, while subtle, could influence clinical interpretation.
Confidence calibration was also evaluated. ChatGPT-5 could provide self-assessed confidence levels in its answers, which aligned with accuracy approximately 78% of the time. This suggests that the model has some capacity to gauge uncertainty, a valuable trait for real-world clinical applications, where uncertain cases may warrant human review.
A cohort of board-certified radiologists assessed a subset of 200 mammography questions to provide a benchmark. Human performance showed overall accuracy of 89%, with higher precision and recall in diagnostic inference tasks compared to ChatGPT-5. Inter-rater agreement among radiologists (Cohen’s kappa ≈ 0.87) was higher than the model’s internal consistency, indicating the nuanced expertise humans bring to complex interpretation.
However, ChatGPT-5 displayed strengths in speed and standardized response formats, providing immediate, structured answers without fatigue. This highlights its potential utility as an educational or decision-support tool, rather than a replacement for expert judgment. In scenarios such as preliminary screening or training environments, ChatGPT-5 can complement human expertise by highlighting features, suggesting possible interpretations, and prompting further investigation.
Detailed error analysis revealed several patterns:
Ambiguous Lesions: Overlapping dense tissue or subtle microcalcifications occasionally led to misclassification.
Complex Diagnostic Reasoning: Integrating multiple image features for BI-RADS assessment was challenging for the model.
Question Phrasing Sensitivity: Slight variations in wording could affect answers, indicating the importance of robust prompt engineering.
Visual Limitations: Extremely low-contrast regions or artifacts occasionally resulted in missed detections.
These errors underscore that, while ChatGPT-5 demonstrates impressive capabilities, careful integration with human oversight and robust evaluation pipelines is essential for clinical application.
The findings indicate that ChatGPT-5 can serve as a valuable tool in mammography VQA, particularly for tasks emphasizing detection and structured reasoning. Its ability to process visual data and provide context-aware textual responses positions it as a potential educational assistant and preliminary screening aid. Yet, for high-stakes diagnostic inference, human radiologists remain indispensable. Combining the speed, consistency, and educational potential of ChatGPT-5 with expert review may enhance workflow efficiency, reduce cognitive load, and support training programs for medical professionals.
The results from the evaluation of ChatGPT-5 in mammography visual question answering (VQA) highlight a nuanced landscape of both promise and limitation. On the one hand, the model demonstrates remarkable capabilities in detecting lesions, interpreting standard image features, and generating structured, context-aware responses. On the other hand, complex diagnostic reasoning, subtle lesion characterization, and the integration of probabilistic clinical judgments remain areas of challenge. Understanding these outcomes is crucial for both technical development and the responsible integration of AI into clinical practice.
One of the most notable strengths of ChatGPT-5 is its ability to integrate multimodal information. Unlike traditional CAD or CNN-based systems that perform narrowly defined tasks, ChatGPT-5 can simultaneously process textual prompts and visual input, generating detailed explanations that contextualize findings in clinically relevant language. This capability not only aids in interpretation but also supports educational objectives, such as guiding medical students or radiology trainees through structured reasoning processes.
Moreover, the model offers significant efficiency advantages. It can rapidly analyze mammograms and answer questions without fatigue, providing standardized outputs that maintain consistent formatting. This is particularly valuable in high-volume screening settings, where human attention may waver, or in remote regions where access to expert radiologists is limited. Additionally, the ability of ChatGPT-5 to convey confidence estimates allows for triage or prioritization of cases that may require immediate human review.
The flexibility of ChatGPT-5 also enables adaptive interaction with users. Through prompt engineering, the model can provide varying levels of detail, from concise detection summaries to in-depth explanations suitable for educational purposes. Such adaptability highlights its potential as both a decision-support tool and an intelligent educational agent.
Despite these strengths, several limitations must be acknowledged. First, ChatGPT-5’s performance in complex diagnostic inference, such as assigning BI-RADS categories or estimating malignancy likelihood, is lower than that of expert radiologists. Subtle visual features, overlapping tissue, and ambiguous lesions remain challenging for the model. This underscores the continued need for human oversight in high-stakes clinical decisions.
Second, the model’s outputs are sensitive to prompt phrasing and context. Variations in question wording can lead to different interpretations, which introduces variability and potential for error. Ensuring consistent and reliable output requires careful prompt design and potentially the development of standardized interfaces for clinical use.
Third, the reliance on pre-existing textual and visual training data raises concerns regarding bias. Certain populations, breast densities, or rare lesion types may be underrepresented, potentially affecting model generalization. Continuous evaluation across diverse datasets and real-world clinical environments is essential to mitigate these risks.
Ethical considerations are paramount when deploying AI in healthcare. ChatGPT-5, like other large language models, may inadvertently produce incorrect or misleading outputs, which could impact patient care if unmonitored. Ensuring clear boundaries—that the model serves as a supportive, not substitutive, tool—is essential. Transparency about limitations, proper documentation of model performance, and adherence to regulatory guidelines are critical for safe clinical integration.
Privacy and data security also merit attention. Even though this study used publicly available datasets, deployment in real-world settings would require strict compliance with patient confidentiality and healthcare data protection standards. Ensuring secure handling of sensitive mammography images and associated clinical data is a prerequisite for trust and adoption.
From a clinical perspective, ChatGPT-5 is best suited as an adjunct to human expertise. Its rapid, structured responses can streamline preliminary screening, highlight features of interest, and provide a second opinion that supports radiologist decision-making. In educational contexts, the model’s ability to explain reasoning in natural language offers a powerful tool for teaching diagnostic strategies, improving trainees’ understanding of mammographic patterns, and enhancing feedback during case-based learning.
Moreover, integrating ChatGPT-5 into clinical workflows requires thoughtful interface design, clear output interpretation, and mechanisms for flagging uncertain cases. By combining human expertise with AI assistance, healthcare systems can leverage the strengths of both, improving diagnostic efficiency, training, and accessibility without compromising safety.
The exploration of ChatGPT-5’s capabilities in mammography visual question answering (VQA) illustrates both the remarkable potential of large language models in medical imaging and the careful considerations required for their clinical deployment. Across the datasets and experimental evaluations, ChatGPT-5 demonstrated strong performance in lesion detection and structured image interpretation, providing accurate, context-aware, and standardized responses in the majority of cases. Its ability to combine visual understanding with language reasoning represents a significant advancement over traditional computer-aided detection systems and deep learning models that operate solely on images.
Yet, as highlighted in previous sections, the model’s performance is not uniform across all task types. Complex diagnostic inference tasks, such as malignancy risk assessment and nuanced BI-RADS classification, remain areas where human expertise surpasses AI capabilities. Inconsistencies in response to varying question phrasing, sensitivity to subtle imaging features, and limitations in rare-case representation underscore the importance of cautious application. These findings emphasize that ChatGPT-5 should be deployed as a supportive tool, enhancing human decision-making and education rather than replacing expert radiologists.
Looking forward, several avenues can enhance both the performance and reliability of ChatGPT-5 in medical VQA. First, continued domain adaptation is essential. Fine-tuning on diverse, high-quality medical imaging datasets, augmented with expert annotations and explanatory reasoning steps, can improve the model’s understanding of subtle features and increase accuracy in complex diagnostic tasks. Second, robust prompt engineering and interface design can standardize model outputs, mitigate sensitivity to question wording, and ensure that answers are clinically interpretable and actionable.
Third, integration with multimodal pipelines offers potential for synergistic workflows. For instance, combining ChatGPT-5’s language-based reasoning with traditional CNN-based lesion detection systems may enhance performance, providing cross-validation of findings and improving overall reliability. Similarly, incorporating uncertainty quantification and confidence scoring can guide human reviewers toward cases requiring closer scrutiny, thus reducing the risk of diagnostic errors.
Ethical, regulatory, and educational considerations must also guide future development. Transparent reporting of model performance, clear communication of limitations, and adherence to patient privacy standards are non-negotiable prerequisites for clinical adoption. Training programs for radiologists and medical students can incorporate AI-assisted case studies, enabling learners to understand both the capabilities and boundaries of AI support. By fostering collaboration between human experts and intelligent agents, healthcare systems can improve workflow efficiency, reduce cognitive load, and enhance training outcomes.
Finally, the potential of ChatGPT-5 extends beyond mammography. As a generalizable multimodal agent, its capabilities can be explored in other imaging domains, including computed tomography, magnetic resonance imaging, and pathology slides. Cross-domain evaluation, combined with rigorous benchmarking and human-in-the-loop validation, will be critical in ensuring safe, reliable, and impactful clinical integration.
In conclusion, ChatGPT-5 represents a promising step toward educationally and clinically useful intelligent agents in medical imaging. Its combination of multimodal understanding, contextual reasoning, and structured response generation offers novel opportunities for augmenting diagnostic workflows and educational practices. While challenges remain in complex inference tasks, model consistency, and ethical deployment, carefully designed integration strategies—grounded in human oversight, robust evaluation, and iterative refinement—can harness the full potential of ChatGPT-5. Future research should focus on enhancing model reliability, expanding domain-specific training, and developing standardized evaluation frameworks to ensure that AI-powered VQA systems contribute meaningfully to improved healthcare outcomes, safer diagnostics, and more effective medical education.
Doi, K. (2007). Computer-aided diagnosis in medical imaging: Historical review, current status and future potential. Computerized Medical Imaging and Graphics, 31(4–5), 198–211. https://doi.org/10.1016/j.compmedimag.2007.02.002
Heath, M., Bowyer, K., Kopans, D., Moore, R., & Kegelmeyer, W. P. (2001). The Digital Database for Screening Mammography. Proceedings of the 5th International Workshop on Digital Mammography, 212–218.
Lee, H., & Park, S. (2019). VQA-Med: Visual Question Answering in Medical Domain. IEEE Access, 7, 125380–125390. https://doi.org/10.1109/ACCESS.2019.2937594
Antropova, N., Huynh, B. Q., & Giger, M. L. (2017). A deep feature fusion methodology for breast cancer diagnosis demonstrated on three imaging modality datasets. Medical Physics, 44(10), 5003–5015. https://doi.org/10.1002/mp.12463
Singh, K., & Raza, S. (2022). Medical Visual Question Answering: A Survey of Datasets, Techniques, and Challenges. Journal of Imaging, 8(10), 256. https://doi.org/10.3390/jimaging8100256
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., … Amodei, D. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33, 1877–1901.
OpenAI. (2023). GPT-4 Technical Report. arXiv preprint arXiv:2303.08774. https://arxiv.org/abs/2303.08774
OpenAI. (2025). ChatGPT-5 Multimodal Capabilities: Technical Overview. OpenAI Research Report.
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022). AI in breast cancer screening: Opportunities and challenges. Nature Reviews Cancer, 22, 563–574. https://doi.org/10.1038/s41568-022-00480-0
Holzinger, A., Biemann, C., Pattichis, C. S., & Kell, D. B. (2017). What do we need to build explainable AI systems for the medical domain? arXiv preprint arXiv:1712.09923. https://arxiv.org/abs/1712.09923
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., … Dean, J. (2019). A guide to deep learning in healthcare. Nature Medicine, 25, 24–29. https://doi.org/10.1038/s41591-018-0316-z
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., … Zaremba, W. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. https://arxiv.org/abs/2107.03374
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016). Hierarchical question-image co-attention for visual question answering. Advances in Neural Information Processing Systems, 29, 289–297.
Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020). The ethics of AI in health care: A mapping review. Social Science & Medicine, 260, 113172. https://doi.org/10.1016/j.socscimed.2020.113172
Topol, E. J. (2019). High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 25, 44–56. https://doi.org/10.1038/s41591-018-0300-7
Buda, M., Maki, A., & Mazurowski, M. A. (2019). A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106, 249–259. https://doi.org/10.1016/j.neunet.2018.07.011
Singh, V., & Vats, R. (2023). Multi-modal AI in healthcare: Challenges, applications, and future prospects. Journal of Medical Systems, 47, 1–15. https://doi.org/10.1007/s10916-023-01914-7
