Abstract

This study proposes an evidence-graph dual-store multimodal retrieval-augmented generation (MM-RAG) framework that combines MRI-derived visual evidence with textual evidence retrieved from similar historical cases and from a curated clinical knowledge repository. The framework couples a case-memory index with a knowledge index, and an agreement-weighted retrieval score selects evidence that is consistent across the two stores. Retrieved items are organised as an evidence graph rather than concatenated, so that supportive signals propagate while unrelated context is suppressed. Experiments use publicly available multiparametric MRI collections and are restricted to three classification tasks: tumor detection, tumor subtype classification, and tumor grade prediction. The framework is compared against three baselines: a vision-only 3D ViT/CNN classifier (M1), an early-fusion multimodal transformer without retrieval (M2), and a text-only retrieval-augmented clinical language model (M3). The proposed method reaches a detection AUROC of 0.985 and AUPRC of 0.980 with an accuracy of 95.8% and an F1 of 95.2%. For the finer-grained tasks, subtype AUROC/AUPRC is 0.952/0.935 and grade AUROC/AUPRC is 0.934/0.915, with F1 values of 89.6% and 87.5%. Agreement-weighted dual-store retrieval improves Recall@5 and nDCG@5 in both stores and yields measurable accuracy gains attributable to retrieved evidence. Confidence gating lowers the error under abstention to 2.9% and improves calibration (ECE = 0.024; Brier = 0.078). These results indicate that retrieval-grounded multimodal fusion supports accurate, interpretable, and better-calibrated brain tumor evaluation.

This work is licensed under a Creative Commons Attribution 4.0 License.