Publications

Large language models (LLMs) can summarize electronic health records and answer patient-specific questions, yet they may produce input-conflicting hallucinations: clinically plausible claims that are unsupported by the source patient record. We present KG-Verify, an auditable framework for detecting these patient-specific faithfulness errors by converting the source discharge summary and the generated answer into typed, record-specific clinical knowledge graphs. Nodes represent normalized clinical concepts, attributes encode details such as dose, route, duration, and laboratory values, and the answer graph is checked against the record graph under one-directional subset semantics. Each mismatch is returned with the offending claim and available source evidence. KG-Verify uses a locally served MedGemma-27B extraction backbone and UMLS concept normalization. We evaluate it on 64 MIMIC-IV discharge summaries comprising 434 question–answer instances, including 259 answers with one injected evidence or mechanism error. KG-Verify detects 232 of the 259 altered answers (recall 0.896, precision 0.663, F1 0.762), whereas direct MedGemma judges at 4B and 27B scales achieve recall no higher than 0.220. The high sensitivity is accompanied by 118 false positive flags, showing that reference-graph completeness and normalization remain the principal bottlenecks. These results support structural verification as a promising, interpretable safety layer, while also indicating that larger clinician-validated cohorts, stronger non-LLM baselines, temporal modeling, and extraction ablations are required before clinical deployment.

Keywords: hallucination detection, large language models, knowledge graphs, clinical AI, patient safety, UMLS, faithfulness