Page 71 - Read Online

Page 485 Bala et al. Art Int Surg. 2025;5:476-89 https://dx.doi.org/10.20517/ais.2025.44

Prevalence measurement of adverse events
To evaluate the model’s ability to capture population-level trends in adverse event reporting, we compared
the prevalence rates of each adverse event category extracted by the RAG model with those in the
Benchmark Baseline. For this analysis, we focused exclusively on clinically meaningful reports - specifically,
those that contained at least one adverse event label in the Benchmark Baseline. Reports without any
adverse event annotations were excluded to ensure a more targeted and interpretable comparison.

As illustrated in Figure 2, the model identified Pain as the most frequently reported complication, with a
higher prevalence than that observed in the Benchmark Baseline. This was followed by increased detection
of other high-frequency adverse events such as Bowel Problems and Device Problems. The model also
demonstrated greater sensitivity in identifying adverse events such as Urinary Problems, Swelling, and Mesh
Erosion. In contrast, the model’s output showed minimal overestimation for lower-prevalence
complications such as Allergic Reaction and UTI, closely aligning with baseline estimates. Statistical
comparison of prevalence rates revealed that the differences between the model outputs and the Benchmark
Baseline were significant for several adverse event categories P < 0.05.

DISCUSSION
The results demonstrate that RAG integration significantly enhances the accuracy and reliability of
structured information extraction from unstructured patient narratives. The substantial improvements
observed across all evaluation metrics support the effectiveness of combining zero-shot prompting with
retrieval augmentation for clinical text analysis.

Performance improvements and clinical significance
The most notable improvement was observed in the Adverse Event Flag classification, where accuracy
increased from 83% to 99% with RAG integration. This near-perfect performance is particularly significant
for clinical surveillance applications, where accurate identification of adverse events is critical for patient
safety monitoring. The improvement suggests that RAG’s contextual enrichment enables the model to
better interpret vague or ambiguously stated symptoms that would otherwise be misclassified.

The Hernia Type classification also showed substantial improvement, from 60% to 82% accuracy, a 37%
relative increase. This enhancement is clinically meaningful, as accurate hernia type identification is
essential for understanding device performance across different surgical applications. The RAG model’s
ability to infer hernia types from contextual signals, even when not explicitly stated, demonstrates the value
of retrieving relevant similar cases from the database. Report Date extraction improved from 67% to 81%,
addressing a common challenge in clinical text processing, where temporal expressions can be ambiguous.
The reduction in Type 1 misinterpretations indicates that RAG helps disambiguate between surgery dates,
symptom onset dates, and actual report dates through contextual understanding.

Error analysis and model reliability
The dual-error framework (Type 1 and Type 2) provided valuable insights into model reliability. The
reduction in both error types with RAG integration demonstrates improved factual grounding and reduced
hallucination. Type 1 errors, primarily involving temporal misinterpretations, were significantly reduced
through access to similar contextual examples. Type 2 errors, involving unsupported outputs, were
minimized by grounding generation in retrieved relevant passages. The adverse event extraction task
revealed notable patterns. The 93% accuracy for clearly articulated events versus 78% for ambiguous
descriptions highlights the continued challenge of processing subjective patient language. Nonetheless, RAG
maintained high precision while improving recall, offering a favorable trade-off for clinical applications.