Page 72 - Read Online

Bala et al. Art Int Surg. 2025;5:476-89 https://dx.doi.org/10.20517/ais.2025.44 Page 486

Figure 2. Prevalence comparison of adverse events identified by the RAG model and the Benchmark Baseline. RAG: Retrieval-
Augmented Generation. (GitHub.com, https://github.com/InduBala-Y/RAG/blob/main/Stats%20test-RAG.png)

Population-level insights
Prevalence analysis revealed that the RAG model captured additional adverse event mentions that
traditional keyword-based approaches might miss. Higher detection rates for common complications such
as Pain, Bowel Problems, and Device Problems suggest that the model successfully identified subtly
expressed or indirectly mentioned events. Importantly, precision was preserved for rare events, avoiding
false-positive inflation that could mislead surveillance efforts. Statistically significant differences in
prevalence rates between RAG outputs and the baseline indicate that the model may uncover previously
under-detected adverse events in patient narratives. This finding has important implications for post-
market surveillance, suggesting that current methods may underestimate the true prevalence of certain
complications.

Methodological contributions
This study contributes to the growing body of work on RAG applications in healthcare by demonstrating its
effectiveness in a specialized domain with high reliability requirements. The zero-shot approach eliminates
the need for expensive domain-specific training data while achieving clinically relevant performance. The
integration of MMR for diverse retrieval ensures that the model accesses varied contextual information,
reducing the risk of bias from repetitive examples. Moreover, the framework’s interpretability through
retrievable source context addresses a critical need in clinical AI applications, where understanding the basis
for AI decisions is essential for trust and validation.

In conclusion, this study presents the development of a RAG framework for extracting structured clinical
information and adverse events from unstructured patient narratives related to mesh implant surgeries. By
integrating zero-shot prompting with LLaMA 2 and retrieval-based context enhancement, the system
achieved high accuracy in summarizing key report elements such as report date, hernia type, and adverse
event flag. It also demonstrated strong performance in extracting multi-label adverse events with minimal
hallucination and high fidelity to input narratives.

Evaluation against a Benchmark Baseline showed that the RAG-enhanced model consistently outperformed
the non-RAG baseline across all key metrics. Statistical analysis using Chi-square tests further confirmed
that improvements in accuracy were significant for most fields, especially for adverse event extraction and
classification. Prevalence comparisons also demonstrated the model’s ability to detect subtle or implicit