Page 65 - Read Online

Page 479 Bala et al. Art Int Surg. 2025;5:476-89 https://dx.doi.org/10.20517/ais.2025.44

we selected a total of 2,422 mesh-related reports for analysis. Each report contained original structured
fields, including Report Date, Report ID, and Adverse Event Flag (Yes/No), as well as an unstructured free-
text narrative field (FoiText) that captures detailed patient descriptions of their post-surgical experiences. A
representative example of these data fields is shown in Table 1.

The free-text patient narratives describe a wide array of subjective experiences and complications. While
these are valuable for post-market surveillance, they present challenges for automated processing due to
their linguistic complexity and variability. Detailed preprocessing steps, data extraction methods, and
[24]
cleaning procedures were performed, as described extensively in our prior publication . The preprocessing
pipeline involved several key steps to ensure data quality and consistency. Raw FDA MAUDE reports were
initially filtered to retain only patient-generated narratives, excluding submissions from manufacturers and
healthcare providers. Text cleaning procedures included the removal of misspelled words, and the
correction of common spelling variations in patient-reported symptoms. Reports with insufficient narrative
content (fewer than 10 words) or missing critical metadata were excluded. Additionally, duplicate reports
were identified and removed based on content similarity and temporal proximity.

In the previous study, we employed a combination of named entity recognition (NER), keyword extraction,
and hierarchical stochastic block modeling (hSBM)-based topic modeling techniques to systematically
extract and create two additional structured data columns from these same 2,422 reports: hernia type
[25]
classification and detailed adverse event categorization . The hernia types were categorized into five
groups: inguinal, umbilical, incisional, SUI, and an unknown category for reports that did not explicitly
specify the surgical site. Adverse events were grouped into 20 clinically relevant categories, including pain,
infection, urinary issues, bowel problems, device-related complications, and others, as outlined in Table 1.

To provide a rigorous evaluation framework for the current study’s RAG model, we utilized the structured
outputs from our previous research as the Benchmark Baseline, which serves as the reference standard for
assessing model performance. This Benchmark Baseline represents ground-truth labels that were
independently generated using different methodological approaches (NER, keyword extraction, and topic
modeling), along with expert evaluations, providing a robust foundation for assessing the accuracy,
consistency, and clinical relevance of the AI-derived results. For the current evaluation, we employed
stratified random sampling to select 200 reports from the original 2,422 reports, ensuring proportional
representation across hernia types and adverse event categories. Importantly, these 200 evaluation reports
were completely independent from the RAG model development process and remained unseen by the
model during any phase of system design or prompt engineering. The remaining 2,222 reports formed the
retrieval corpus accessible to the RAG framework during inference, enabling the model to provide relevant
context for generating structured summaries and extracting adverse events.

The evaluation methodology employed a hold-out validation approach, where model performance was
assessed by directly comparing the RAG-generated outputs with the corresponding Benchmark Baseline
labels for the same 200 reports. No cross-validation was employed, as this study focuses on zero-shot
evaluation against pre-established reference standards, rather than model training or fine-tuning. By
comparing the structured outputs generated by the model - such as summaries containing report date,
hernia classification, and extracted adverse events - we assessed the accuracy, consistency, and clinical
relevance of the AI-derived results while ensuring complete independence between the label creation
process and the RAG evaluation framework.