Page 66 - Read Online

Bala et al. Art Int Surg. 2025;5:476-89 https://dx.doi.org/10.20517/ais.2025.44 Page 480

Table 1. Sample mesh implant report with structured fields
Field Example value or description
Report ID 123456
Report date YYYY-MM-DD
Flag adverse Y/N
event
Foi text (patient I felt something bulging in my belly button, causing agonizing discomfort. The umbilical hernia surgery was supposed to help,
narrative) but instead, it made things worse. My belly button swelled, and the pain in my abdomen intensified. To make matters worse, I
experienced vomiting and skin rashes, adding to the misery.
Hernia types [18] Incisional, umbilical, inguinal, SUI, unknown
Adverse Pain, bowel problem, device problem, sexual problem, infection, urinary problem, stress, chronic inflammation, incontinence
events [18] recurrence, nausea, prolapse recurrence, swelling, mesh erosion, bleeding, discomfort, sleeping issue, constipation, diarrhea,
UTI, allergic reaction

ID: Identity of the report; SUI: stress urinary incontinence; UTI: urinary tract infection.

An important aspect of the benchmark comparison is its ability to systematically identify potential
inaccuracies or hallucinations in the model-generated outputs. Specifically, we defined two types of errors:
Type 1 (Misinterpretations) - where the model’s output deviates from or misrepresents the original input,
and Type 2 (Unsupported Outputs) - where the model generates information not found in the input
narratives. These distinctions were crucial for thoroughly assessing the reliability and fidelity of the RAG-
enhanced outputs.

To evaluate these aspects rigorously, we conducted zero-shot experiments using two model variants: a
standard generative language model and a version enhanced by the RAG retrieval component. As illustrated
in Figure 1, the prompt-based instruction approach guided the model in generating structured summaries
by populating predefined output templates with information explicitly found in the narrative texts. The
model was specifically instructed to leave any template fields blank if the input narratives lacked the relevant
information, thereby minimizing speculative or unsupported generation and ensuring the outputs remained
closely aligned with the patient-reported experiences.

RAG framework
To enhance the model’s ability to generate accurate and contextually grounded outputs, we implemented a
RAG framework tailored for analyzing unstructured patient narratives related to mesh implant surgeries.
This approach combines information retrieval with generative modeling, enabling the language model to
produce more reliable and evidence-backed summaries and extractions .
[26]

The workflow begins with the segmentation of lengthy FDA mesh reports into smaller, semantically
meaningful text chunks. This step ensures that each text unit can be efficiently encoded and searched while
preserving contextual integrity. In parallel, structured data - such as report metadata - is processed row by
row to maintain alignment with the unstructured narrative.

Each text segment is then embedded using a sentence-transformer model from Hugging Face, which
converts natural language into dense vector representations that capture semantic meaning and inter-word
relationships . These embeddings, along with associated metadata (e.g., Report ID, date), are stored in a
[27]
vector database to enable high-speed, similarity-based retrieval. To identify the most relevant context for
each query, we employ the Maximum Marginal Relevance (MMR) algorithm via the LangChain
framework . MMR was selected for its ability to balance relevance and diversity when selecting retrieved
[28]
segments. For this study, we used a fixed parameter value of k = 20, consistent with common MMR
practices , which balances retrieval comprehensiveness with computational efficiency. This setting allows
[28]