Page 70 - Read Online

Bala et al. Art Int Surg. 2025;5:476-89 https://dx.doi.org/10.20517/ais.2025.44 Page 484

Table 3. Comparison of structured information extraction performance with and without RAG
Accuracy without RAG (%, Accuracy with RAG (%, P-value (0.05)
Field evaluated Common error
95% CI) 95%CI) (exact)
Report date 67 (60.1-73.5) 81 (75.1-86.2) Misinterpretation (Type 1), > 0.05 (0.062)
Unsupported output (Type 2)
Flag adverse events 83 (79.4-87.8) 99 (96.4-99.9) Misinterpretation < 0.05 (0.001)
Hernia type 60 (56.1-66.7) 82 (76.1-87.2) Misinterpretation, missing < 0.05 (0.032)
context
Adverse events 60 (58.4-65.4) 86 (80.7-90.5) Partial event capture < 0.05 (0.018)
Jaccard overlap (event 75 (69.7-80.7) 88.9 (84.1-92.8) Misinterpretation (Type 1), < 0.05 (0.025)
match) Unsupported output (Type 2)
RAG: Retrieval-Augmented Generation.

Table 4. Adverse event extraction template from FDA reports
Adverse adverse_events_template =
events AE 1:
template AE 2:
AE 3:
AE 4:
...
AE 10:
Prompt prompt_template = [INST]<<SYS|||<*** You are an intelligent system designed to extract adverse event information from
template patient reports. <</SYS|||<***
Using the context provided, identify up to ten adverse events (each in four words or fewer) related to pain, discomfort,
reactions, or device complications. If no adverse event is clearly mentioned, respond with “no event mentioned.” Do not include
inferred or speculative information.

context: [FDA_mesh_report]
adverse events template: [adverse_events_template]
filled template: [/INST]
Input [ID] had severe pain, not been able to eat, vomiting. I suspect that the mesh migration is adding to my issues. Disturbing
sensation of something moving inside.
Output AE 1: severe pain
AE 2: vomiting
AE 3: mesh migration
AE 4: no event mentioned
...
AE10: no event mentioned

FDA: Food and Drug Administration; AE: adverse event; ID: identity of the report.

In the 100 reports with clearly articulated adverse events, the model correctly identified the relevant events
in 93 cases, resulting in an accuracy of 93%. Errors in the remaining 7 reports typically involved missing one
or more relevant events due to subtle linguistic cues, paraphrased terms, or long and complex sentences.
These were classified as partial captures rather than hallucinations. For the 100 reports with vague, implied,
or non-standard event descriptions, the model adhered to prompt instructions correctly in 78 cases,
identifying only events explicitly supported by the text. However, in 7 instances, the model generated
adverse events not present in the original report (Type 2 hallucinations), and in 15 reports, it failed to
identify events that were subtly embedded in the language, resulting in a 78% accuracy rate for this group.

Combining both subsets, the overall extraction accuracy was 86%, reflecting instruction compliance and
contextual understanding. Comparison with the non-RAG model revealed consistently lower performance
and a higher frequency of hallucinated responses, as detailed in Table 3.