Page 68 - Read Online

Bala et al. Art Int Surg. 2025;5:476-89 https://dx.doi.org/10.20517/ais.2025.44 Page 482

RESULTS
Performance of structured summary generation
We evaluated the performance of a zero-shot structured summarization approach applied to FDA mesh
implant reports. Using prompt-based instructions, the model was tasked with extracting key clinical
variables from patient-written narratives, specifically focusing on Report Date, Hernia Type, and Adverse
Event Flag. The model’s outputs were compared against a Benchmark Baseline, which served as the
reference standard for evaluating model performance in terms of both accuracy and clinical validity.

To quantify how well the model’s output aligned with the Benchmark Baseline, we used the Jaccard
Similarity Coefficient. This metric is particularly effective in scenarios involving multi-label classification, as
it accounts for partial overlaps [30,31] . Given that individual reports may contain several adverse events - such
as pain, vomiting, or mesh migration - the Jaccard score helps capture the degree of agreement between the
model’s extractions and those previously labeled. The Jaccard coefficient is calculated as the size of the
intersection divided by the size of the union of the two sets.

The Jaccard similarity is defined as:

where A is the set of adverse events extracted by the model, and B is the corresponding set in the
Benchmark Baseline. Beyond accuracy, we also examined the reliability of the model’s outputs by
[32]
categorizing its errors into two types :

• Type 1 (Misinterpretation): The model generates content based on the input but distorts or misrepresents
it.

• Type 2 (Unsupported output): The model generates content that is not supported by the input text at all.

To illustrate these error categories, consider the following examples from our evaluation dataset. For Type 1,
a patient narrative stating ‘I experienced discomfort after my hernia surgery in 2019, but the pain started
getting worse in 2021’ resulted in the model incorrectly extracting the report date as 2019 instead of 2021,
demonstrating misinterpretation of temporal references. For Type 2, a narrative describing ‘I felt something
bulging and had severe discomfort after mesh surgery’ led the model to generate adverse events including
‘mesh migration’ and ‘infection’, which were not mentioned in the original text, representing hallucinated
content.

This dual-error framework allowed us to assess not only extraction accuracy but also the trustworthiness of
the outputs - critical factors in clinical NLP tasks where incorrect interpretations can lead to misleading
conclusions. A template of Structured Summary output generated from FDA reports using zero-shot
prompting is provided in Table 2.

To evaluate model performance empirically, we randomly selected 200 patient reports from the FDA dataset
and compared the structured outputs produced by the model to those in the Benchmark Baseline. Accuracy
was measured at the field level, with a prediction counted as correct only if it exactly matched the
corresponding label. Partial, inferred, or hallucinated responses were marked as incorrect unless they were
clearly supported by the input text. To determine whether the observed differences between the RAG model
and the Benchmark Baseline were statistically significant, we applied a Chi-square test for independence to