Page 34 - Read Online

Page 270 Yoseph et al. Art Int Surg 2024;4:267-77 https://dx.doi.org/10.20517/ais.2024.38

Table 1. Inclusion and exclusion criteria for websites containing frequently asked questions regarding ACDF surgery
Inclusion criteria Exclusion criteria
Patient-focused, relevant questions and answers Nongeneralizable physician anecdotes and physician-specific inquiries
Evidenced-based medical websites Proprietary surgical techniques and devices which are not widely available
Information presented in the form of questions and answers Research articles, non-patient-centered information

ACDF: Anterior cervical discectomy and fusion.

Table 2. Ten consolidated frequently asked questions administered to ChatGPT-3.5, Gemini, and doctors
Question 1 What is ACDF surgery, and how is it performed?
Question 2 How long is the typical recovery and fusion period, and when can I expect to return to work and daily activities?
Question 3 What materials are used for fusion in ACDF surgery, and what are the complications associated with both the procedure and the
materials?
Question 4 Why is ACDF surgery recommended, and what are its potential risks and benefits?
Question 5 What are the long-term outcomes, success rates, and potential long-term effects or risks associated with ACDF surgery?
Question 6 What restrictions or precautions should I be aware of during my recovery, including wearing a neck brace and certain activities to
avoid?
Question 7 What should I expect post-surgery in terms of incisions, scars, pain management, and potential discomfort?
Question 8 How long will I need to stay in the hospital post-surgery, and will I require physical therapy or rehabilitation?
Question 9 How long does ACDF surgery typically take?
Question 10 Are there any alternative treatments to ACDF, and under what circumstances might this surgery be repeated for other disc issues?

ACDF: Anterior cervical discectomy and fusion.

responses. At the end of phase 2, we had a total of 40 answers (10 each from two spine surgeons, ChatGPT-
3.5, and Gemini) to our 10 commonly asked questions [Supplementary Table 1]. All searches for phase 2
occurred on November 23, 2023.

Phase 3 involved recruiting study participants according to the inclusion and exclusion criteria shown in
Table 3. In total, there were 10 participants including 5 patients who had previously had cervical spine
surgery and 5 gender- and age-matched controls. All study participants were given the same questionnaire
with the 10 commonly asked ACDF surgery questions and 40 answers from phase 2. Participants were asked
to use a 5-point Likert scale (Strongly Disagree, Disagree, Neutral, Agree, and Strongly Agree) to rate every
response on both clarity and completeness. The exact prompts in the questionnaire were “This answer is
clear and easy to understand” and “This answer completely answers the question”. All participants were
blinded to LLMs or physicians being involved in the generation of responses, and the responses for every
question appeared in a random order for each participant. Phase 3 produced a total of 800 data points for
subsequent analysis.

Statistical methods
Data normality was assessed using the Shapiro-Wilk test. For non-normal data, the Kruskal-Wallis test and
the Mann-Whitney U test were utilized, as appropriate, to assess differences in word counts by LLMs vs.
physicians and overall ratings provided by patients and controls. Dunn-Bonferroni post-hoc test was
conducted to pinpoint specific differences. Categories for the 100% stacked bar charts were set as follows:
Likert ratings are converted to numbers (strongly disagree = 1, disagree = 2, neutral = 3, agree = 4, and
strongly agree = 5), and then positive and negative feelings are combined to create categories (1 and 2 =
disagree, 3 = neutral, 4 and 5 = agree). For our analyses, physician-generated response ratings were