Page 54 - Read Online

Page 317 Boyd et al. Art Int Surg 2024;4:316-23 https://dx.doi.org/10.20517/ais.2024.53

Conclusion: Doximity’s AI platform produces reasonable, accurate information in response to common patient
queries. With continued reinforcement learning with human feedback (RLHF), Doximity GPT has the potential to
be a useful tool to plastic surgeons and can assist with a range of tasks, such as providing basic information on
procedures and writing appeal letters to insurance providers.

Keywords: Artificial intelligence, natural language processing, ChatGPT, AI, generative AI, plastic surgery, AI
integration in surgery

INTRODUCTION
Artificial intelligence (AI) is positioned to be a disruptive technology for healthcare. Over the past several
years, AI has demonstrated exponential growth and transitioned from a theoretical idea to a tangible reality.
This excitement for AI has blossomed within plastic surgery as the literature denotes countless descriptions
[1-4]
of use cases, potential applications, and discussions of ethical implementation . While the indications for
AI-powered assistance are innumerable within plastic surgery, natural language processing (NLP) - a form
of generative AI - is poised to be rapidly integrated into the clinical workflow for plastic surgeons, practice
managers, and staff. The NLP capabilities of AI allow it to analyze, comprehend, and produce language .
[5]
Prior work analyzing an NLP program demonstrated efficacy in generating medically sound
recommendations for common patient inquiries surrounding perioperative surgery for a breast reduction
procedure . Despite the integrity of medical recommendations, this study highlighted limitations in the
[6]
accessibility of AI-generated outputs with a higher-than-recommended average reading level of content [7-10] .
Such analyses have been duplicated for a variety of large language models (LLMs) [11-13] .

Limitations of this prior work include the assessment of LLMs without a specific emphasis or additional
training in medicine. Doximity has launched a healthcare-specific, Health Insurance Portability and
Accountability Act (HIPAA)-compliant AI tool. This platform, Doximity GPT, was created to facilitate
written outputs specifically designed for healthcare, including patient instructions, appeals to insurance
providers, and educational pamphlets. While backed by the same NLP program as ChatGPT (OpenAI, San
Francisco, CA), Doximity GPT incorporates additional healthcare-specific training and utilizes
reinforcement learning with human feedback (RLHF) to improve ChatGPT’s programming.

As it is being marketed as having specific training in healthcare, one of the aims of this study was to assess
how this might impact outputs from the LLM. To date, no studies have assessed the product of this
healthcare-specific NLP program. In this study, we sought to compare the quality and accessibility of
outputs from Doximity GPT to outputs from a generic NLP program in response to common questions
about breast augmentation procedures, which are among the most frequently performed procedures in the
[14]
United States .

METHODS
The new HIPAA-compliant, healthcare-specific AI platform Doximity GPT (Doximity, San Francisco, CA)
is readily accessible to physicians. This AI program was compared to the publicly available NLP ChatGPT
3.5 (OpenAI, San Francisco, CA). Both AI interfaces were accessed in April 2024. A list of 20 frequent
patient inquiries regarding a breast augmentation procedure was generated. The list was adapted from a
previously published study, following the clinical experience of the senior authors (N.S.K. and M.C.) and
their expertise in perioperative management . Breast augmentation was selected as it is one of the most
[6]
frequently performed procedures in the United States annually . Inquiries were entered into each of the
[14]
two NLPs in two distinct formats: utilizing a general search term (“breast augmentation driving”) and