The rapid expansion of virtual health care has caused a surge in patient messages concomitant with more work and burnout among health care professionals. The COVID-19 pandemic hastened the adoption of virtual health care, concomitant with a 1.6-fold increase in electronic patient messages, with each message adding 2.3 minutes of work in the electronic health record and more after-hours work.

As a result, some medical researchers began thinking that perhaps artificial intelligence (AI) assistants could potentially aid in creating answers to patient questions by drafting responses that could be reviewed by clinicians.

And they then proposed a study to evaluate the ability of an AI chatbot assistant (ChatGPT) to provide quality and empathetic responses to patient questions. The results were just published in the JAMA Internal Medicine.

In this cross-sectional study of 195 randomly drawn patient questions from a social media forum, a team of licensed health care professionals compared physician’s and chatbot’s responses to patient’s questions asked publicly on a public social media forum. The chatbot responses were preferred over physician responses and rated significantly higher for both quality and empathy.

Of the 195 questions and responses, evaluators preferred chatbot responses to physician responses in 78.6% of the 585 evaluations.

Mean physician responses were significantly shorter than chatbot responses. Chatbot responses were rated of significantly higher quality than physician responses. The proportion of responses rated as good or very good quality for instance, was higher for chatbot than physicians.

This amounted to 3.6 times higher prevalence of good or very good quality responses for the chatbot. Chatbot responses were also rated significantly more empathetic than physician responses. The proportion of responses rated empathetic or very empathetic was higher for chatbot than for physicians. This amounted to 9.8 times higher prevalence of empathetic or very empathetic responses for the chatbot.

Thus the researchers concluded that “Further exploration of this technology is warranted in clinical settings, such as using chatbot to draft responses that physicians could then edit. Randomized trials could assess further if using AI assistants might improve responses, lower clinician burnout, and improve patient outcomes.”

John W. Ayers, PhD, MA – one of the researchers – told MedPage Today. “I think of our study as a phase zero study, and it clearly shows that ChatGPT wins in a landslide compared to physicians, and I wouldn’t say we expected that at all.”

He said they were trying to figure out how ChatGPT, developed by OpenAI, could potentially help resolve the burden of answering patient messages for physicians, which he noted is a well-documented contributor to burnout.

Ayers said that he approached this study with his focus on another population as well, pointing out that the burnout crisis might be affecting roughly 1.1 million providers across the U.S., but it is also affecting about 329 million patients who are engaging with overburdened healthcare professionals.

“There are a lot of people out there asking questions that maybe go unanswered or get bad answers. What do we do to help them?” he said. “I think AI-assisted messaging could be a game changer for public health.”

He noted that AI assistant messaging could change patient outcomes, and he wants to see more studies that focus on evaluating these outcomes. He said he hopes this study will motivate more research on this use of AI because of its potential to improve productivity and free up the time of clinical staff for more complex tasks.