Assessing the Accuracy of ChatGPT in Answering Questions About Prolonged Disorders of Consciousness.
Sergio Bagnato, Cristina Boccagni, Jacopo Bonavita
Brain sciences April 13, 2025 Peer reviewed DOI: 10.3390/brainsci15040392 via PubMed
Summary
AI chatbots show remarkable accuracy in supporting families dealing with disorders of consciousness, with ChatGPT correctly answering up to 97% of caregiver questions about conditions like vegetative state. Testing both English and Italian queries, researchers found these AI tools provided accurate, empathetic responses about diagnosis and treatment options, while appropriately encouraging users to consult healthcare professionals.
Abstract
Objectives: Prolonged disorders of consciousness (DoC) present complex diagnostic and therapeutic challenges. This study aimed to evaluate the accuracy of two ChatGPT models (ChatGPT 4o and ChatGPT o1) in answering questions about prolonged DoC, framed as if they were posed by a patient's relative. Secondary objectives included comparing performance across languages (English vs. Italian) and assessing whether responses conveyed an empathetic tone. Methods: Fifty-seven open-ended questions reflecting common caregiver concerns were generated in both English and Italian, each categorized into one of three domains: clinical data, instrumental diagnostics, or therapy. Each question contained a background context followed by a specific query and was submitted once to both models. Two reviewers evaluated the responses on a four-point scale, ranging from "incorrect and potentially misleading" to "correct and complete". Discrepancies were resolved by a third reviewer. Accuracy, language differences, empathy, and recommendation to consult a healthcare professional were analyzed using absolute frequencies, percentages, the Mann-Whitney U test, and Chi-squared tests. Results: A total of 228 responses were analyzed. Both models provided predominantly correct answers (80.7-96.8%), with English responses achieving higher accuracy only for ChatGPT 4o on clinical data. ChatGPT 4o exhibited greater empathy in its responses, whereas ChatGPT o1 more frequently recommended consulting a healthcare professional in Italian. Conclusions: Both ChatGPT models demonstrated high accuracy in addressing prolonged DoC queries, highlighting their potential usefulness for caregiver support. However, occasional inaccuracies emphasize the importance of verifying chatbot-generated information with professional medical advice.