ChatGPT wants access to your health records so it can be a better not-doctor

OpenAI is expanding its healthcare footprint by encouraging users to connect Apple Health data and supported medical records to ChatGPT - one day after a user sued the company, alleging the chatbot's medical advice nearly killed him. Health in ChatGPT, as the new feature is called, allows eligible users to connect ChatGPT to Apple's Health app in iOS, giving the chatbot access to health information users choose to share, including medical records, sleep, activity, and other data stored in the app. For those unfamiliar with Apple's Health app, it's a health dashboard that collects data from an iPhone, Apple Watch, and compatible health and fitness apps, while also allowing users to add personal health information such as medications, allergies, and medical conditions. Connecting ChatGPT to the app, says OpenAI, gives the chatbot additional context for health-related conversations, allowing it to provide more personalized responses based on a user's health information. With your permission, ChatGPT can consider relevant information you have connected, such as medications, lab results, recent visits, sleep, and activity, alongside the goals and context you share," OpenAI said in its announcement. According to the company, the integration of health information into general ChatGPT chats comes after testing a dedicated Health experience in ChatGPT earlier this year that it found users often bypassed, instead conducting more than 70 percent of their health-related conversations outside the dedicated health space. Did no one think of the timing? OpenAI said that it won't use connected health data to train its foundation models or serve ads. Beyond those privacy concerns, however, there are some serious reasons to be skeptical of the Health in ChatGPT rollout. Take, for example, the fact that OpenAI was sued on Wednesday in San Francisco Superior Court by a Florida man, who alleges ChatGPT gave him "extremely dangerous medical recommendations" that discouraged him from seeking medical care as his pulmonary embolism worsened. Later hospitalized, the plaintiff claims he now faces years of intensive physical and psychological recovery after the chatbot allegedly repeatedly reassured him that his symptoms were not serious and urged him to remain at home. An OpenAI spokesperson told The New York Times that ChatGPT's terms of service make clear it is not intended for medical diagnosis or treatment. The spokesperson also told the newspaper that the company's models have improved since the incident at the heart of the lawsuit - a point OpenAI reiterated to The Register. Similarly, the Health in ChatGPT announcement notes that it can still make mistakes and shouldn't replace qualified medical professionals, even though OpenAI says it worked with hundreds of physicians to test the feature for performance and safety. "ChatGPT is not a doctor and should never be used as a substitute for medical care, diagnosis, or treatment," an OpenAI spokesperson told The Register in an email. "Treating chatbots as the whole story behind people's medical decisions or outcomes oversimplifies a much bigger challenge, and risks getting in the way of people accessing powerful new tools that can aid them in their health journey." The inclusion of medical records in ChatGPT's contextual data may help prevent some instances of bad advice (that plaintiff's data may have included prior treatment that suggested increased risk for pulmonary embolism, for example), but that still won't stop ChatGPT from simply being wrong. We've published multiple stories in recent months about AI's lack of accuracy being a major hurdle for medicine. One study earlier this year found that AI chatbots were no better at helping people make health decisions than conventional online resources, including search engines. Another study we covered found that LLMs failed at early differential diagnosis in more than 80 percent of cases. In Canada, the Auditor General of Ontario found that AI note-taking systems used by doctors routinely introduced medication errors into patient notes, omitted critical details, inserted incorrect information, and hallucinated things that neither patients nor clinicians brought up during appointments. In other words, whether or not it's armed with a patient's data, involving an AI in healthcare decisions or advice sure seems like a bad idea. We use dedicated training to improve the models behind health conversations in ChatGPT," OpenAI said, noting that its models have made meaningful gains in recognizing when urgent care may be needed, asking for relevant context, explaining uncertainty, and making complex information easier to understand." We're told OpenAI has trained its models to address uncertainty in health queries when information is incomplete rather than just making up an answer. Whether that's enough to make it safe and reliable will ultimately be up to users to decide. For those determined to put their wellness in the hands of a company defending a lawsuit alleging ChatGPT dispensed dangerous medical advice, Health in ChatGPT is rolling out now to logged-in iOS and web users aged 18 and older in the US on Free, Go, Plus, and Pro plans. (R)