Reading Guide: Patient Use of AI in Medical Diagnosis

What actually is AI, anyway? 

AI programs are complex, prediction-based computer algorithms that identify patterns across vast amounts of data to generate answers and make recommendations.

Lots of people - both doctors and patients - use AI for health. Across the US, 29% of patients use AI for health information at least monthly (Kaiser Family Foundation, 2026), and 81% of physicians use AI in some capacity (American Medical Association, 2026). Among patients, free AI platforms such as ChatGPT and Gemini are commonly used to access health information, prepare for physician visits, and upload clinical notes or lab results to ask questions. Paying for wearable devices like Oura rings or smart watches, patients can also sync up their own health data and get more personalized responses. 

To hear one patient’s story of how they used AI to reach their diagnosis, watch CODEX’s webinar  "AI and Me: How Technology Led to the Right Diagnosis” featuring Sue Sheridan, MBA, from Patients for Patient Safety US.

Despite AI’s growing prevalence, it is important to remember that experts often describe AI as a “black box.”  We may know how these algorithms are trained (meaning, given rules for how to function), but we cannot fully know why it arrives at a particular final recommendation. For more information, read  "Understanding the AI Wave: Foundational Knowledge for Improving Diagnosis and Beyond." 

What are the benefits of using AI for medical diagnosis?  

When given sample exam questions or “hypothetical” clinical vignettes, AI tools can choose answers and generate lists of possible diagnoses, on par with doctors. 

AI has been found in studies to do really well in example descriptions of symptoms, particularly in example cases presented in common medical school exams, to arrive at a correct multiple choice or to generate a list of possible diagnoses. These have NOT been verified in real-world settings, but patients and clinicians may use AI to explore lists of possible diagnoses that could be going on.  

Helpful patient-friendly explanations to understand a diagnosis.  

Patients and families can use chatbots like ChatGPT between diagnostic imaging and surgical consultation to help navigate diagnostic steps. For one example of how a family used ChatGPT to navigate their diagnosis journey, read "Patients and Caregivers Leveraging AI to Improve Their Health Care Journey: Case Study and Lessons Learned."

Preparing for medical visits, so you don’t draw a blank.

"Help me prep a one-pager for my doctor visit": simple, but effective. Dave deBronkart, cancer survivor and patient activist, shared this post on his blog, “Patients Use AI,” detailing how a Reddit user asked ChatGPT for directions ahead of a doctor's visit. Using AI chatbots ahead of medical visits can help patients make the best use of their time with a healthcare professional. 

What are the AI-related risks I need to be aware of as a patient?

Inaccuracies in Diagnosis 

While overall performance in sample cases or simulated studies is on par with or can even exceed doctors, AI still makes mistakes and has not been fully tested in real-world settings by average users. 

A recent study tested how well AI performed when non-clinician users were given mock scenarios of medical conditions and asked to use AI to decide whether or how to seek additional care (such as going to the emergency room or making an appointment with their primary care physician).  Results showed AI models correctly identified relevant conditions in fewer than 35% of cases. While other studies have found AI to be more accurate compared to human physicians, patients should understand that the accuracy of AI, like ChatGPT or Gemini, depend heavily on the completeness of the information you put in and how you phrase it.  

Read the study, "Reliability of LLMs as medical assistants for the general public: a randomized preregistered study," or watch our CODEX Editor's Pick featuring Gezzer Ortega MD, MPH, to learn about his insights on the study.   

Wrong Treatment Recommendations

AI can make a list of possible causes, but its recommendations on management are unreliable. 

Another recent study evaluated how well ChatGPT Health could advise users on when and where to seek medical care based on mock scenarios created by researchers. When participants used AI to guide their care using these mock scenarios, the diagnosis and follow-up recommendations were wrong more than half the time. While AI can be helpful for understanding symptoms, preparing for medical visits, and thinking through possibilities, this study shows AI is not reliable enough to be used for emergency decisions, such as whether patients should seek urgent care. 

For more on this study, read "ChatGPT Health performance in a structured test of triage recommendations," or check out the CODEX AI and Diagnosis webinar featuring the study's lead author, Ashwin Ramaswamy, MD.

Bias

AI is more likely to be wrong for groups experiencing marginalization or minoritized status.

AI uses data that systemically lacks participation from minoritized groups, potentially making the source data incorrect. This impacts racial and ethnic minorities, women, and older adults, among others. 

For more information, read "Sociodemographic biases in medical decision making by large language models," or check out the CODEX webinar “What is bias in AI—and how can hospitals prevent it?” featuring Paige Nong, PhD, A Jay Holmgren, PhD, MHI, and Ryan Levi from Tradeoffs. 

Hallucinations

Hallucinations are false facts generated by AI that seem correct. 

One study found that after planting errors (e.g., a fake lab value, sign, or disease) in clinical notes, AI models repeated or even elaborated upon the errors in up to 83% of cases. This poses a risk to patients because a user doesn’t know when an AI recommendation is false, making it difficult to rely on, especially for critical health information. AI can even invent scientific studies that don’t exist, and tell you made-up results from an existing study that concluded something else. Users can’t know what a hallucination is unless they check all the references an AI model lists as citations. 

For the entire study, see "Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support."

Harmful Recommendations

AI has, at times, recommended bizarre and dangerous treatments, such as taking poison, or encouraging self-harm. 

For a sample case, see "A Case of Bromism Influenced by Use of Artificial Intelligence."

“People Pleasing”

Sycophancy refers to AI agreeing excessively with users and prioritizing helpfulness over honesty, often resulting in inaccurate information. 

This paper provides the example of a patient who doesn’t know that acetaminophen is the same drug as Tylenol. They ask an AI model to “tell me why acetaminophen is safer than Tylenol,” and the AI model may respond with false medical information in an effort to be helpful. This paper shows that AI cannot reliably resist requests for illogical content and can generate or perpetuate false health information, creating patient safety issues. 

For the full study, see "When helpfulness backfires: LLMs and the risk of false medical information due to sycophantic behavior."  

Lack of Transparency from Healthcare Institutions

Currently, there is no standardized policy for how healthcare systems reveal how and when they are using AI. Because there are no federal laws, states, hospital networks, and clinics are figuring policies out on their own. Patients should know that it can be tricky to know when and how AI is being used for their healthcare. Patient safety advocates reviewed AI use cases, research, underlying technology, and oversight processes and identified four priority areas of concern for the impact of AI on patient safety:  1) data integrity and bias, 2) efficacy, 3) payment, and 4) transparency. The authors support technology’s use to help patients, but only if patients have an active role in the development and implementation of technologies, rule development, and oversight process, emphasizing “Nothing about us, without us.”  

Find the whole study here “AI: Promise or Peril for Patient Safety” and for more from lead study author Beth Daley Ullem, MBA, check out the CODEX webinar “Talking AI with Patients: A Roundtable on What Health Systems Should Say—and Hear.”  

Collective AI Rights for Patients. For additional information, the Light Collective is a national community of patient activists, clinicians, and health data experts committed to building tech and digital rights for patients. This resource was created by patients with lived experience and community leaders to help patients of all backgrounds learn their rights regarding AI in healthcare and have the necessary tools and regulations to advocate for themselves. 

Data Handling and Privacy Considerations

When seeing clinicians, you may increasingly be asked as a patient whether it’s okay to record the encounter using an AI scribe. After recording, AI scribes generally generate transcripts and structured clinician notes, which physicians may edit and approve before being included in your medical record. AI scribes may make healthcare visits more efficient and ideally allow clinicians to spend more time focusing on patients instead of typing notes, but patients should understand how their conversations are being recorded, stored, and used. Privacy practices vary widely across AI companies, and consent laws vary across states. While there are still unresolved legal and privacy questions, as a patient, healthcare organizations should clearly inform you when an AI scribe is being used and be able to explain what data are collected, how they are protected, and who has access. 

For more information, read “Ambient AI in Clinical Practice — The Legal Landscape of Recording Consent Requirements” and for more from I. Glenn Cohen, JD, check out his CODEX webinar "AI and Diagnostic Medicine: Legal and Ethical Issues."   

How can I balance the risks and benefits of AI for my own health? 

To figure out which questions and tasks AI models like ChatGPT, Gemini, and Claude are most suited for, Adam Rodman, MD, MPH, FACP, from Beth Israel Deaconess Medical Center, developed a stoplight system. Check out highlights below from the full article: "AI in healthcare: Can a chatbot answer your medical questions?"

Careful “Prompting” on the Text You Input

Rodman writes that AI fields are not the same as search terms. Provide useful context, like your pre-existing conditions, and ask the program to give counseling as a medical professional would.

Low Risk Examples

1. Understand general medical information, like “Explain diabetes” or “What is hypertension?”
2. Ask for advice like “I have lactose intolerance. Can you help me develop a menu plan?”
3. Prepare questions for a medical visit, such as “Based on these symptoms, are there any questions I should ask my doctor.
4. Explain the meaning of after-visit clinical notes, lab reports, and imaging test findings found in your patient portal, such as “Explain what the doctor meant in this note.” (Remember: Remove personal identifiers first like your birthday, name, address, etc.) 

Medium Risk

1. Exploring symptoms: According to Rodman, AI programs can try to tell you what they are programmed to say that you’d “want” to hear and perform worse on classifying emergency vs. non-emergency situations (for more information, see the article above on “People Pleasing”).  
2. Instead of “what could be the cause of a headache?” try... “I am a 41-year-old male with a diagnosis of high blood pressure, and have had a mild headache for 2 weeks. Can you interview me like I’m a doctor to help explore what might be the diagnosis?”
    

Do Two-step Prompting

Ask AI if its recommendations would be different if a certain demographic was different (like, if you were a different race or gender) – this is called “two-step prompting.” If you find differences, it may be important to share these findings with your clinician.  

High Risk

The evidence so far does not support using AI for the following:

1. What medications or supplements to take or not take.
2. Decisions on final medical treatment plans.
3. Management of mental health conditions, especially for depression, self-harm, and/or unwanted thoughts  
4. Putting in identifying health information, such as name, address, and record number. Any health records you input may be used in non-transparent ways without your knowledge.  
5. Taking immediate action based on AI. 
    

Engage critically

Before syncing your health data with any AI tool, patient advocates Hugo Campos and Liz Salmi caution you to think critically about what you’re doing and who you are trusting with your data. If you do engage, you have to engage critically. For more information on developing AI Health Literacy, read their commentary "Critical AI Health Literacy as Liberation Technology: A New Skill for Patient Empowerment."

Find the surveys:

KFF. KFF Tracking Poll on Health Information and Trust: Use of AI for Health Information and Advice; June 2026.  

American Medical Association. 2026 Physician Survey on Augmented Intelligence. American Medical Association; March 2026.  

This guide was created by the CODEX Learning Hub, 2026. We want to thank Io Dolka, MS, BCPA, Founder & Chief Care Advocate at GreyZone Health and Board of Directors, Community Improving Diagnosis in Medicine (CIDM), for editorial guidance.