Who Answers When Medical AI Gets It Wrong? A Practical Risk Checklist

Adopting AI in a clinic does not remove risk; it moves it. This checklist shows healthcare leaders how to choose, test, supervise and document medical AI. When a scan-reading assistant or a triage score gets something wrong, there is no neat universal answer to who is responsible: the clinician, the hospital, or the company that built the tool. Rules differ from country to country, and courts and regulators are still working out how older ideas about negligence and faulty products apply to software that learns from data. One thing does not change. Patients expect the people treating them to act with reasonable care, and that expectation does not switch off because a computer was involved.

Why is AI different from other equipment? A conventional device tends to fail in ways people notice. A machine learning model can be right most of the time and wrong in quiet, patterned ways. It may do well on the data it was trained on and worse on patients from a different age group, on images from another scanner brand, or in a population where a disease is more or less common. Those failures rarely announce themselves. So responsibility with AI is less about a single moment of error and more about choices an organisation made long before it: which tool it picked, how it tested it, who was allowed to rely on it, and what records it kept.

A quick note on scope. This article is general information for clinic owners, hospital managers and health-tech founders, not legal advice. If you are signing a contract or rolling out a tool that affects diagnosis or treatment, speak to a lawyer who knows healthcare law where you operate, and check with your regulator whether the tool counts as a medical device in your market. Requirements for certification, reporting and data handling vary, and they change.

We write from the builder's side of the table. At MLMind we develop MedicMinder, which analyses MRI, CT and X-ray images with segmentation, classification and detection tools so doctors can read scans faster. We design it as an assistant: it outlines, measures and flags, and the doctor makes the call. That is not only a product decision. It is a risk decision, because it keeps a trained person between the model's output and the patient, and it shapes much of the advice below.

Here is the idea that ties the checklist together. You cannot make an AI tool error-free, but you can make its errors visible, catchable and traceable. Visible means clinicians can see what the tool did and how sure it was. Catchable means someone qualified has the time and the information to disagree with it. Traceable means that if something goes wrong, you can reconstruct what the tool was meant to do, how it was checked, who reviewed its output and what happened next. Almost every step below serves one of those three goals.

A Six-Point Checklist for Adopting Clinical AI

1. Pick a tool with a narrow, clearly stated job. Write down in one or two sentences exactly what the tool will do and for which patients, for example: flag possible fractures on adult wrist X-rays for a radiologist to review. Ask the vendor for its intended-use statement, any regulatory clearance or certification it holds in your market, and a plain description of how it was evaluated. Be cautious with tools that promise to do many things at once, and with accuracy figures that arrive without any detail about the data behind them.

2. Test it on your own patients before you trust it. Results reported elsewhere are a starting point, not proof. Run the tool silently in the background on a sample of your recent cases, compare its output with what your clinicians concluded, and study the cases where they disagree. Look at the groups that matter locally: children and adults, different machines, rare but serious conditions. Repeat the check after software updates, new equipment or a shift in the patients you see, because a model that performed well last year can slowly drift.

3. Keep a clinician in charge, and make that real. A human in the loop means little if the human is rushed or simply clicks accept. Decide who reviews each AI result, how much time they have, and what they should do when they disagree. Present outputs in a way that invites checking: confidence levels, highlighted regions on the image, and a clear note of what the tool was not built to assess. Train staff on its known weak spots, and watch for two opposite habits, blind reliance and automatic dismissal. Both waste the tool and both put patients at risk.

4. Write things down as you go. Keep a record of why you chose the tool, the results of your local testing, which version was running on which dates, and the clinical workflow around it. In patient records, note when an AI result informed a decision and whether the clinician agreed with it. Good documentation protects patients first, because it helps you spot and fix problems. It also lets you show later, to a regulator, an insurer or a court, that the organisation acted with care rather than on faith.

5. Protect patient data and read the contract twice. Scans and records are sensitive personal data. Follow the principles behind GDPR and similar laws: use only the data the tool needs, have a lawful basis for processing it, tell patients in plain words how AI is used in their care, limit who can access it, and know where it is stored and processed. Then look hard at the vendor agreement. Who carries responsibility if the tool fails? What insurance does the vendor hold? How are updates announced and tested before they reach you? May the vendor reuse your data to train its models, and on what terms? And can you get all of your data back if you part ways?

6. Plan for the day something goes wrong. Agree in advance how staff report a suspected AI error, who looks into it, and who has the authority to pause the tool. A review should cover the model's output, the clinician's reading and the workflow around them, not just the software. Tell the vendor quickly, be honest with any affected patient, and meet whatever reporting duties apply to you. Then feed the lessons back into testing and training. No checklist removes risk entirely, but an organisation that can show careful choices, real oversight and honest follow-up is in a far stronger position, both ethically and legally, than one that simply switched a tool on and hoped for the best.