When Machine Learning Breaks in Production

Machine learning rarely fails loudly in production: models keep answering while their inputs change and their accuracy slips. Here is how to catch it early. A crashed server gets noticed within minutes. A model that has quietly become less accurate can run for months before anyone connects the dots, because it still returns a tidy answer every time it is asked.

We build machine learning systems for a living, including MedicMinder, which helps doctors read MRI, CT and X-ray images, and ArchiML, which turns 2D architectural drawings into structured BIM models. Both taught us the same lesson: training a good model is maybe half the job. The other half is everything around it that decides what happens on the day it is wrong. This piece is written for owners and managers who are about to put a model in front of customers or staff and want to know what they are signing up for.

How ML Systems Actually Break

Ordinary software fails in ways people recognise. A button stops working, a page throws an error, a report comes out empty. A model does something harder to spot. It produces an answer that looks exactly like a correct one. There is no error message for "this prediction is wrong", so the failure has to be caught by something outside the model. A lending model that approves the wrong applicant, or a drawing parser that skips a door, records the result as a success. Whatever sits downstream simply carries on.

The most common cause is not the model at all but the pipes feeding it. Someone upstream renames a field, switches a measurement from kilograms to pounds, changes a default value on a form or replaces a camera with a different one. The model has no idea anything happened. It takes the new input, treats it as normal and returns a confident, plausible and useless result.

Next comes a slower problem: the world moves on while the model stays still. A model is a snapshot of the data it was trained on. Customer habits change, prices shift, new products appear, and the patterns it learned become less true each month. In our own areas the risk looks like a hospital installing a new scanner with different image settings, or an architecture firm adopting a drafting template the model never saw. Nothing breaks on the surface, yet the model is now working on inputs outside its experience. Retraining helps, but only if you know when it is needed and only if the fresh data is checked as carefully as the original set.

Then there is the long tail. Most real inputs are routine, but a steady trickle are rare, messy or strange: a blurred photo, a half-finished form, a drawing with hand-written notes over it. A model has seen few or none of these during training, and it will still answer. Confidence scores do not save you here. A model can be very sure and very wrong at the same time, especially on inputs unlike anything it has learned from.

Some failures are built into the way a model is used. When its own decisions shape the data it learns from next, errors can feed themselves. A fraud filter that blocks a payment never finds out whether that payment was honest. A recommendation engine that only learns from what it chose to show will keep showing more of the same. Over time the model grows more certain about a narrower picture of reality.

Finally, people change their behaviour around the tool. Once a system has been right a few hundred times, staff stop checking it. Other teams start using it for jobs it was never designed for, because it is already there and it seems to work. Neither shows up in any test set, and both turn small model errors into real business mistakes.

What these cases share is timing. None of them appear in a demo or in the evaluation done before launch. They appear weeks or months later, in live data, often first noticed by a customer. So the useful question is not whether your model will ever be wrong. It will. The question is how quickly you will find out, and what the system does in the meantime. A model that is wrong one time in fifty can be perfectly good business, provided the fiftieth case gets caught or is cheap to fix. The same model with no safety net around it becomes a liability.

What to Put in Place Before Launch

Start with ownership and paperwork, because they are cheap and they settle arguments before they begin. Name one person who owns the model in production, not just the team that built it. Write down, in plain language, what the model is for, what inputs it expects, what level of accuracy is acceptable in business terms and what it must not be used for. If a vendor supplies the model, put the operational details into the contract: who monitors it, who retrains it and how often, how fast you are told about a known problem, whether you get access to logs, and what happens to your data. Who carries responsibility when a decision goes wrong should be agreed here, with your lawyers, not worked out after an incident.

Next, give the model eyes and an exit. Log every input and output so problems can be traced later. Check incoming data against the ranges and formats the model was built for, and raise an alert when they drift. Watch the spread of predictions over time; a sudden change in the mix of answers is often the first sign of trouble. Whenever true outcomes become known, compare them with what the model said, even on a small sample. And decide in advance what happens when a check fails: fall back to a simple rule, send the case to a person, or have the system decline to answer. Keep old model versions so you can roll back in minutes instead of days.

Last, decide where a human signs off. High-stakes decisions, low-confidence answers and unusual inputs are the obvious places. Make that review real by showing the reviewer the evidence, not only the verdict. MedicMinder, for example, marks what it found on the scan so the doctor can see the reasoning and make the call. Treat retraining like any software release: test the new version on held-back data and on recent real cases, compare it with the one it replaces, and roll it out gradually. Then rehearse a failure: switch off the model for an afternoon and see whether the business keeps running. This is how we approach every ML project at MLMind. A model you can monitor, override and roll back is one you can trust with real work, even though it will sometimes be wrong.