Your AI Feature Isn't Failing Because of the Model

A client calls me. He wants AI in his product. He has the budget, he has the team, he even has the model picked.

Six months later the feature exists and nobody uses it.

I have now seen this play out enough times to be fairly sure the problem is almost never the model.

The problem is that the product was built over several years for a system that is always right. You click, you get. The interface exists to remove doubt. And then you bolt onto it something that is often right, sometimes wrong, and always sounds certain.

Two bad answers and the user stops coming back. Not because the output was bad. Because they had no way of knowing when it was.

Retrofit is harder than greenfield, and nobody prices that in

Every article about AI product design assumes you are starting from nothing. You are not. You have users with habits, a support team with scripts, a backlog with two years of debt in it, and an interface where every pixel was justified by someone at some point.

That changes three things.

Your users did not ask for this. They came for the thing your product already does. An AI feature is an interruption until it proves otherwise.

Your existing workflows are load-bearing. Break one to make room for the AI and you will hear about it within a day.

And your interface has no vocabulary for uncertainty. Everything in it is stated as fact. Drop a probabilistic output into that grammar and the user reads it as fact too, which is exactly the failure mode you cannot afford.

Put the first feature where the pain already is

The instinct is to put AI on the flagship workflow, the one in the demo, the one the CEO shows investors. It is the wrong call.

Put it on the step your users already dread and already do badly. The manual tagging. The report nobody writes. The thirty-minute reconciliation at the end of the month.

Two reasons. The bar is low, so a mediocre first version still beats what exists. And if it fails, it fails somewhere that was already broken, which costs you nothing in trust.

Ask your support team, not your product team. Support knows which screen generates tickets.

Decide what you are not going to automate

This is the part that gets skipped and it is the part that decides the outcome.

Anything where being wrong is expensive and hard to detect should stay manual. Anything the user is personally accountable for should stay manual, or at least stay editable before it leaves the building.

I have had clients push back on this. The argument is always the same: if we can automate it, why wouldn't we. Because the value of an AI feature is not how much it does. It is how much of what it does the user can safely stop checking. Automate something people still have to verify line by line and you have added a step, not removed one.

Your AI will be wrong. Design that screen

Most teams spend ninety percent of the effort on the happy path, which the model handles anyway.

The state that matters is different: the output looks plausible and is not. There is no error, nothing red, nothing to catch the eye. Just a confident answer that happens to be false.

So the question I put to every team before a line of UI gets drawn: what does the user see at that moment, and how do they check it in one action?

Usually the answer is a source. The rows the summary came from, the date range it used, the filter it applied. One click to the underlying data. That single affordance does more for adoption than any improvement in the model, because it converts a leap of faith into a verification.

Make correcting cheaper than redoing

If fixing one detail means regenerating the whole thing, your users will stop using the feature. Not immediately. Gradually, and without telling you.

The rule I work to: editing the output has to cost less than producing it by hand. If the AI drafts something, the draft must be editable in place. If it categorises something, the category must be changeable in one click, not through a settings panel three levels down.

And when a user corrects the output, that correction should be visible to them afterwards. People need to see that their input mattered, otherwise they conclude the system is not listening and go back to the old way.

Show reasoning, not a log

Transparency has become a slogan and it usually gets implemented badly. Teams either show nothing, or they dump a chain of intermediate steps that nobody reads.

What a user actually needs is enough to decide whether to trust this result. Which source. Which period. Which records. Three lines, not thirty.

The test is simple: if a user cannot tell from your explanation whether to trust the answer, the explanation is decoration.

AI inside a data-heavy interface

Most of the products I work on are dashboards, and this is where AI usually lands: a summary at the top, an anomaly flag, a natural-language query box.

The trap is that a dashboard already has a hierarchy problem before AI arrives. Everything is shown at equal weight, nothing calls for action, and the user reads instead of acting. Adding an AI summary to that does not fix it. It adds one more block competing for the same attention, with the added risk that this one can be wrong.

Fix the hierarchy first. Decide the three questions the user opens the product to answer, fit those answers on one screen, and push everything else down a level. Then ask whether the AI summary earns its place above the fold. Often the better move is to let AI handle the thing underneath the surface, the anomaly detection or the segmentation, rather than the headline.

I go deeper on that in the dashboard patterns piece.

Do not break what already works

Every AI feature needs an exit. A way to do the same job by hand, findable, not buried.

Teams remove it because they want adoption numbers. What they get is resentment, and a support queue full of people asking where the old button went. Adoption you force is not adoption, it is a countdown.

Keep the manual path for at least two release cycles after launch. If the feature is good, usage migrates on its own, and then you have a real number instead of a coerced one.

Scope the pilot properly

One workflow. One question you are trying to answer. One number that would tell you to stop.

Most teams define success and never define failure, which means the pilot can never conclude anything and quietly becomes permanent. Decide up front what result would make you kill it.

And fake whatever is not the thing you are testing. Manual onboarding, a human in the loop, a spreadsheet instead of a pipeline. If the mechanism is not what you are validating, do not build it.

What it costs, and where the money goes

The model is the cheap part. The budget goes into deciding what not to automate, designing the wrong-answer and correction states, and testing with people who actually use the product.

A pilot on one workflow is a matter of weeks. A full integration across a mature product is longer, and the length is driven by how much existing behaviour you are disturbing, not by how clever the AI is.

If you want the equivalent numbers for interface work, the redesign cost breakdown covers it.

The short version

Adding AI to an existing product is not a modelling problem. It is an integration problem wearing a modelling problem's clothes.

Decide where it goes. Decide what stays manual. Design the moment it is wrong, and make the correction cheap.

Do that and the model almost stops mattering. Skip it and no model will save you.

If you are weighing this up for your own product, the platform guide covers the architecture side, and I am happy to look at your case directly.