What Happens When an AI Agent Fails? Designing for the Bad Day

Vendors talk about accuracy. The more useful question is what happens on the occasions the system gets it wrong, because that is where the cost lands.

The failure modes, in order of how bad they are

Silent failure — the worst

The system does something wrong and nobody notices until the consequence arrives. A quote goes out twenty percent low. A booking is not created. An escalation never fires.

This is the failure mode to design against hardest, because the damage compounds while everyone believes things are fine.

Confident wrong answers

The agent gives a caller information that sounds right and is not. Wrong price, wrong availability, wrong process.

Prevented architecturally rather than by better prompting: an agent should retrieve facts rather than generate them. If it quotes from your actual price list and checks real availability, there is nothing to invent. A system that composes a plausible answer from general knowledge will eventually compose a wrong one.

Loops

The agent misunderstands, the caller rephrases, the agent misunderstands again. Three or four rounds and the caller hangs up angry.

Fixed with a counter: two failed attempts at the same thing and it stops trying and escalates. Simple and routinely absent.

Loud failure

The system stops. The agent does not answer, the integration fails, the job does not create.

This is the *least* bad kind because you find out immediately. Most people have their priorities backwards here — an outage is annoying, a silent error is expensive.

What good escalation looks like

Escalation is the most important part of an agent and the least discussed in sales conversations.

It triggers on the right things. Explicit request for a human. Detected distress or anger. Anything outside the defined scope. Two consecutive failures. A keyword list you control.

It reaches someone. Ringing a number that goes to voicemail is not escalation. There should be a defined chain: primary contact, secondary, and a fallback that at minimum captures the details and promises a callback with a time.

It hands over context. The person picking up should see what has already been said. Making a caller repeat everything undoes the goodwill.

It is logged. Every escalation should be reviewable, because escalation patterns are the best available signal about what the agent handles poorly.

The design principles we build to

Fail loudly, never silently. Where a choice exists between stopping and guessing, stop.

Retrieve, do not generate, anything factual. Prices, availability, policies and dates come from your systems.

Bound every action. An agent should be able to do a defined list of things and nothing else. The blast radius of a mistake should be knowable in advance.

Validate before committing. Before creating a booking, check the slot exists and is free. Most silent failures are missing validation.

Log everything a human might need to reconstruct. When something goes wrong at 2am you want to know what happened without guessing.

Set a review threshold. Anything above a value or risk threshold gets human sign-off. You choose the threshold.

What to ask a vendor

1. Show me an escalation happening. Not described — demonstrated.
2. What triggers it, and can I change the triggers myself?
3. What happens if my escalation contact does not answer?
4. How would I find out if the system silently stopped working?
5. What is the largest thing the agent can do without a human approving it?
6. Show me the log of a call that went wrong.

That last one is the tell. Any system that has run in production has calls that went wrong. A vendor who cannot show you one either has no production usage or is not logging properly.

An honest note about our own

Agents we have built have failed. A misheard suburb name that sent a technician to the wrong side of town. An escalation that fired correctly to a number that had been changed and not updated in the config.

Neither was a modelling problem. Both were operational — one needed better read-back on addresses, the other needed a check that escalation contacts still work.

That is the pattern. Failures are usually mundane, and they are found through logs and review rather than better technology. Any vendor claiming their system does not fail is either new to this or not looking.