Back to Insights
Enterprise AI3 days agoJustin Pennington

Before You Deploy an AI Agent, Define What It's Allowed to Break

Before You Deploy an AI Agent, Define What It's Allowed to Break

A chatbot that gives a wrong answer wastes a minute of someone's time. An agent that takes a wrong action issues a credit memo, emails a customer, changes a price, or reschedules a delivery. Same underlying model. Completely different risk profile — and completely different design work.

Most teams skip that design work. They pilot a chatbot, like it, then wire it into a system of record and call it an agent. What's actually missing isn't intelligence. It's a boundary.

Answers Are Cheap to Undo. Actions Aren't.

The useful way to think about this: a chatbot produces a recommendation that a human then acts on. An agent collapses those two steps. That collapse is exactly where the value is — nobody wants to pay a person to copy a suggestion into a field — and it's also exactly where the exposure is.

So the first question isn't "what should the agent do?" It's "what happens if the agent does it wrong, and who finds out?"

If the answer is "a customer finds out" or "the accountant finds out at month-end," you don't have an agent problem. You have a scoping problem.

Scope the Blast Radius First

Before you write a prompt, write down the worst realistic outcome of the agent acting incorrectly. Not the catastrophic sci-fi version — the mundane one. It wrote the wrong account code on fifty transactions. It sent the follow-up sequence to a customer who already complained. It approved a purchase order that should have gone to a human.

Then ask whether that outcome is detectable and reversible. Those two properties matter far more than model benchmarks:

  • Detectable and reversible — ship it. Draft replies, internal data enrichment, categorization with a review queue.
  • Detectable but expensive to reverse — ship it behind an approval threshold.
  • Hard to detect — do not ship it until you've built the detection. Silent errors compound.

That single sort will tell you more about your AI roadmap than any vendor demo.

Four Things Every Agent Needs Before Production

When we scope agent work for clients, the same prerequisites come up regardless of industry:

A real system of record to act in. If your order data lives in your ERP, your customer data lives in your CRM, and the truth lives in someone's spreadsheet, the agent will act on the wrong version. Data foundations aren't a philosophical prerequisite — they're the thing the agent reads and writes.

A scoped identity. The agent should have its own account, with its own permissions, in every system it touches. Not an admin key borrowed from whoever set it up. If it only needs to read invoices and write a status field, grant exactly that.

An audit trail that names the actor. Every record the agent touches should show that the agent touched it, when, and on what input. When something looks wrong three weeks later, you need to reconstruct the decision — not guess.

A defined failure mode. What does the agent do when it isn't confident, when an API times out, or when the data doesn't match the pattern it expects? "Try its best" is not a failure mode. "Stop, flag, route to a named human queue" is.

Approval Thresholds Beat Blanket Review

The reflex response to agent risk is to review everything. That's a tax that erases the value you built the agent for, and reviewers stop reading carefully by the second week anyway.

Thresholds work better. Let the agent act freely under a dollar amount, a record count, or a confidence level — and escalate above it. Refunds under a set amount go through; anything larger routes to a manager. New vendor records get created; changes to bank details never do.

This is the same logic you'd use to delegate to a new hire. You don't review their every email. You tell them where the line is and make sure crossing it is visible.

Measure Exceptions, Not Accuracy

Accuracy scores from a pilot rarely survive contact with real operations, because real operations send the weird cases. The metric that actually tracks whether an agent is working is the exception rate: how often does a human have to intervene, and is that number trending down?

A stable or rising exception rate means the agent is absorbing easy work and leaving your team the hard work — which is fine, as long as you know that's what you bought. A falling rate means it's genuinely learning the edges of your business.

Track where exceptions cluster, too. Three recurring exception patterns usually point at one upstream data problem, not three model problems.

Start Narrow, On Purpose

The agents that survive in mid-market businesses tend to be unglamorous: triaging inbound requests, reconciling records between two systems, drafting the first version of something a person finishes. They work because the scope is tight enough that failure is obvious and cheap.

If you're evaluating where an agent could genuinely remove work from your team — or you've got a pilot that demos well but you're nervous about connecting it to your ERP — that's the conversation we have most often. Reach out to Infraxio and we'll map the blast radius with you before anything goes live.