A client we had worked with in the past recently came back to us looking for advice on an AI agent workflow they had implemented.
The agents were handling roughly 80% of the standard cases without human intervention, which looked promising at first.
But the remaining 20% was starting to change the economics.
Those cases were being pushed into an exception queue, where people had to review outputs, resolve ambiguities, and correct edge cases. As the queue grew, so did the amount of human work around the automation.
Some of the savings they had gained by automating the process were starting to disappear into exception handling.
What they wanted help thinking through was why so many cases were reaching the queue in the first place, and whether some of that human review could be avoided.
Not every case required real human judgment. Some of the human review was not there because the case was inherently complex. It was there because the uncertainty had never been resolved upstream.
That 80/20 split is anecdotal, not an industry benchmark. But it raises a much broader question:
How much human intervention can an AI workflow absorb before it starts undercutting the savings it was supposed to create?
Human-In-the-Loop Has a Cost
Human-in-the-loop, or HITL, is an important part of many agent architectures.
There are decisions we should not fully delegate. High-risk actions, unusual customer situations, regulatory requirements, irreversible transactions, and genuinely ambiguous cases may all justify human review.
The problem starts when too many cases routinely require human intervention.
Every review adds labor. It can also add latency, create queues, and limit how quickly the workflow can scale.
And if we want to understand what it really costs to automate a business process with agents, all of that needs to be part of the math.
The calculation is not simply:
Human cost vs. Agent cost
A more realistic comparison is:
Cost of the current process vs. Cost of the agentic workflow
And that agentic workflow may include:
Inference + retries + exception handling + human review + integrations + monitoring + maintenance
That is the number that ultimately tells us whether automating the process with agents actually makes economic sense.
CloudZero makes a similar argument in its analysis of HITL economics: pricing the agent alone can significantly understate the cost of the production system because reviewer labor and operational overhead remain part of the workflow.
There is another problem. As the review queue grows, the business may need more reviewers to preserve the same level of scrutiny.
The agent scales like software.
The supervision layer can start scaling like headcount.
But Not Every Exception Is Really an AI Problem
Before adding more reviewers, there is a more useful question to ask:
Why is the agent escalating the case in the first place?
Some exceptions legitimately require judgment.
Imagine a procurement agent processing purchase requests. A request significantly above normal thresholds, involving an unusual supplier or creating a compliance risk, may deserve human approval.
But other exceptions look very different.
- Two systems contain different addresses for the same customer.
- Finance and Sales use different definitions of an active customer.
- The agent retrieves two versions of the same policy.
- An old document ranks above the current one.
- The database contains the data, but the business definition needed to interpret it exists only in someone’s head.
These cases may end up in a human review queue, but they are not necessarily failures of AI reasoning.
They are often data problems appearing downstream as HITL problems.
When Poor Data Turns Humans into the Resolution Layer
A recent CIO.com article gives a good example of how this can happen.
In this case, an agent system was pulling customer information from several APIs. The same customer field could come back with different values depending on the source, exposing the agent to semantic ambiguity.
In simple terms, semantic ambiguity appears when the data is available, but its business meaning has not been fully defined. For example, if one source contains a billing address, another the most recently verified address, and another the address used in the latest transaction, what exactly should “the customer’s address” mean for this particular task?
Without a business rule defining that meaning and identifying which value should be treated as authoritative, the agent is left to interpret it on its own.
At first, the agents were expected to resolve that ambiguity themselves.
That didn’t work consistently.
So the team moved the decision upstream. They added business rules to reconcile the records before the data ever reached the agents, giving them one approved value to work with.
It sounds like a small architectural change, but in production it makes a big difference.
Without resolution:
Multiple values → Agent interprets → Uncertainty or error → Human reviews
With resolution:
Multiple values → Business rule resolves ambiguity → Agent receives trusted data
The second approach removes an entire category of potential exceptions.
And that matters for more than just reducing the headcount needed for human review.
Human attention is limited. The more exceptions reviewers have to work through, especially repetitive ones, the harder it becomes to give each case the same level of attention.
That creates two problems.
First, reviewing hundreds of routine cases can create fatigue. The reviewer becomes used to seeing the same patterns over and over, making it easier for the genuinely unusual case to slip through.
Second, highly accurate AI can create a different kind of problem: the reviewer starts expecting the AI to be right.
Eric Billingsley, COO and CTO of TrustScale, makes this point in the CIO article. When a system is correct most of the time, reviewers can gradually shift from independently evaluating each decision to simply confirming what the system has already suggested.
After hundreds of correct recommendations, the rare mistake becomes easier to approve automatically.
Trusted Data Can Shrink the Exception Queue
When it comes to data, there is no single “trusted data layer” that solves every agent problem.
Different types of ambiguity require different mechanisms.
- Data reconciliation can determine which value should be used when several applications return different information for the same field.
- Semantic layers can encode agreed business definitions for metrics and entities such as ARR, margin, churn, customer status, or qualified opportunity.
- Governed retrieval can establish which documents take precedence based on source, version, customer, geography, or recency.
- Metadata and data contracts can make ownership, meaning, freshness, and expected quality explicit before the agent acts on the data.
The implementation changes depending on the problem.
The objective does not:
Resolve predictable ambiguity before the agent encounters it.
That can have a direct impact on the economics of HITL.
A stronger data architecture can reduce avoidable exceptions, so human review is reserved for cases that actually require judgment, rather than becoming the fallback when the agent lacks context or a clear rule.
That means less review overhead, less latency, and more human attention available for the cases where it actually matters.
Human intervention rate should become an AI FinOps metric
Most conversations about AI costs still start with models and token consuption.
- How many tokens did we consume?
- Which model handled the request?
- How much did inference cost?
Those metrics are important, but they are not enough to get a realistic sense of what it will actually cost to implement an agent in a business process. And that is a fundamental part of putting AI into production.
One of the biggest risks is successfully automating a process, only to realize later that the automation costs more to operate than the previous workflow, when there were no agents involved and people were simply handling the work themselves.
That is why we also need to account for everything around the agent: the people involved, potential bottlenecks, delays, exceptions, monitoring, and maintenance.
For example, if a $0.50 execution creates ten minutes of work for an expensive specialist, looking at the inference cost alone gives us a very incomplete picture of the economics.
Teams operating agents at scale should also start measuring things like:
- Human Intervention Rate: What percentage of executions require a person?
- Exception Rate: How often does the agent leave the expected path?
- Cost per Exception: How much does each escalation cost to resolve?
- Cost per Successful Task: What does the complete workflow cost after inference, retries, review, and rework?
- Repeat Exception Rate: How many escalations come from problems the organization has already seen before?
Looking at all of these costs gives us a much better basis for understanding whether agents are actually cost-effective for that process, or whether a simpler and cheaper alternative, such as predictive AI or traditional automation, may make more economic sense.
It’s not about taking people out of the equation
The point of agentic AI is not to leave agents running wild with no human oversight.
Engineers and domain specialists still have an important role to play when a case involves real uncertainty, business judgment, compliance, or higher-risk decisions.
The problem starts when their time gets absorbed by an endless exception backlog because the data, business rules, or architecture around the agent are not working as they should.
At that point, the economics can stop making sense.
The challenge is finding the right balance between what agents can handle reliably and where human supervision is actually worth the cost.
If you are evaluating an agentic AI use case, we’d be glad to learn more about the process and help you assess where it can be optimized.
Book a discovery call and tell us about your business case. We can help you evaluate the workflow, the role of HITL, and the overall economics so the process can scale without adding unnecessary cost and latency.