The AI harness is starting to get a lot more attention.
OpenAI, Anthropic, NVIDIA, and others are increasingly talking about the piece of software that wraps around the model and determines how an AI system actually works.
That matters because the model tends to get most of the attention.
But thinking that the model is the whole AI system is a little like thinking the graphics engine is the whole video game.
The engine matters, of course. But so do the rules, the narrative, the art direction, and the mechanics that determine what the player can actually do.
AI is beginning to look a lot like that.
In this article, we’ll look at what the AI harness is, what harness engineering involves, and where its importance really comes from. We’ll also explore how it relates to AI gateways and model routing, when companies may want to build their own versus use a vendor solution, and how this emerging layer connects with two topics we’ve been discussing lately on this blog: AI governance and token economics.
First, let’s define what we mean by the AI harness.
What Is the AI Harness?
At its simplest, the AI harness is the software built around the model that helps turn its reasoning into a working system.
David Pan, director and AI industry practice lead at Moody’s, put it well in a recent conversation with The Wall Street Journal: the model is the brain, while the harness is what allows companies to control and direct that brain.
A simple example helps.
Imagine a customer is logged into a company’s support portal and writes:
“My order never arrived. Can you refund it?”
The application already knows who the customer is. The harness can use that customer ID to retrieve the information the model needs, such as the order status, purchase amount, and refund policy.
It can then give the model the relevant context:
Order #45821
Status: Delayed
Value: $85
Refund policy: Eligible after 10 days
The model can reason over that information and determine whether the customer qualifies for a refund.
But that is only part of the job.
The harness can also control what happens next. It may check whether the agent has permission to issue the refund, call the appropriate system to process it, or apply a rule such as:
Refund under $100 → process automatically
Refund over $100 → require human approval
So, the model is doing the reasoning, while the harness is bringing together the context, tools, rules, and actions needed to turn that reasoning into a real business process.
That surrounding layer can include memory, business data, tools and APIs, permissions, approval rules, workflow state, logging, and even logic for choosing which model should handle a particular task.
And harness engineering is simply the work of designing and building that layer around the model.
Why Separate the Harness from the Model?
Companies, and some of the most prominent leaders in tech, are starting to look at the AI harness as a strategic layer for protecting data, context, and the operating knowledge that builds up around AI systems.
A company can use the harness provided by a model vendor. But it can also choose to keep that layer under its own control and use models such as GPT, Claude, Gemini, or open-weight alternatives underneath it.
Microsoft CEO Satya Nadella recently argued for exactly that separation. His point was that companies should keep the harness, context, memory, and usage metadata separate from the underlying model so they can preserve control over how AI is actually used inside the organization.
It is important to make one clarification here.
The harness is not a repository of employee prompts. Prompts are simply one of many inputs that may pass through the broader AI system, alongside business data, retrieved documents, tool results, workflow state, permissions, and other context.
And retaining useful AI context does not mean companies need to record every employee conversation or monitor individuals.
In many cases, what matters is the operational knowledge that can be derived from AI usage, with the appropriate privacy, access, and retention controls in place.
For example, a company may learn at an aggregate level that:
- Finance frequently uses AI to compare invoices with purchase orders.
- Legal repeatedly asks AI to identify certain clauses in contracts.
- Sales often combines the same account data before preparing for customer meetings.
The objective is not to know which individual employee wrote each prompt.
The useful information is that these recurring patterns can reveal where AI is being used, what context employees need, and which manual processes might be candidates for a more structured workflow or agent.
Over time, those patterns can guide how the company builds its own agents.
Model Portability Is Part of the Strategy
Owning more of that surrounding layer can also reduce dependence on any single model provider.
That is a central part of Nadella’s argument. If the harness, context, and memory remain separate from the model, companies can use different models for what each one does best. And if one provider becomes too expensive, falls behind, or is no longer the right fit, the company does not have to rebuild the surrounding system from scratch.
The architecture can look more like this:
This separation gives companies something increasingly valuable: the ability to change the intelligence underneath without giving up the operating layer they have built around it.
We explored this question in more detail in our article on AI governance and model portability. As AI environments expand, governance is not only about knowing which agents exist, who owns them, and what they can access. It is also about making sure that the organization can preserve the context and learning that accumulate as those agents run.
Harness vs. Scaffolding: Are They the Same Thing?
Before going further, there is another distinction worth making.
Harness and scaffolding are often used interchangeably, but they do not always mean exactly the same thing.
The terminology is still evolving. Hugging Face, for example, separates the two concepts more precisely. It describes scaffolding as the behavior-defining layer around the model: the system instructions, tool descriptions, response formats, and context-management choices that shape what the model sees and how it approaches a task.
The harness, in the narrower sense, is the execution layer that actually runs the process. It calls the model, handles tool requests, sends results back, manages the loop, and decides when the agent should continue or stop.
A simple way to think about the distinction is that scaffolding shapes how the model works, while the harness makes that work run.
In practice, however, many vendors and developers use harness more broadly to describe almost everything built around the model. Claude Code and Codex are often discussed this way. Hugging Face explicitly notes that this broader usage is common, even if separating harness from scaffolding can be useful when discussing the architecture in more detail.
For this article, we use AI harness in that broader enterprise sense, while keeping in mind that the surrounding architecture can include several distinct pieces.
This is changing how teams fix agent failures
This broader view is also changing the way teams think about agent failures.
Until recently, when an agent made a bad decision, got stuck, or failed to complete a task, the easiest explanation was often that the model simply was not capable enough.
The response was usually to wait for a better model.
Harness engineering introduces a different way of looking at the problem.
Instead of asking only “Is the model good enough?”, teams are starting to ask whether the system around it gave the model the right conditions to succeed.
In an O’Reilly article on harness engineering, Addy Osmani puts it particularly well:
“A decent model with a great harness beats a great model with a bad harness.”
His argument is that many failures that look like model limitations can actually be traced back to the surrounding system: missing context, weak tools, poor task decomposition, insufficient feedback, or a lack of enforceable constraints.
That is where the methodology of harness engineering becomes important.
If an agent repeatedly ignores a rule, that rule can become an enforced control. If it gets lost during a long task, the workflow can be split into planning and execution stages. If it finishes work before validation is complete, the harness can require tests or checks before allowing the task to close.
In that sense, harness engineering is not only about building the layer around the model. It is also about continuously improving that layer based on the failures the agent actually produces.
Osmani describes this as a kind of ratchet: recurring mistakes become opportunities to improve the harness so the same failure becomes less likely to happen again.
How Does the Harness Relate to AI Gateways and Model Routing?
This is another area where the concepts can easily blur together.
The harness, AI gateway, and model router are not the same thing, although their functions can overlap.
The harness is primarily concerned with how the agent performs its work. It manages the interaction between the model, context, tools, rules, and execution process.
An AI gateway plays a different role. It centralizes access to AI models, so applications and agents do not need to connect independently to OpenAI, Anthropic, Google, or another provider.
That gateway can then help the company manage approved providers, authentication, usage policies, logging, quotas, and other controls across multiple AI applications.
Model routing solves a narrower problem inside that architecture: deciding which model should handle a particular request.
Imagine our finance agent needs to perform several tasks during the same workflow. A simple document classification might be perfectly adequate for a smaller model, while a complex exception may require a more capable reasoning model.
Routing does not necessarily require a separate product. It can live directly inside a harness.
But the larger an AI environment becomes, the more useful it can be to centralize that capability.
If 50 different agents each contain their own model-selection rules, changing those policies becomes difficult. If those agents instead use a common gateway and routing layer, the organization can change the policy once and apply it more consistently.
So we can think about the relationship this way:
- The harness manages the agent’s execution.
- The router chooses an efficient model for the task.
- The gateway can centralize model access and those policies across many apps.
Do Companies Need All of These Layers?
Not necessarily.
If a company has a simple application that summarizes documents using one approved model, there may be little reason to introduce sophisticated routing or a centralized gateway.
But the requirements change when an AI system starts accessing business data, using several tools, making decisions across multiple steps, or taking actions in enterprise systems.
They change again when dozens of applications and agents are using multiple model providers across different departments.
The architecture should therefore follow the complexity of the use case.
A simple copilot may need very little harness logic. A long-running agent operating across SAP, Salesforce, email, internal databases, and external APIs needs a much more deliberate execution environment.
And an organization running hundreds of those agents may eventually need centralized gateways, routing, observability, and governance around them.
Build Your Own Harness or Use a Vendor Solution?
Once the harness becomes important, another question follows naturally:
Should companies build their own?
There is no single answer.
Model providers are increasingly packaging more of the surrounding infrastructure themselves. Claude Code and Codex, for example, provide much more than access to an underlying model. They include agent loops, tools, context management, and other infrastructure that would otherwise have to be built separately.
Hugging Face points out that some of these harnesses are tightly coupled to a particular provider, while others are designed to work with different models underneath.
For many organizations, using one of these existing environments will make sense. There is little value in rebuilding standard infrastructure if the vendor solution already provides the capabilities the company needs.
But there are also cases where owning more of the harness becomes strategically useful.
Moody’s is one example. According to The Wall Street Journal, the company built its own harness around applications. That allows it to connect its own datasets while retaining the ability to use different models on the backend.
Bristol Myers Squibb took a different path. The company chose Claude as a standard harness rather than rebuilding basic agent infrastructure itself.
Both approaches can be rational.
A vendor-built harness can reduce engineering work and accelerate deployment. A more custom approach can give the company greater control over proprietary workflows, context, tool permissions, model portability, and how the system evolves over time.
The question is therefore less about build versus buy in absolute terms and more about identifying which parts of the surrounding system are sufficiently differentiated, sensitive, or strategic that the company wants to own them.
Where the Harness Meets AI Governance
This brings us to one of the areas where the harness becomes particularly important for enterprises: governance.
We have written before about why AI agent governance becomes more difficult as companies deploy more autonomous systems. Organizations need visibility into which agents exist, who owns them, what systems they can access, and how much autonomy they have.
But defining the policy is only part of the problem.
The policy also has to become executable.
Suppose the company establishes a rule that payments above $10,000 require human approval.
Writing that rule in a governance document does not prevent an agent from taking the action.
The surrounding architecture has to enforce it.
The harness can check the payment amount during execution and stop the process when the threshold is reached. It can then request approval and resume the workflow only after that approval is received.
The same idea applies to permissions.
A customer-service agent may be allowed to read an order history and issue a small refund, but it may have no reason to change a customer’s bank information. A finance agent may be able to prepare a payment but not release it.
The model can reason about what it thinks should happen.
The harness helps determine what the agent is actually allowed to do.
That distinction becomes increasingly important as agents move from generating recommendations to interacting directly with business systems.
The governance question is no longer just:
What did the model say?
It also becomes:
What could the agent access, what actions could it take, and what happened when it tried?
This is where observability also enters the picture.
If an agent modifies an important record, companies may need to reconstruct which model was used, what context it received, which tools it called, whether an exception occurred, whether the system retried, and whether a human approved the final action.
O’Reilly includes logs, traces, cost, and latency metering among the components that can sit inside a mature harness.
In other words, governance establishes the boundaries, while the harness is one of the places where those boundaries can be enforced during execution.
Why the Harness Can Improve Performance Without Changing the Model
Governance is only part of the reason the harness is attracting attention.
The more surprising argument is that changing the system around a model can materially change what the model is able to accomplish.
NVIDIA recently demonstrated this..
Researchers tested Claude Opus 5 on ARC-AGI-3, an interactive reasoning benchmark in which an AI system has to understand and complete unfamiliar 2D games.
The model initially scored 30%.
NVIDIA then introduced a custom harness with better memory management and a supervising agent that could intervene when the primary agent became stuck or started moving in an unproductive direction.
The result reached 100% on the benchmark without changing the underlying model.
The example is unusually dramatic, and it should not be interpreted to mean that every company can triple the performance of every AI workload by changing the harness.
But it illustrates an important principle.
The model may already possess capabilities that the surrounding system is not using particularly well.
- Better context management can prevent it from becoming distracted.
- Better tools can give it more useful ways to act.
- A stronger feedback loop can help it recognize when an approach has failed.
- A supervisor or evaluator can keep a long-running task from drifting away from its goal.
This is especially relevant for long-horizon tasks, where an agent needs to maintain coherence across many decisions rather than simply answer one prompt.
A raw model may be able to reason through each individual step and still fail at the complete workflow.
Harness engineering tries to close that gap.
Efficiency Also Depends on the Harness
The same principle appears when we look at efficiency.
Databricks recently benchmarked coding agents on real tasks taken from its multi-million-line codebase and found that the model alone was not a reliable predictor of how efficiently those tasks would be completed.
One of the study’s conclusions was that the harness had a major impact on both cost and quality.
More interestingly, Databricks ran the same model with the same reasoning effort through different harnesses.
In some cases, the cost per completed task differed by more than 2X even though quality remained the same.
One harness sent roughly three times less context on each turn, maintained a tighter working set, and completed tasks in fewer runs.
That has an important implication for how we think about AI efficiency.
A model has a published token price, but the harness influences how much context gets sent, how many calls are made, how often the agent retries, and how long the execution loop continues before the task is actually finished.
So two systems built on the same model can have very different operational profiles.
This is also why looking only at price per token can be misleading.
Databricks found that a model that was approximately 1.7 times cheaper per token could still end up costing more per completed task because it consumed significantly more tokens while working through the problem.
For enterprises, the more useful metric is therefore not necessarily:
How much does this model cost per million tokens?
It may be:
How efficiently does this model and harness combination complete the business task?
Model Routing Adds Another Efficiency Lever
Model routing takes that idea one step further.
Not every task requires the same level of intelligence.
Databricks found that its engineers perform work across very different levels of complexity. Routine operational tasks often did not require the most capable models, even though premium models had frequently been used as the default.
That creates an opportunity for the surrounding AI architecture to make another decision:
Which model is sufficient for this particular step?
- A simple classification task may work well with a smaller model.
- A complicated legal analysis may require a frontier reasoning model.
- An exception that cannot be resolved confidently might escalate to a more capable model or to a person.
This does not mean routing every request to the cheapest option.
It simply means matching the model to the work.
And once again, the harness, gateway, and router can work together.
- The harness understands where the agent is in the workflow.
- The router selects an appropriate model.
- The gateway can apply those choices and policies consistently across multiple applications.
This is where token economics becomes an architecture problem, rather than simply a procurement question about which model has the lowest published price.
The Bigger Shift Is in How We Engineer AI
The terminology around AI agents is still evolving. Harness, scaffolding, runtime, gateway, routing: some of these concepts are relatively new, and the boundaries between them are not always perfectly defined.
But the bigger takeaway is not the vocabulary.
For a long time, much of the AI conversation revolved around model benchmarks and model pricing: which model was more capable, which one scored higher, and how much those tokens would cost at the end of the month.
That discussion is becoming much broader.
As agents move into production, it is increasingly clear that the engineering around the model can directly influence how well an AI system performs.
That is also how we have been approaching AI implementations with our clients and partners. In our recent Document AI workshops with SAP, for example, the goal has not been simply to demonstrate what an AI capability can do. We work from a concrete business process, look at the architecture and integration requirements, quantify the potential ROI, and define a PoC that can prove whether the use case is worth scaling.
The same approach applies to any other AI initiative.
If your organization is building an AI roadmap, evaluating an agentic use case, or trying to move from experimentation into production, book a discovery call with us. We can help you define the right PoC, evaluate the architecture around it, and provide the certified resources needed to accelerate the implementation.
