AI agent governance is, without a doubt, one of the biggest challenges companies will be dealing with over the next few years.
So far, much of the effort around governance has focused on mapping the AI tools already in use and, to some extent, trying to get ahead of shadow AI. Basically, IT departments are working hard to understand what AI models and agents are being used across the organization, who is using them, what they have access to, and whether they are operating within company policies.
But Satya Nadella, Microsoft CEO, recently brought up another aspect that is much less discussed: What happens to all the knowledge a company generates, day after day, while using AI?
In July, Nadella published an essay titled The Reverse Information Paradox. His argument starts from an interesting premise. Companies already pay for AI every time they consume tokens. But using those models effectively also requires them to continuously feed in something of their own: business context, prompts, workflows, feedback, corrections, and, increasingly, very detailed information about how the company works.
In Nadella’s words, companies can end up paying for intelligence twice: once with money and again with the proprietary knowledge they provide to make that intelligence useful.
A few weeks later, during an interview with CNN’s Fareed Zakaria, he went a step further. Nadella argued that companies should retain the metadata generated every time they use a model, partly because that history could eventually be used to improve their own AI systems, adapt open models, or even train their own weights.
For Nadella, the harsh truth is that a company that gives up control over this layer has, in his words, “essentially outsourced your thinking.”
There is clearly some self-interest behind the argument. Microsoft sells much of the cloud and AI infrastructure needed to build the kind of architecture Nadella is advocating for. But that doesn’t make the underlying question less interesting.
In fact, his argument helps explain why architectures such as AI gateways, model routing and orchestration layers, as well as less familiar concepts like the AI harness, are becoming part of the governance talks.
So, in this article, we are going to unpack some of Nadella’s ideas and opinions and look at the architecture behind them. The goal is to understand why he thinks companies should pay more attention to this less explored side of AI governance.
What Does Nadella Mean by the AI Harness?
When we use ChatGPT or Claude, it is easy to think of the model as the product.
Inside a business application, things are usually more complicated.
Take a coding agent. The model can read and generate code, but something has to give it access to a repository, decide which files it can read, maintain the context of the task, execute commands, run tests, interpret the results, and decide whether another model call is necessary.
That surrounding system is often called the harness.
The same idea applies to a procurement agent. The model may help reason through an invoice exception, but the harness connects that reasoning with supplier information, SAP, internal policies, previous steps in the workflow, and whatever tools the agent is allowed to use.
This distinction is central to Nadella’s argument. He has been advocating for companies to keep the harness, context, and memory separate from the underlying model.
A very ordinary business scenario explains why.
Imagine a company uses Model A for thousands of customer service interactions every day. The model works well, employees are happy with the results, and the average cost per completed case is within the company’s target.
Then the economics change.
Maybe the provider increases its token prices. Maybe the agent starts processing longer conversations and consuming substantially more tokens. Either way, the company calculates that the cost per case is now higher than it is willing to pay.
There is another model on the market that looks almost as good and costs substantially less.
How difficult is it to switch?
That depends largely on what the company built around Model A.
If its applications rely heavily on the provider’s proprietary memory, agent framework, tools, or other model-specific capabilities, replacing it can quickly turn into an architecture project.
If the company controls more of those layers itself, it has a much easier starting point. It can run Model B against its existing evaluations, compare performance on real business cases, and decide whether the savings justify the change.
Perhaps Model A remains necessary for the most difficult customer interactions while Model B handles the majority of routine requests. Perhaps a smaller open model turns out to be good enough for another category entirely.
The company doesn’t need every model to behave identically. It needs enough separation to preserve the ability to choose.
And that ability has a direct relationship with governance.
What Do AI Gateways Have to Do with It?
AI gateways are one of the architectural layers that can help companies keep that separation with the AI providers.
This layer also addresses two interlocked problems that are already becoming familiar to IT departments: different parts of the company can quickly end up building around completely different AI ecosystems. What’s popularly known as Shadow AI and AI Sprawl:
Marketing → Provider A
Finance → Provider B
Engineering → Provider C
Customer Service → Provider D
That can make an already difficult problem even harder: governing agents in a multi-LLM landscape.
With an AI gateway, the company can support several models while keeping access within an approved architecture under its control.
Marketing could still use Model A. Finance may have a legitimate reason to use Model B. The same goes for Engineering with Model C. The difference is that those connections can now pass through a common governance layer where the company can decide which providers are approved, apply policies, retain usage records, and understand what different teams are spending.
This is also a more realistic way to deal with Shadow AI than simply trying to block every alternative tool. More importantly, giving teams several approved options can reduce the incentive to create direct, unsanctioned connections outside IT’s visibility.
The gateway does not discover or eliminate every instance of Shadow AI by itself. But it can give companies a controlled way to support the model diversity that is already emerging across departments.
Model routing solves the next problem. Once several models are available, the company needs to decide which one should handle a particular task.
That decision might depend on complexity, cost, latency, context requirements, or the type of work being performed.
That gives the company room to change providers without asking every department to rebuild the applications and workflows around them. It also explains why open models increasingly enter these conversations. Companies do not necessarily need to replace proprietary models with open ones, but having that option changes the economics and reduces dependence on any single provider.
Nadella’s point is largely about preserving that freedom to choose.
The Knowledge Building Up Behind Every AI Interaction
Every time employees interact with AI, and every time agents carry out a workflow, they leave behind a trail of information about what happened.
An employee who corrects a response or discovers that a prompt works better with a particular piece of context. A finance analyst who rejects a recommendation and explains why. An agent may call one tool, fail, try another, and eventually complete the task successfully.
Over time, teams also start building evaluations around the mistakes they see repeatedly. And, according to Nadella, all of this could be capitalized on at scale by AI providers, potentially giving them valuable insight into how a company works, how its employees make decisions, or even how its products are built.
He is particularly wary of model providers that, in his words, “reserve the right to learn from customer usage and interaction data.” His point is that prompts, tool usage, feedback, and especially the corrections employees make can contain knowledge that took a company years to develop.
Think about a company using coding agents across its engineering teams. A developer gives an agent access to a repository; the agent reads files, proposes changes, calls tools, runs tests, and receives feedback when something is wrong. The developer may reject part of the solution, provide more context, or correct the approach before the code is finally accepted.
One interaction may not say much. But repeat that process thousands of times across engineering teams and the company starts accumulating a detailed history of what worked, what failed, how experienced developers corrected the model, and so on.
This also connects back to Nadella’s argument about keeping the company’s context, memory, harness, and learning layer separate from commercial models. That separation matters because it allows the model underneath to change without forcing the company to rebuild everything around it.
If that information remains under the company’s control, changing providers does not mean leaving years of accumulated learning behind. The company can take that context, evaluation history, and feedback to another provider, use it to fine-tune an open model, build more specialized models around its own workflows, or eventually develop proprietary weights of its own.
Does Using an External Model Mean Giving That Knowledge Away?
This point needs some nuance.
Using an enterprise AI provider does not automatically mean that everything employees type will be used to train the provider’s next model. Enterprise agreements often include specific commitments about how customer data is processed and whether it can be used for model training.
So the issue is broader than simply worrying that a model provider is secretly learning everything about the company.
It also connects with a wider question around intellectual property protection. In our latest AXIS report, we looked at how software, data, research, and other forms of IP are increasingly being treated as strategic assets, while more vendors, platforms, and external partners interact with them every day. AI adds another point of exposure to that environment.
The report uses the pharmaceutical industry as a clear example. Drug discovery now depends heavily on software, clinical data, simulations, and regulatory systems. Once AI enters those workflows, everyday interactions between scientists and AI systems can become another channel for unintended data leakage if the right guardrails are not in place.
That does not mean companies should assume their AI providers are taking their IP. The practical takeaway is that companies need to think much more carefully about where sensitive knowledge goes, which systems can access it, and how much of that knowledge leaves the environments they directly control.
Even when the provider never trains on the information, the enterprise can still fail to capture it for itself.
That brings us back to the metadata Nadella wants companies to control because, over time, that record can become part of the company’s intellectual property.
Can Model Lock-In Become Knowledge Lock-In?
Vendor lock-in is hardly a new problem.
Companies have spent decades dealing with databases, cloud platforms, ERPs, and other technologies that become increasingly difficult to replace as more applications and processes depend on them.
AI creates another version of that dependency.
Suppose an insurer spends five years using agents across claims processing. During that period, its employees make hundreds of thousands of corrections. The company improves its prompts, develops evaluations, learns which cases require human intervention, and generates millions of agent traces.
Then a better model appears.
If the company retained that history independently, it has a valuable dataset for deciding whether to move. It can run the new model against the same cases, study where it performs better or worse, and gradually transfer workloads.
If most of the useful context and memory exists inside the previous provider’s ecosystem, the decision becomes harder.
The model itself may be replaceable. The accumulated learning around it is not.
This is where Nadella’s warning about “outsourcing your thinking” becomes less dramatic than it initially sounds. He is pointing to the possibility that companies will spend years teaching AI how to participate in their business without keeping enough of that learning under their own control.
What Does This Change About AI Governance?
AI governance still needs to deal with the problems companies are already working on.
IT teams need visibility into the models and agents being used. Security teams need to know what those agents can access. Someone needs to establish ownership, permissions, budgets, and acceptable levels of autonomy.
Those issues are not going away.
Nadella’s argument adds another question to the same governance conversation: what happens to the knowledge created after those systems start working?
The model market is moving quickly. Prices change. New providers appear. Open models improve. Regulatory requirements can shift. And, as we explored in our latest AXIS report, Global Software Delivery in a Fragmented World, geopolitical alignment and the location of critical technology assets are increasingly becoming part of technology decisions as well.
When the company’s context, memory, evaluations, and learning history remain separate from the commercial model, that transition becomes much more manageable. The same knowledge can be used to benchmark another provider against real business cases, fine-tune an open model, or eventually help develop company-controlled models and weights.
That is perhaps the broader implication of Nadella’s warning.
Companies will continue buying intelligence from outside providers. There is little reason to think that will stop.
But every day employees interact with that intelligence, and every day agents execute work around it, companies are creating something of their own.
AI governance also needs to make sure they can keep it.