In the previous articles in this series, we looked at SAP tokenomics from two angles: how SAP AI Units work, and how companies can gain better visibility into AI consumption.
Now it is time to examine one of the key architectural disciplines behind SAP tokenomics: model routing.
It addresses a simple question with a significant economic impact:
Which model should handle each task so that the organization can minimize token and BTP consumption without sacrificing the quality the process requires?
The economic logic is simple: get the best acceptable result while minimizing unnecessary model consumption.
Over the rest of this article, we will look at how SAP teams can put that logic into practice as part of a broader AI FinOps strategy.
A key SAP policy to understand first:
SAP’s 2026 API Policy raised questions because it restricts some generative and semi-autonomous systems that plan, select, or execute sequences of SAP API calls.
The immediate concern was whether companies could continue building custom agents and using external models around SAP.
They can. The policy still allows multi-model applications and third-party AI systems. What SAP controls is how those systems may access its applications and APIs.
SAP determines which architectures, data services, and service-specific pathways qualify as approved routes for agentic access.
Implementations outside those routes may require explicit authorization from SAP. SAP may also monitor API usage and throttle, suspend, or terminate access when it identifies noncompliance.
This is where AI Gateways enter the discussion.
An AI Gateway is a control layer placed between an application and the models it uses. Instead of connecting every application directly to GPT, Claude, Gemini, or another provider, the gateway provides a common entry point.
From that position, the gateway can route requests, restrict model access, apply budgets, record token consumption, enforce rate limits, and move a request to another model when the first option is unavailable or fails to meet the required quality.
SAP Generative AI Hub performs several of these gateway-like functions for applications built on SAP AI Core. It provides governed access to supported generative models and a common environment for model interactions and orchestration.
The word gateway can also describe a layer placed between an AI agent and SAP APIs.
SAP’s policy draws a firm boundary around this second use. A gateway cannot create an alternative route around SAP’s API controls. Passing an external agent through custom code, a proxy, or an AI Gateway does not make its access compliant by itself. SAP still decides which access patterns are permitted and requires them to operate through published APIs and endorsed pathways.
That leaves us with two separate architecture decisions:
1) MODEL ROUTING
Which model should process the task?
2) SAP ACCESS GOVERNANCE
How may the system read SAP data or execute SAP actions?
What Does Model Routing Mean in SAP?
Model routing is an architectural pattern placed between an application and the models available to it.
Instead of connecting every application to one fixed model, the routing layer examines the request and chooses an appropriate destination.
The router may consider the type of task, its complexity, the expected context size, the response time, the risk of an incorrect answer, and the cost of each available option.
The name can make the pattern sound narrower than it is.
A useful router should also be able to decide that the request does not belong in a language model.
Task routing decides which technical approach should handle the work.
Model routing begins once the architecture has decided that a generative model is appropriate.
The two decisions often sit in the same service, but they solve different problems.
The First Question Is Whether the Task Needs a Model
Most discussions about AI cost start by comparing model prices. That skips the first source of waste.
Consider an accounts-payable process.
The system may need to extract an invoice number, find the supplier, confirm that a purchase order exists, compare the total with the available balance, interpret an unstructured note, and explain the exception to an analyst.
These tasks belong to the same process. They do not belong to the same model.
A supplier-number lookup gains little from generative reasoning. A calculation gains even less.
The unstructured supplier note is different. It may contain incomplete language, unusual terminology, or an explanation that cannot be reduced to a fixed rule.
That is where a language model starts to earn its cost.
A practical hierarchy looks like this:
- Level 0 Deterministic rule
- Level 1 SAP query, API, or workflow
- Level 2 Specialized or predictive AI
- Level 3 Small or fast language model
- Level 4 General-purpose language model
- Level 5 Advanced reasoning model
- Level 6 Agentic execution
The architecture should begin with the least expensive level that can perform the task reliably.
This principle matters more than small differences in token prices. Removing an unnecessary model call saves the full cost of the request. Switching to a model that is 15 percent cheaper only saves 15 percent.
SAP Has Two Different AI Cost Layers
Before assigning models, teams need to know which costs they can actually influence.
SAP-delivered AI and customer-built AI follow different commercial models.
1) SAP-delivered AI:
When a company activates Joule or another premium capability delivered by SAP, it normally selects a business function. SAP manages the models and technical components behind that function.
The organization can still control who uses the capability, where it is activated, and how much business volume passes through it.
SAP AI Units provide a common currency for premium AI capabilities. Consumption can depend on metrics such as users, requests, records, documents, or actions, depending on the product.
Model routing has limited influence here because the customer may not control the model that SAP uses internally.
The main cost controls are activation, user assignment, business volume, and process design.
2) Customer-built AI:
The situation changes when a team builds an application with SAP AI Core and Generative AI Hub.
Here the customer can control the model, prompt, context, output length, retry policy, and fallback path.
SAP measures generative AI usage in tokens. It then applies model-specific conversion factors to calculate GenAI tokens and BTP capacity units. Output tokens generally cost more than input tokens, although the exact rates depend on the model.
This is where model routing can directly reduce consumption.
The distinction prevents a common mistake. Routing a custom request from one foundation model to another may reduce BTP consumption. It does not automatically change the AI Units charged for an SAP-delivered feature.
The Cheapest Model May Produce the Most Expensive Result
A model with a low token rate can still be a poor economic choice.
Suppose a fast model costs one-third as much as a general-purpose model. It looks attractive until the team measures what happens after the first response.
The fast model may produce more invalid outputs. It may miss fields, require a second attempt, or send more cases to a human reviewer.
The cost of the task then becomes:
Imagine that two models classify support requests.
The fast model costs 1 unit per initial request and produces an acceptable first answer 82 percent of the time.
The general model costs 3 units and produces an acceptable first answer 96 percent of the time.
The fast model could still be the better first route:
For a routine internal support process, that may work well.
For financial reporting or legal analysis, the same uncertainty could create more review work and more risk than the saving justifies.
The router needs to understand the cost of failure, or at least the business policy that represents it.
What Information Does a Router Need?
So, how does a router decide where a request should go?
It starts by looking beyond the cost of each model. The router needs to understand the task itself, how demanding it is, and what the final result needs to achieve.
That information can be captured in a request profile like this:
- Application: Accounts Payable
- Task: Invoice exception classification
- Risk: Medium
- Required output: Strict JSON
- Maximum latency: 3 seconds
- Expected context: 6,000 tokens
- Human review: Required below 90% confidence
The router can compare this profile with the available model options.
The main criteria are usually the following:
- Task type tells the router whether the request involves classification, extraction, generation, summarization, or reasoning.
- Complexity describes how much interpretation the task requires. A five-category classification is different from comparing three long contracts with conflicting clauses.
- Business risk defines the cost of a mistake. A poorly classified internal ticket may cause a small delay. An incorrect conclusion about an employee payment or regulatory obligation can create a much larger problem.
- Context size affects the number of input tokens. A model with a low token rate may still become expensive when the application sends a long system prompt, several documents, and the complete conversation history.
- Latency matters when the model sits inside a user-facing workflow. A stronger model may provide better reasoning but respond too slowly for a high-volume service process.
- Output requirements matter when the response feeds another system. Elegant prose is of limited use when the application needs valid JSON with four required fields.
- Quality defines the minimum acceptable result. The organization has to decide this before it can compare models properly.
- Cost should include the expected full route, including retries, fallback calls, and human review.
No router can infer all of these requirements from the prompt alone. Some need to come from application metadata and business policy.
The Main Types of Model Routing
Model routing ranges from a fixed configuration to a shared service that classifies requests, evaluates responses, and controls several model providers.
The levels tend to evolve as the number of applications grows.
1) Static model assignment
The simplest version assigns one model to one task or application.
The team tests a few models, chooses one, and references that model in the deployment or orchestration configuration.
This approach works when the task is stable and requests have similar levels of complexity.
It also makes debugging easier. When the output changes, the team knows which model generated it.
The limitation appears when the application begins receiving both simple and difficult requests. Every request still goes to the same model.
2) Rule-based routing
The next level adds explicit conditions.
IF task_type = "classification"
THEN use fast model
IF document_pages > 30
THEN use long-context model
IF business_risk = "high"
THEN use reasoning model and human review
IF deterministic_rule_available = true
THEN do not call a language modelThe application or a shared service evaluates the rules before sending the request.
This approach captures much of the potential saving without adding a separate AI classifier.
It also gives process owners something they can inspect. They can see why a request went to a particular route and challenge the rule when it produces a poor result.
3) Semantic routing
Rule-based routing becomes difficult when users submit open-ended requests and the application has many possible routes.
A classifier can then examine the meaning of the request.
The classifier may use rules, embeddings, traditional machine learning, or a small language model.
Using a premium model to make every routing decision can consume part of the expected saving before the real task starts. The routing step should remain lightweight.
4) Model cascading
A model cascade begins with an economical model and moves to a stronger option only when the first response fails validation.
The validation step carries most of the difficulty.
The system needs a credible reason to accept or reject the response. That reason might be an invalid JSON schema, a missing field, a confidence threshold, a contradiction with SAP data, or a high-risk category.
Without a reliable validation mechanism, the cascade becomes a series of guesses.
How Is Model Routing Implemented?
The basic form is configured manually.
An architect or developer selects a model, creates or references its deployment, and sets it as the default for an application. Changing the model may require only a configuration update because Generative AI Hub provides access through standardized foundation-model and orchestration scenarios.
That first level does not require advanced machine-learning skills.
A developer who works with SAP BTP, CAP, Java, Node.js, or Python can implement a fixed assignment with guidance from an architect.
Rule-based routing moves the decision into application logic.
The code itself is usually straightforward. The harder work sits in the rules.
Someone has to define what counts as a simple task, what level of risk justifies a stronger model, and what should happen when the first result fails.
Semantic routing and cascading require more skills.
The team needs representative test data, quality metrics, evaluation logic, thresholds, and production monitoring. A semantic router also needs a classifier that can separate requests reliably.
At enterprise scale, model routing may become a shared service rather than logic repeated inside every application.
A shared service can enforce common model policies, record consumption, manage fallbacks, and prevent each team from building its own version of the same routing logic.
It also creates a central dependency. The service needs proper ownership, availability, versioning, and change control.
Who builds and maintains it?
Model routing usually involves several roles because the technical team cannot define the economics of a business process on its own.
- The process owner defines what a correct result looks like and which mistakes the process can tolerate.
- The enterprise or solution architect designs the routes, approved models, fallback paths, and review points.
- The application developer implements the routing logic or integrates the application with a shared routing service.
- An AI or machine-learning engineer becomes more important when the design includes semantic classification, automated evaluations, or complex cascading.
- Security and governance teams decide which models may process each type of data and which regions or providers are acceptable.
- The AI FinOps function monitors consumption, retries, escalations, and cost per completed result.
A smaller project may combine these responsibilities across two or three people. A large organization may assign each one to a different team.
The important part is that no single profile can answer every routing question.
The developer can implement a confidence threshold. The process owner has to explain what a low-confidence mistake costs.
The architect can approve a premium model. The FinOps team has to show whether it produces enough improvement to justify its consumption.
What Does SAP Provide Natively?
SAP provides much of the infrastructure that model routing needs.
Generative AI Hub gives applications governed access to supported models. The orchestration service provides a consistent way to call multiple models using shared code, configurations, and deployments.
SAP AI Launchpad allows teams to experiment with prompts and models, save and manage prompts, and compare model configurations. Its evaluation features support benchmarking models and prompts through orchestration configurations.
SAP AI Core also supports allowlists and restrictions that control which models applications may use. This gives the organization a practical boundary for routing. The router chooses among approved options instead of the complete catalogue.
The platform provides:
- Model access
- Standardized interfaces
- Deployments
- Orchestration configurations
- Prompt management
- Model comparison
- Evaluations
- Usage metering
- Model restrictions
SAP does not know the cost of an incorrect classification in the company’s accounts-payable process. It cannot know that a legal team requires human review for one clause type but accepts automated summaries for another.
The organization still has to design that policy.
SAP’s current AI architecture describes fit-for-purpose model selection as an optimization service in its foundation layer. That indicates where the platform is heading, although companies should not assume that a complete business-aware router arrives preconfigured for their processes.
Example: Routing 8,000 Monthly Invoices
Suppose a company processes 8,000 invoices every month.
A simple architecture sends every document to an advanced language model.
The model can probably handle much of the work. That does not make the architecture economical.
A task-based design separates the process.
Assume that 92 percent of invoices pass extraction and deterministic validation.
| Metric | Value |
|---|---|
| Total invoices | 8,000 |
| Invoices requiring an LLM | 640 |
| LLM calls avoided | 7,360 |
| Reduction | 92% |
The router can then divide the remaining 640 exceptions.
Most of the saving came from changing where generative AI entered the process.
The model comparison only affected the smaller set of tasks that actually needed language understanding.
Final Takeaway
Tokenomics is becoming part of the AI conversation in almost every company.
The first phase was about adoption. Teams wanted to test AI, deploy copilots, experiment with agents, and find as many use cases as possible.
Now, companies are reviewing those decisions more carefully.
This tokenomics reconsideration means asking whether every process needs AI, whether every task requires an LLM, and whether the most capable model is always the right choice. In many cases, a workflow, business rule, API call, predictive model, or smaller language model can deliver the required result with less cost and complexity.
SAP provides the infrastructure to support these choices through services such as Generative AI Hub, AI Core, and AI Launchpad. Companies still need to decide how those tools should be used across each process.
At Inclusion Cloud, our SAP certified technical and functional consultants help organizations turn AI ideas into focused PoCs, define where agents and LLMs should enter the architecture, and reduce unnecessary consumption from the beginning.
Book a discovery call with our team so we can understand your needs and explore the most efficient approach for your use case.