In the previous post, we talked about how AI is evolving from simple chat interfaces into agents that can pursue goals, use tools, and act on the information they encounter. We also discussed context: the instructions, conversations, files, and tool results that an agent provides to a model while it works.
But as anyone who manages infrastructure knows, software doesn’t run on magic. It runs on physical hardware. When you ask an AI model to analyze a routing table or review a 10,000-line log file, that context has to be processed somewhere.
That “somewhere” is an inference service.
If you’re planning to deploy AI within your organization (whether to assist staff with day-to-day tasks, purchase access to a hosted service, or stand up an on-premises AI cluster) you need to understand both how inference works and where it takes place.
What is an inference service?
At its core, an inference service is the bridge between a trained AI model and the user or agent asking it to do something.
Think of the model itself as a large mathematical formula stored across one or more files. It contains the parameters learned during training, but those files do not answer questions on their own. An inference service loads the model into memory, accepts a request, uses the parameters to generate a response, and returns the result.
The inference service also handles much of the operational machinery around the model: converting text into tokens, scheduling requests, managing memory, enforcing limits, authenticating users, recording metrics, and sometimes applying safety or policy controls.
When you send a prompt to a commercial AI provider or to a locally hosted open model, an inference service is doing the heavy lifting. The experience may look similar from the user’s point of view, but the infrastructure, governance, and economics behind it can be very different.
Where does inference run?
We often describe AI as either “in the cloud” or “on-premises.” In reality, there is a spectrum of service models between those two points.
Public Consumer Services
The most familiar option is a public service where individuals create their own accounts and interact through a vendor’s website or application. This is easy to adopt and useful for experimentation, but the organization may have limited control over identity, retention, auditing, feature changes, and what information users submit.
This does not automatically make a public service unsafe. It does mean that a free or individually purchased account should not be treated as an organizational AI strategy.
Organization-Managed Tenants
Many vendors offer business, enterprise, education, or government versions of their services. These give an organization a managed tenant with capabilities such as centralized identity, access policies, audit records, retention controls, and contractual commitments about how data is handled.
This is an important distinction. A tenanted service can give an institution much stronger administrative and data protections than a consumer account, even though the vendor still owns and operates the underlying infrastructure.
However, the word tenant does not tell us everything. A service can be logically isolated at the application and data layers while still sharing the same physical accelerators with other customers. Tenancy describes a governance boundary; it does not necessarily mean a dedicated GPU is sitting in a data centre with your institution’s name on it.
Shared API Inference
An organization can also call a model through an application programming interface, or API. Instead of staff using the vendor’s chat interface, an internal application or agent sends requests directly to the provider’s inference service.
This gives developers more control over the user experience and allows AI to be integrated into existing workflows. It is usually metered by the number of tokens processed. The provider manages the model and capacity, and requests from many customers may be scheduled across a shared pool of hardware.
Provisioned or Dedicated Capacity
For organizations with sustained workloads or strict performance requirements, some providers offer reserved throughput or dedicated managed compute. The customer pays for a defined amount of capacity rather than competing entirely for a shared pool.
This can make performance and budgeting more predictable, but “dedicated” still requires careful reading. The accelerators (GPUs) might be reserved for one customer while the provider continues to manage the runtime, networking, storage, and control plane. Other offerings provide a private endpoint but use shared compute behind it. These are different forms of isolation, and the contract and architecture need to say which one is being provided.
Self-Hosted Inference
At the other end of the spectrum, an institution can operate an inference service on infrastructure it controls. This might be a single workstation, a small departmental cluster, a shared provincial service, or a national research computing facility.
Self-hosting provides the greatest control over model selection, access, data location, logging, and network connectivity. It also transfers responsibility for hardware, power, cooling, model licences, security updates, capacity planning, monitoring, and service reliability to the operator.
For NRENs, there is an especially interesting middle ground: a shared, institutionally governed inference service for the research and education community. Such a service could offer many of the operational advantages of a hosted platform while keeping policy, identity, network paths, and data handling under community control.
Tenancy is more than location
When assessing an inference service, “Where is the server?” is only the first question. We should also ask:
- Who operates the model, inference runtime, and surrounding control plane?
- Is accelerator (GPU) capacity shared, reserved, or physically dedicated?
- In which jurisdiction are prompts, outputs, logs, and backups processed and stored?
- How long is that data retained, and can it be used to train or improve another model?
- Which users, vendor personnel, and system administrators can access it?
- Can the service use institutional or federated identity, private network paths, and organization-specific access policies?
- Who is responsible for monitoring, incident response, upgrades, and capacity shortages?
- Can the organization move its workloads and data elsewhere if its requirements change?
None of these deployment models is automatically right or wrong. A public service may be entirely appropriate for low-risk experimentation. A managed tenant may meet the needs of most administrative work. Sensitive research data or privileged operational access may require a dedicated or self-hosted environment.
The important thing is to make an informed decision rather than assuming that every service behind a familiar chat window offers the same protections.
Now let’s talk about how inference works.
The two phases of inference
Regardless of where an inference service runs, it has to perform the same fundamental work. For a text-generating model, we can simplify that work into two phases: prefill and decode.
Phase 1: Prefill
When you first hand a large context window to an agent (for example, an entire firewall configuration and several thousand lines of logs) the inference service must process that input before it can begin answering. This is the prefill phase.
During prefill, the model reads everything you have given it before it starts writing a response. If you provide a short question, this can happen quickly. If you provide a router configuration and thousands of lines of logs, the model has much more material to process.
Think of it like handing an engineer a stack of documents. Before answering your question, they first need time to read and understand the stack. The larger the stack, the longer that initial reading takes.
We measure this waiting period as “time to first token”: the time between submitting a request and seeing the first part of the response appear.
Phase 2: Decode
Once prefill is complete, the model begins generating its response one token at a time. This is the decode phase.
For each new token, the inference service reads the model’s parameters and uses a cache of information calculated for the preceding tokens. This cache, often called the “KV cache” (KV meaning “Key-Value”), prevents the service from recalculating the entire context from scratch each time. Even with that optimization, a great deal of data must move through memory for every generated token.
As a result, decode performance is often constrained by memory capacity and memory bandwidth. This is why AI infrastructure discussions focus not only on raw computing power, but also on High Bandwidth Memory (HBM), accelerator interconnects, model size, and quantization.
The speed of this phase is commonly measured in “tokens per second”.
Why does this matter to us?
If you are an NREN partner or research institution looking to provide shared AI infrastructure, GPU specifications are not enough to guide an architecture or procurement decision.
Interactive assistants, batch document-processing services, and coding agents place different demands on the same hardware. A coding agent might send a large context, generate a small response, invoke a tool, and repeat that cycle many times. A classroom service might receive hundreds of short requests at once. A researcher processing a document collection may care more about total throughput than the delay before any one response begins.
That means you need to evaluate several dimensions together:
- Time to first token — How long does a user wait before a response begins?
- Generation speed — How quickly are output tokens produced?
- Throughput — How many requests or tokens can the service process over time?
- Concurrency— How does performance change when many institutions or users are active?
- Context capacity — How much input can the service accept, and how much memory does that consume?
- Reliability — What happens when a model crashes, hardware fails, or demand exceeds capacity?
- Utilization and cost — Are expensive accelerators doing useful work, or sitting idle while still consuming budget and power?
Therefore, the right design is a workload decision, not simply a contest to buy the biggest GPU.
An emerging architecture: disaggregated inference
We are also seeing active work on “disaggregated inference”, where prefill and decode can be scheduled on different groups of accelerators1. In principle, each group can be optimized for a different workload, and capacity can be scaled independently.
This is particularly interesting to NRENs because we already operate high-capacity networks between research facilities. It raises the possibility of inference resources distributed across multiple sites rather than concentrated in a single cluster.
But this is not as simple as running prefill in one city and decode in another. The intermediate cache can be very large, and transferring it introduces latency, bandwidth consumption, scheduling complexity, and new failure modes. Research projects are exploring how ordinary network protocols and commodity infrastructure might reduce dependence on specialized interconnects2, but the result depends heavily on the model, workload, and distance involved.
For now, disaggregated inference is best viewed as a promising architecture to experiment with, not proof that geographically separated accelerators can work like one local system.
Grounding the cloud
It’s easy to treat AI as a nebulous cloud service. But underneath every agent is a physical and administrative chain: models, software services, accelerators, memory, networks, identities, policies, and people responsible for operating them.
If we want to build AI infrastructure that serves our education and research communities effectively, we have to design for both the physics and the governance of inference.
Up next, we’ll look at another part of that decision: the tradeoffs between proprietary services and open-weight models, and the role each can play in a sovereign AI strategy.
Missed the first blog in this series? Read A primer on agentic AI for network administrators
- For an accessible technical overview of this architecture, see ByteByteGo’s A Guide to AI Inference Engineering. ↩︎
- Teams such as Morph are researching TCP-based approaches to transferring inference state across nodes. These results are promising, but should be evaluated against the latency, scale, and reliability requirements of a specific service. ↩︎