Azure AI agent architecture: from prototype to production

Getting an AI agent to work in a demo takes an afternoon. Getting one to serve real customers, connect to enterprise data, and stay secure in production is a different challenge entirely. Here is how the Azure stack makes it possible.

Getting an AI agent to work in a demo takes an afternoon. Getting one that can securely serve customers, access enterprise data, scale reliably, and run in production without breaking things is a different problem entirely. The gap between those two things is architecture.

We have written more on this via our AI work, and AI Will Not Replace Your Analysts. It Changes The Job takes a closer look at a related part of the picture.

Microsoft Azure is the platform most enterprises are building on when they make that jump. Not because it is the only option, but because it brings all the required layers together in one connected stack, with managed services at each level and identity and security threaded through the whole thing.

Here is how the architecture fits together, and what each layer actually does.

The client layer: where users actually are

Every AI agent starts with a user. On Azure, those users arrive through whatever interface they already use. Microsoft Teams and Microsoft 365 are the most common entry points for enterprise agents, allowing organisations to deploy agent capabilities directly into the tools their people work in every day. Copilot Studio extends this by letting teams build and publish agents across multiple channels without writing a line of code. For more technical teams, Visual Studio and Visual Studio Code provide the environment for building custom applications that call agent APIs directly. The client layer is not glamorous, but choosing the right interface matters. An agent that lives inside Teams has immediate adoption. One that requires a separate application needs a reason to be opened.

The application layer: receiving requests and hosting services

Once a request leaves the client, it hits the application layer. Azure Application Gateway acts as the front door, handling load balancing and protecting against common web threats before anything reaches your agent services. Azure API Management sits behind it, exposing your agent capabilities as governed APIs with rate limiting, authentication, and versioning built in.

The actual agent services run on Azure Web Apps for simple deployments, Azure Functions for event-driven or serverless workloads, and Azure Container Apps for more complex agents that need to scale independently or run as microservices. These are not interchangeable choices. A straightforward question-and-answer agent can live comfortably on Web Apps. An agent that coordinates multiple tools in parallel, or that needs to scale individual components differently, is better suited to Container Apps.

The AI and orchestration layer: where reasoning happens

This is the layer most people think of when they say AI agent. Azure AI Foundry is Microsoft’s unified platform for building, evaluating, and deploying AI applications. It gives teams a single environment to manage models, monitor quality, and govern deployments. Azure OpenAI Service provides access to GPT-4o, o1, and other frontier models with enterprise-grade SLAs, data privacy commitments, and no training on your prompts or data.

Microsoft’s Agent Framework, now part of Azure AI Foundry, handles orchestration: the sequencing of reasoning steps, the calling of tools, and the management of multi-agent workflows where multiple specialised agents collaborate on a task. GitHub sits here too, providing version control and CI/CD pipelines so agent prompts, code, and configurations are managed with the same rigour as any other software asset. Custom models, whether fine-tuned versions of open-weight models or entirely bespoke builds, can also be deployed and served from this layer when the general-purpose models are not the right fit.

The context and memory layer: what the agent knows

A model without context is a general-purpose tool. An agent with the right context becomes something genuinely useful to your business. This layer is where most of the real engineering work happens, and it is the layer that most often determines whether an agent delivers value or just sounds convincing.

Azure AI Search provides retrieval-augmented generation capabilities, indexing your documents, knowledge bases, and structured data so the agent can retrieve relevant information rather than relying on what was baked into training. Azure Blob Storage holds unstructured content, documents, images, and files that the agent may need to read or reference. PostgreSQL and Azure Cosmos DB handle structured and semi-structured data respectively, with Cosmos DB particularly suited to scenarios that need global distribution or flexible document schemas. Microsoft Graph APIs connect the agent to the fabric of Microsoft 365, giving it access to calendar data, emails, SharePoint content, and user profiles when that organisational context is relevant. Redis provides short-term memory and caching, allowing agents to maintain conversational state without repeatedly querying slower data stores. Azure Data Lake rounds out the layer for organisations with large analytical datasets that the agent needs to reason over.

The security and operations layer: what keeps it trustworthy

This layer is not optional, and it is not something you add later. Security has to be part of the architecture from the start, because retrofitting it into a production system is significantly harder than building it in.

Microsoft Entra ID handles identity and access management, ensuring that every call the agent makes is authenticated and that users only see data they are authorised to access. Azure Key Vault stores secrets, certificates, and API keys so they never appear in code or configuration files. Private endpoints keep agent-to-service traffic inside the Azure virtual network, preventing it from traversing the public internet. Azure Application Insights provides real-time monitoring of agent behaviour, capturing latency, errors, and usage patterns. Azure Log Analytics aggregates logs across the full stack, giving teams the visibility they need to investigate incidents and demonstrate compliance.

How the end-to-end flow works

Put it together and the flow looks like this. A user submits a request through Teams, a custom application, or another client. The Application Gateway receives it and routes it through API Management, which authenticates the caller against Entra ID and enforces any rate limits or policies. The request reaches the agent service hosted on Web Apps, Functions, or Container Apps. The agent calls the model through Azure OpenAI, with the orchestration layer coordinating any tools or sub-agents the task requires. To answer well, the agent queries Azure AI Search for relevant documents, checks a database for structured data, and retrieves conversational context from Redis. The response is returned to the user, the interaction is logged in Application Insights and Log Analytics, and the agent’s memory is updated for the next turn.

Every step in that flow is observable, authenticated, and governed. That is what separates a production agent from a demo.

Which layer matters most?

Every layer is necessary. Removing any one of them creates a system that either does not work or is not safe to deploy. But if forced to choose where most production readiness problems actually originate, it is usually the context and memory layer. The model is capable. The infrastructure is reliable. What lets most enterprise AI agents down is that they do not have access to the right information at the right time, or they surface information that the user asking should not be able to see.

Getting the data architecture right, connecting the right sources, applying the right access controls, and building the retrieval layer with care, is what turns a technically impressive prototype into something you can actually stand behind in production.

If any of this sounds familiar, talk to us about your data.

From prototype to production: the Azure architecture behind enterprise AI agents