By JD Raimondi, Head of Data Science at MakingSense.
Most conversations about AI agents still begin with the model: which one reasons better, which one is faster, which one is cheaper per request. Those questions matter, but they rarely explain why an agent performs well in a controlled demo and then becomes inconsistent once it’s running a real, multi-step task.
That gap usually comes down to something else: not which model is running, but what information that model had available at the exact step where it made a wrong call.
That is the core of context engineering: the discipline of designing everything the model receives before it generates a response or takes an action. System instructions, retrieved documents, tool definitions, user history, task state, memory, constraints, examples, and intermediate outputs all compete for space inside the context window. That window is the total amount of information the model can work with at any one step, nothing beyond it factors into what the model decides to do. The order in which information enters that window shapes what the model actually does with it.
For a single-turn chatbot, a strong prompt may be enough. For an AI agent that needs to plan, call tools, interpret results, preserve decisions, and continue across multiple steps, context becomes architecture.
From prompt writing to context architecture
Prompt engineering taught teams how to give better instructions to a model. Context engineering asks a broader question: what information environment does the model need at each step to act correctly?
That shift matters because agents work through sequences. A workflow agent may retrieve a customer record, check a policy, call an internal API, interpret an exceptional case, ask for human approval, and continue based on the result. Each step depends on the right context being available at the right time.
A better instruction at the top of the workflow can help, but it cannot compensate for missing state, irrelevant retrieval, ambiguous tools, or a prior decision that was summarized away too early. Once AI systems become agentic, the prompt is one component inside a larger context pipeline.
That is why context engineering is becoming a production concern in 2026. Datadog’s 2026 State of AI Engineering report describes teams moving beyond single model requests into systems with model fleets, orchestration frameworks, tool calls, long prompts, retries, and multiple service boundaries. The report also found that more than 70% of organizations in its dataset now use three or more models in production, which adds evaluation, routing, compliance, and operational complexity.
In that environment, model choice still matters, but it’s the harness around the model that increasingly determines whether the agent can be trusted.
More context does not automatically mean better context
Long context windows created a new temptation: load more into the prompt and let the model sort it out.
That sounds reasonable until the system reaches production, because more context can also mean more duplicated instructions, more stale history, more irrelevant tool output, more retrieved documents competing for attention, and more cost on every request. Even worse, models suffer from a problem called “lost in the middle”, where bigger contexts give more probability of information not being used by the model even when it should.
The Datadog report makes this visible. It found that 69% of the text sent to models in production was system prompts: internal instructions, policy definitions, and tool guidance, not the actual task at hand. And among models built to reuse that information instead of reprocessing it on every request, only 28% of requests actually took advantage of that shortcut. In other words, many teams are paying repeatedly to process stable scaffolding that changes very little from one request to the next.
The same report found that average tokens per request more than doubled for median customers and quadrupled for 90th-percentile power users year over year. The takeaway for teams building agents is clear: context quality is becoming the limiting factor, rather than raw context volume.
This is where building an AI agent starts to feel more like engineering a complex operational system than writing a good prompt. Teams need to know what stays stable, what changes at each step, what can be cached, what should be retrieved just in time, what should be compressed, and what should leave the active context entirely.
The failure pattern behind many agent issues
When an AI agent gives the wrong answer, teams often look first at the model. That’s sometimes the right place to look, but more often, the model responded correctly to the context it received, and the context itself was the problem.
Here’s where the difficulty actually sits. Long-running agents eventually fill up their context window, so systems compress older parts of the conversation into a summary to keep going, a process Anthropic calls compaction. The hard part is deciding what to keep and what to throw away, since over-aggressive compaction can quietly drop a detail that turns out to matter several steps later, and there’s no reliable way to know in advance which details those will be.
That’s not a bug you patch by adding one more document to one more step. It’s a standing decision every long-running agent has to make, over and over, about what’s safe to forget. Get it slightly wrong on one rule out of a hundred, and the agent looks fine for weeks until it quietly miscalculates a price, misses a decision, or contradicts a policy nobody remembered to re-check.
From the outside, a failure like this looks like a reasoning problem. Under the hood, it’s a context problem: the agent had information, just not the right slice of it at the step where it mattered. And this is why production agents need context architecture: a deliberate way to assemble, order, preserve, compress, and observe the information that shapes each model call.
What good context engineering looks like in practice
The best context architecture is specific to the workflow, but a few patterns show up consistently in systems that hold up beyond the demo.
- Separate stable scaffolding from dynamic task state. System instructions, policy guardrails, tool schemas, and reusable examples should be structured so they can be cached or reused efficiently. The user request, latest tool result, current task state, and decision-specific data should stay dynamic.
- Scoping tools to the current step cuts down on noise. A step that only needs to read account data shouldn’t have access to the same range of tools as a step that can update records, trigger workflows, or send messages.
- Retrieve for decisions, not for documents. A retrieval layer should answer what the agent needs to decide next, which is a narrower question than pulling everything that might be related to the topic. Strong retrieval design includes ranking, filtering, freshness checks, and clear rules for when structured data should override unstructured text.
- Agents that run for a long time, through dozens of steps or multiple sessions, need a way to carry decisions, constraints, approvals, and exceptional cases forward without losing them. Anthropic’s 2026 guidance on context engineering frames memory, compaction, and tool-result clearing as complementary strategies for this, each addressing a different part of how context grows over a long task.
- Make context observable. Teams should be able to inspect what entered a given model request, which tools were available, what was retrieved, what got compressed, and what changed between runs. Without that visibility, debugging an agent failure turns into guesswork.
These patterns work best together, because each one solves a different part of the problem. Limiting an agent’s tools keeps individual steps reliable. Deciding what to compress, as covered earlier, keeps long tasks coherent. A production agent usually needs every layer working at once, not just one.
Why this matters to the business
Context engineering can sound like an implementation detail, but it carries direct business consequences:
- Reliability depends on the information the agent actually receives, not on what it’s theoretically capable of.
- Cost climbs when the same scaffolding gets reprocessed on every call instead of cached.
- Latency grows with every unnecessary token in the prompt.
This matters most for mid-market and PE-backed organizations moving an AI initiative from pilot to measurable value, where the real test isn’t a clean demo but whether the system holds up against workflow variation, edge cases, and business rules that weren’t in the test set.
A short list of questions tends to separate a pilot that’s ready to scale from one that isn’t:
- What information does the agent need at each step?
- Which parts of that context are stable enough to cache?
- Which decisions must persist across the workflow?
- Which tools should be available and when?
- How would the team detect drift or unnecessary token spend before it becomes a cost problem?
None of these are exotic questions. They’re the same operational discipline that distributed systems work has applied for years, aimed at a system where a model is making the decisions instead of a server routing a request.
Building this capability without starting from zero
None of this requires rebuilding an agent from scratch or waiting for the next model release. Start by auditing what actually lands in the context window today, separate what’s stable from what changes call to call, and give the agent only the tools and information relevant to the step it’s on.
That’s also the most practical entry point for teams early in their AI adoption, since it doesn’t require a large platform investment to show results. Strengthening context architecture on an existing pilot is often what separates an agent that looks promising in a demo from one a team is ready to put in front of real operations.
