There’s a quiet assumption embedded in most AI products today. It says the model is the product, that intelligence lives in the weights, and that your job is to route input in and output out, maybe with a nice interface and some caching on top.
That assumption is wrong, and it’s about to get expensive for the people who built on it.
I want to make a specific, technical case. Orchestrated systems, the ones that decompose a problem, validate intermediate work, and reason in layers, aren’t just better than single-model wrappers. They sit in a structurally stronger position. They get more capable, not obsolete, every time the underlying models improve. The reason is architectural, not magical.
The illusion of intelligence
Most of what gets called “AI-powered” right now is a wrapper: a thin layer of prompt engineering around an API call.
User input → [your code] → LLM API → [your code] → User output
No reasoning happens in your system. The reasoning, such as it is, happens inside the model, in a single forward pass, with no way to check its own work, no structured decomposition, no feedback from intermediate steps.
That works fine for simple, atomic tasks. Summarize this. Translate that. Classify this ticket. But the moment a task needs judgment, several steps, or actions that carry consequences, single-pass systems fail in ways that are hard to predict and harder to debug.
The failure isn’t dramatic. It’s subtle. The model sounds confident, the output looks right, and it’s wrong in exactly the ways that matter. You have no structural mechanism to catch it, because there’s no structure to begin with. There’s a prompt and a hope.
Reasoning is a system property
This claim is worth defending carefully, because the obvious objection is stronger than it used to be.
A language model predicts the next token, and it does this extraordinarily well. Modern reasoning models (the o-series, extended-thinking modes) spend a lot of inference-time compute doing it, exploring intermediate steps internally before they answer. That’s a real capability jump. So a skeptic asks a fair question: if the model can reason internally now, why orchestrate anything?
Because in-model reasoning and system-level orchestration operate at two different scales, and they compose instead of competing.
A reasoning model reasons within a step. However long it thinks, one inference call still can’t:
- enforce your domain’s data contracts
- call your deterministic validation logic
- fan out to parallel information sources and join the results
- checkpoint its progress and resume after a failure
- be audited, step by step, by a human or a downstream system
Orchestration is reasoning across steps, with persistent state, typed boundaries, and recovery. The reasoning model is a better engine. The orchestration is still the vehicle. A smarter engine is an argument for building a good vehicle, not against it, because now every node in your system thinks better while the system keeps doing the things one call structurally cannot.
State, contracts, and the graph
The real unit of an orchestrated system isn’t “an agent.” It’s a graph with typed state.
Every serious orchestration framework, and LangGraph is the canonical one, models the system as a directed graph. Nodes are functions, edges are transitions, and one explicit state object flows through the graph, updated at each node. This is what separates an engineered system from a pile of chained prompts.
import { Annotation } from "@langchain/langgraph";
const AnalysisState = Annotation.Root({
documents: Annotation<Document[]>(),
sections: Annotation<SectionPlan[]>(),
findings: Annotation<Finding[]>({
reducer: (current, incoming) => current.concat(incoming), // nodes append
default: () => [],
}),
contradictions: Annotation<Contradiction[]>({ default: () => [] }),
draft: Annotation<string | null>({ default: () => null }),
qaPassed: Annotation<boolean>({ default: () => false }),
revisionCount: Annotation<number>({ default: () => 0 }),
});
That typed state does a lot of quiet work. The system becomes inspectable (dump the full state at any point), recoverable (checkpoint and resume from it), and testable (assert on its shape). None of this exists in a wrapper, where “state” is whatever happens to be sitting in the prompt string.
The second idea is the one I’d argue is the actual heart of the discipline: nodes communicate through data contracts, not free text.
A naive pipeline passes prose between steps. The planner writes a paragraph, the researcher reads it, and so on. That’s fragile, because natural language has no schema. The downstream node has to re-parse intent every time, and a small change in phrasing breaks it silently.
A robust pipeline passes typed artifacts.
import { z } from "zod";
const Finding = z.object({
sectionId: z.string(),
claim: z.string(),
evidence: z.array(z.string()).describe("verbatim source spans"),
confidence: z.number().min(0).max(1),
sourceDocIds: z.array(z.string()),
});
type Finding = z.infer<typeof Finding>;
Now the boundary between two nodes is a contract, not a vibe. The research node emits valid Finding objects. The synthesis node consumes them. Each one gets developed, tested, and improved on its own. You can swap the model behind either node without renegotiating how they talk, because the contract is the interface, the same way it would be between two microservices.
This is why I think of the artifacts as the architecture. The models are interchangeable. The contracts are the thing you actually designed.
The loop, not the line
A pipeline is a directed acyclic graph. A → B → C, each step once. That’s already useful. The higher-leverage pattern is the cycle, and cyclic graphs are exactly what something like LangGraph adds on top of a plain DAG runner.
The canonical loop is generator and critic, the “reflexion” pattern. One node produces output, a second node critiques it against explicit criteria, and a conditional edge decides whether to accept it or send it back for another pass.
function routeAfterQa(state: typeof AnalysisState.State): string {
if (state.qaPassed) return "finalize";
if (state.revisionCount >= 3) return "escalate"; // bound the loop, never spin forever
return "revise";
}
graph.addConditionalEdges("qaAgent", routeAfterQa, {
finalize: "output",
revise: "synthesisAgent",
escalate: "humanReview",
});
Two details separate a toy from a production system here. First, the loop is bounded. revisionCount caps the iterations, so a stubborn failure escalates instead of burning budget forever. Second, there’s an escape hatch to a human. Orchestration done right doesn’t pretend the system is infallible. It routes the cases it can’t resolve to someone who can.
That’s what “feedback loop” actually means in an engineered system. Not a metaphor. A conditional edge in a graph, with a termination condition and a fallback.
Error doesn’t compound the way you hope, and that’s the real insight
It’s tempting to claim orchestration produces multiplicative quality gains. Five nodes, each 20% better, so 1.2⁵ ≈ 2.5× better output. I’ve seen the argument made. It’s wrong, and being precise about why gets you to something more useful.
“Quality” isn’t a multiplicative quantity with a natural unit. You can’t meaningfully multiply “better planning” by “better synthesis.” And in a real pipeline, errors propagate forward: a flawed plan poisons every downstream step that trusts it. Naive orchestration can actually do worse than a single call, because you’ve added more places for error to start and a chain for it to travel down. This cascading-failure mode is one of the genuinely hard problems in multi-agent systems, and any honest treatment has to name it.
So the value of orchestration isn’t that errors shrink multiplicatively. It’s that a well-placed validation gate bounds the error that reaches the next stage.
Compare the two regimes. A single call has unbounded error: one shot, no recovery, whatever comes out ships. An orchestrated pipeline with validation gates has bounded error: each gate caps how much garbage flows downstream, and in a cyclic graph it can send the bad output back for repair instead of passing it on.
That’s the mechanism. Not magic compounding. Error containment plus recoverability. The synthesis node produces better output not because it got smarter, but because a validation gate already checked its inputs against the source documents. You narrowed the input distribution before the hard step ran.
This reframing tells you where to spend nodes. You don’t add steps for their own sake. You add a gate wherever an undetected error would be expensive to let through.
The cost of thinking
Every node is at least one more model call. An orchestrated system doing real work might make six, ten, twenty calls where a wrapper makes one. A CTO reading this should immediately think: that’s 10× the latency and 10× the cost. It’s the right objection, and the answer is that orchestration hands you levers a single call doesn’t have.
Routing by difficulty. Not every node needs the frontier model. Classification, extraction, and formatting can run on a small, cheap, fast model. Reserve the expensive one for the genuinely hard synthesis and judgment steps. A wrapper pays the frontier price on every token. A system pays it only where it matters.
function selectModel(nodeType: string): string {
const cheap = new Set(["classify", "extract", "format", "route"]);
return cheap.has(nodeType) ? "small-fast-model" : "frontier-model";
}
Parallelism. Independent nodes run concurrently. Analyzing ten sections, you fan out and join, so wall-clock latency is one section, not ten. The graph structure tells you exactly what’s parallelizable, because the edges already encode the real dependencies.
Early termination and caching. A confident, validated result short-circuits the remaining steps. Deterministic sub-results (embeddings, parsed documents, prior findings) get cached and never recomputed. A wrapper recomputes everything, every time.
The honest framing for a decision-maker: orchestration turns a flat, uncontrollable cost into a structured cost you can engineer against. You give up naive simplicity and get a cost-latency-quality surface you can actually tune. For trivial tasks that trade isn’t worth it, so build the wrapper. For tasks where being wrong is expensive, it almost always is.
Evals are the moat
Here’s the question that sorts teams that talk about orchestration from teams that ship it: how do you know any of this is working?
If the answer is “the output looks good,” you don’t have a system. You have a demo. What turns orchestration into an engineering discipline instead of prompt astrology is evaluation, and evals turn out to carry the whole moat argument.
You evaluate at two levels. Node-level evals pin each component to a fixed dataset: given these documents, does the extraction node emit the expected Finding set? End-to-end evals measure the whole graph against held-out cases with known-good outcomes, scored by deterministic checks where you can and an LLM judge where you can’t.
test("extraction recall", () => {
for (const c of fixtures) {
const findings = extractionNode(c.documents);
const recall = covered(c.expectedClaims, findings) / c.expectedClaims.length;
expect(recall).toBeGreaterThanOrEqual(0.95);
}
});
Evals tell you whether the system is right. They don’t tell you why it went wrong, and for that you need traces.
I learned this the hard way. Early on I wrote my own observability layer, just structured logs around each node, inputs and outputs and timings. It was fine with three nodes. Then the graph grew, the loops started cycling, nodes began running in parallel, and one run could touch a node four times with slightly different state each pass. My logs turned into a wall of text I couldn’t reconstruct a story from. I was correlating timestamps by hand to figure out which Finding came from which revision. I spent more time maintaining the logging than the pipeline, and I still couldn’t answer basic questions about a failed run.
I moved to LangSmith and stopped fighting it. The point isn’t the specific tool. The point is that a multi-agent system is effectively undebuggable without a trace view that shows you the full tree of calls, the state at each hop, and where the tokens and latency actually went. Building that yourself is a real project, and it’s not the project you’re supposed to be working on. This is one of the few places I’d tell anyone to buy, not build, on day one.
The eval suite isn’t overhead either. It’s the asset that makes everything else in this essay safe, because it’s what lets you capture model improvements without gambling.
Think about what happens when a better model drops and you swap it in. With no evals you’re blind. The new model might lift your synthesis node and quietly regress your extraction node, and you won’t find out until a customer does. With a node-level suite, the swap becomes a measured operation: change one line, run the suite, read the diff. You keep the nodes that improved and catch the ones that regressed in an afternoon.
A wrapper has none of this. When the new model lands, the wrapper author re-tests by vibes. The system author re-tests by running a suite that encodes years of accumulated knowledge about what “correct” means in their domain.
That suite, the fixtures and the edge cases and the domain-specific definitions of right and wrong, isn’t something a better model hands you for free. You built it. It’s the moat.
Your moat is not the model
Now the fear can be answered directly: “GPT-5 will make my product obsolete.”
If your product is a wrapper, probably true. Not because the new model is smarter, though it is, but because the thing you built has no value apart from the model it wraps. You were arbitraging a capability gap that’s now closing.
If you built a system instead, typed state, data contracts, validation gates, bounded loops, a routing layer, and an eval suite that defines correctness in your domain, a better model doesn’t threaten you. It accelerates you, and you can prove it did, because you measured.
The vehicle is the valuable thing. It took months to design. It encodes your domain knowledge, your edge cases, your failure modes, your definition of correct. It knows which questions to ask and in what order. A better engine doesn’t replace any of that. It makes all of it run better, at every node, at once, and your evals let you bank the gain safely.
That’s a structurally different position from owning a wrapper. Wrapper builders are betting on a model capability gap that will close. System builders own infrastructure that gets more valuable as the models improve, because every improvement flows through architecture they control, checked by tests they wrote.
What this means for how you build
None of this is an argument for complexity. It’s an argument for intentional architecture. The first question is never “which model should I use,” because model access is a commodity. The question is what reasoning process the problem actually requires, and where an undetected error would be expensive.
If the task is simple and atomic, one call is the right answer. Build the simple thing. Don’t orchestrate for the sake of it. But if the task needs judgment, decomposition, validation, or synthesis across sources, no prompt is going to save you. You solve it by designing a process and encoding it in architecture you own.
A few heuristics for when to orchestrate:
- When errors are expensive, add a validation gate. A model can’t reliably validate itself in the same pass that produced the error.
- When the task decomposes naturally, the steps you’d take by hand are your nodes, and the dependencies between them are your edges.
- When you need auditability, and someone will ask “how did you get here,” you need typed intermediate artifacts, not one opaque blob of text.
- When the domain is specialized, general models reason generally. Orchestration is how you encode specific judgment at specific points.
- When you’ll swap models often, which right now is everyone, build the eval suite that makes swapping safe before you build much else.
The teams that win the next five years won’t be the ones with the best models. That access is a commodity, available to everyone, including your competitors. They’ll be the ones who built systems that reason: typed state, real contracts, bounded loops, honest cost engineering, and an eval suite that turns every new model release from a threat into an upgrade.
The model is the engine. Build the vehicle.