Designing Multi-Agent Systems That Don't Collapse Under Load

The Coordination Problem

A single LLM agent is a well-understood system. You send a prompt, you get a completion. The failure modes are predictable: hallucination, context overflow, tool call errors. Multi-agent systems introduce a different class of problem — coordination failure. Agents that individually work correctly can collectively produce catastrophic behavior through emergent interaction patterns.

Most multi-agent frameworks address the happy path well. Few address what happens when the system runs hot under real load.

Token Budget Management Across Agents

The first thing that breaks in production is token budgets. Each agent in a pipeline has a context window. When you chain agents, the output of one becomes input to the next. Without explicit budget management, context windows fill with redundant information, previous agent outputs, and scaffolding that should have been stripped.

The pattern that works:

type AgentBudget = {
  maxInputTokens: number;
  maxOutputTokens: number;
  reservedForTools: number;
  reservedForSystemPrompt: number;
};

function computeAvailableContext(budget: AgentBudget, currentUsage: number): number {
  const overhead = budget.reservedForTools + budget.reservedForSystemPrompt;
  return budget.maxInputTokens - overhead - currentUsage;
}

Track token consumption at the orchestrator level, not inside individual agents. Agents are bad at self-reporting accurate token usage — model providers often count differently than SDK estimates. Use the actual token counts returned in API responses, accumulate them in a shared budget registry, and make routing decisions based on remaining capacity before dispatching the next agent.

When an agent's context budget is nearly full, you have three options: summarize and compress, route to a fresh agent instance with the essential context injected, or terminate the subtask and return a partial result. Never silently truncate. Truncation is the hardest bug to diagnose in a multi-agent system because the symptom (wrong output) appears far downstream from the cause (missing context).

Shared Context Strategies

Agents need shared state without duplicating it into every context window. The naive approach — serializing the full state and injecting it into each agent's prompt — does not scale. At five agents with moderate state, you're burning 20k tokens on context that most agents don't need.

Three viable patterns:

Pattern 1: Selective context injection. The orchestrator maintains a central state store. Each agent receives only the subset of state relevant to its task, described in its system prompt. The agent's output is merged back into the central store. Requires well-defined agent responsibilities and a state schema the orchestrator understands.

Pattern 2: Tool-mediated state access. State is stored externally (a key-value store, a structured file, a database). Agents access it through tool calls — read_state(key), write_state(key, value). This adds latency per read but keeps context windows clean. Works well when agents need sparse, unpredictable access to a large state space.

Pattern 3: Event sourcing. The orchestrator maintains an append-only event log. Agents receive only the events relevant to their current task, materialized as a short summary. New agents bootstrap from a point-in-time snapshot plus recent events. This is the most robust pattern for long-running systems but has the highest implementation complexity.

In practice, start with selective injection (Pattern 1). It's the easiest to debug. Move to tool-mediated access (Pattern 2) when state grows large or access patterns become unpredictable. Reserve event sourcing for systems that need complete auditability or that must support agent restart mid-task.

Tool Call Routing

Not all agents should have access to all tools. An analysis agent that can write to a database is a liability. A summarization agent with internet access is a liability. Tool access should be scoped to each agent's declared purpose.

The routing layer enforces this:

type AgentToolPolicy = {
  agentId: string;
  allowedTools: string[];
  deniedTools: string[];
  requiresConfirmation: string[]; // tools that need human approval
};

function resolveToolAccess(agentId: string, toolName: string, policies: AgentToolPolicy[]): boolean {
  const policy = policies.find(p => p.agentId === agentId);
  if (!policy) return false;
  if (policy.deniedTools.includes(toolName)) return false;
  return policy.allowedTools.includes(toolName) || policy.allowedTools.includes('*');
}

Critically: validate tool call arguments before execution, not just tool access. An agent with legitimate access to a send_email tool should still have its recipient and content validated against a schema before the call fires. Agents hallucinate arguments. Defense in depth catches this.

Preventing Agent Loops

Agent loops are the multi-agent equivalent of an infinite loop. An agent calls a tool, the tool output prompts another tool call, which produces output that prompts the same first tool call again. Without detection, this burns tokens and money until the context window fills or the bill exceeds budget.

Detection requires tracking call sequences, not just individual calls:

type CallRecord = {
  toolName: string;
  argumentHash: string; // stable hash of normalized arguments
  timestamp: number;
};

function detectLoop(history: CallRecord[], windowSize: number = 5): boolean {
  if (history.length < windowSize * 2) return false;
  const recent = history.slice(-windowSize).map(r => `${r.toolName}:${r.argumentHash}`).join(',');
  const previous = history.slice(-windowSize * 2, -windowSize).map(r => `${r.toolName}:${r.argumentHash}`).join(',');
  return recent === previous;
}

When a loop is detected, interrupt the agent, inject a system message explaining the detected cycle, and either force termination or attempt to break the loop by injecting a different framing. Do not silently retry — that restarts the same loop.

Beyond exact loops, watch for semantic loops: the agent rephrases the same tool call with different arguments but equivalent intent. These are harder to detect mechanically. Heuristics: if an agent has made more than N calls of the same tool category within a task, require it to produce an intermediate result before continuing.

Observable State

A multi-agent system you cannot observe is a system you cannot debug. Every production deployment needs:

• A trace ID that propagates through all agent invocations within a task
• Per-agent token consumption logged to a time-series store
• Tool call logs with arguments (sanitized of secrets), response times, and success/failure status
• Agent-to-agent message passing logged as events with sender, receiver, and payload size

Structured logging beats unstructured every time. If your agent outputs go to a log file as free text, you will not be able to answer "how many tokens did the analysis agent consume on average last week" without parsing. Emit JSON from the start.

For tracing, OpenTelemetry works if you're already in that ecosystem. For simpler setups, a correlation ID injected into every agent's system prompt (and echoed back in every response) is sufficient to reconstruct execution chains from logs.

Failure Isolation

The canonical failure mode: one agent in a pipeline produces bad output. That bad output propagates through the rest of the pipeline, which amplifies the error and produces a confidently wrong final result.

Isolation requires explicit validation boundaries between agents:

type AgentOutput = {
  success: boolean;
  data?: T;
  error?: string;
  confidence?: number; // 0-1, agent's self-reported confidence
  warnings: string[];
};

function validateAgentOutput(output: AgentOutput, schema: ZodSchema): Result {
  if (!output.success) return { ok: false, error: output.error ?? 'Agent reported failure' };
  const parsed = schema.safeParse(output.data);
  if (!parsed.success) return { ok: false, error: `Schema validation failed: ${parsed.error.message}` };
  return { ok: true, value: parsed.data };
}

Validate at the boundary. If an agent's output doesn't pass the schema for what the next agent expects, do not forward it. Return a structured error to the orchestrator and either retry with a different agent, escalate to a human, or terminate the task cleanly.

Retry policies need backoff and budget awareness. A retry that immediately re-invokes the same agent with the same input will produce the same failure. At minimum, retry with modified context or after a delay. After two failures, consider routing to a simpler fallback agent rather than a more capable one — sometimes the expensive model is getting confused by complexity that a targeted simpler model handles correctly.

The Orchestrator Pattern

The orchestrator is the component that coordinates everything above. Its responsibilities: task decomposition, agent selection, budget tracking, loop detection, failure routing, and result aggregation. It does not do work itself — it routes work to agents that do.

Keep the orchestrator stateless between invocations. Its state comes from the central store, not from instance variables. This makes it horizontally scalable and eliminates an entire class of subtle state corruption bugs that appear under concurrent load.

The most important design decision for an orchestrator is what it does when it doesn't know what to do. The answer should be: escalate to a human with a full trace, do not guess. An orchestrator that guesses under uncertainty is a liability. One that escalates predictably is an asset.