Building Production AI Agents: Tool Use, Guardrails, and Observability

Introduction

The gap between an impressive agent demo and a production agent is the same gap that has always separated demos from products: error handling, limits, security, and observability. LLMs just make each of those harder. The model is non-deterministic, its failure modes are novel (it can be confidently wrong, or be talked into things by its own inputs), and every loop iteration costs real money.

This post lays out the architecture I use for agents that call tools against real systems: a bounded agent loop, typed tools with schemas, tiered permissions, and tracing from day one.

The Agent Loop Is Just a While Loop

Strip away the framework marketing and an agent is this:

def run_agent(task: str, tools: ToolRegistry, max_steps: int = 15) -> AgentResult:
    messages = [{"role": "user", "content": task}]

    for step in range(max_steps):
        response = llm.chat(messages, tools=tools.schemas())

        if response.tool_calls:
            for call in response.tool_calls:
                result = tools.execute(call.name, call.arguments)   # guarded, see below
                messages.append(tool_result_message(call.id, result))
            continue

        return AgentResult(answer=response.content, steps=step + 1)

    return AgentResult(answer=None, error="step budget exhausted", steps=max_steps)

Three production details hiding in those few lines:

• max_steps is not optional. An agent that can loop can loop forever, retrying a failing tool with slight variations, burning tokens the whole time. Budget steps, tokens, and wall-clock time, and treat hitting any budget as a first-class outcome to report, not an exception to swallow.
• Tool errors go back into the conversation, not up the Python stack. "The query timed out" is information the model can act on (narrow the query, try another tool). Raising instead of responding turns recoverable situations into failed runs.
• The return type admits failure. Callers need to distinguish "answered," "gave up," and "hit budget", especially when another system consumes the result.

Tools: Schemas Are Your Contract

Every tool gets a typed schema, validation, and its own timeout. The model is a text engine; your tool layer is the actual boundary between it and the world.

from pydantic import BaseModel, Field

class QueryOrdersArgs(BaseModel):
    customer_id: str = Field(pattern=r"^cus_[a-zA-Z0-9]+$")
    since_days: int = Field(default=30, ge=1, le=365)

def query_orders(args: QueryOrdersArgs) -> str:
    rows = db.fetch(ORDERS_SQL, args.customer_id, args.since_days)
    return json.dumps(rows[:50])          # cap output size, it becomes prompt tokens

Rules that have saved me repeatedly:

• Validate arguments with real schemas (Pydantic, JSON Schema). Models produce malformed or subtly-wrong arguments under pressure; reject them with a clear error message the model can read and fix.
• Cap output size. A tool that returns 200 KB of JSON silently destroys your context window and your latency. Truncate with an explicit marker ("...truncated, 1,432 more rows") so the model knows to refine.
• Make tools idempotent or clearly effectful. Retries happen, by your code and by the model itself.

Guardrails: Permission Tiers, Not Vibes

The prompt-injection problem is unsolved, so design as if any text the agent reads, user input, tool output, a web page, can steer it. The defense isn't a smarter system prompt; it's limiting what a steered agent can do:

• Tier 0 (read-only) (search, fetch, query): auto-approved.
• Tier 1 (reversible writes) (draft an email, create a ticket): auto-approved, logged, undoable.
• Tier 2 (consequential actions) (send, refund, delete, deploy): require human confirmation, always.

def execute(self, name: str, raw_args: dict) -> str:
    tool = self.tools[name]
    args = tool.args_model.model_validate(raw_args)     # schema gate

    if tool.tier >= 2 and not self.approvals.confirm(name, args):
        return "Action requires human approval; it was queued for review."

    with timeout(tool.timeout_s):
        return tool.fn(args)

Two more guardrails that cost little and prevent the worst incidents: run the agent with scoped credentials (it gets a database role that cannot drop tables, not a promise not to), and never feed secrets through the context window, the model will happily echo them into a log or a reply.

Observability: Trace Every Step or Debug Blind

When an agent misbehaves, the question is always "what did it see and why did it choose that?" If you can't replay the full trajectory, every prompt, tool call, tool result, and token count, you cannot answer it.

Log per step, structured: run ID, step number, model, input/output tokens, tool name, arguments, result summary, latency, and cost. Aggregate per run. Whether you use OpenTelemetry spans, LangSmith, or a Postgres table matters much less than doing it from the first day; retrofitting tracing after your first production incident is miserable.

The metrics that matter operationally: task success rate (define it explicitly), steps per task, cost per task, tool-error rate per tool, and human-escalation rate. Cost per task is the one that ambushes teams, an agent that succeeds 4% more often but takes 3× the steps is usually a regression.

Evaluation: A Golden Set Before Launch

You cannot unit-test a model, but you can regression-test an agent. Build a suite of 30-100 real tasks with programmatically checkable outcomes ("the ticket was created with the right fields", "the answer contains the correct order total"), and run it on every change, prompt tweaks, model upgrades, new tools. Prompt changes are code changes; treat them with the same discipline. A one-line system-prompt edit can drop success rate ten points, and without an eval suite you'll find out from users.

Takeaways

• An agent is a while loop with budgets; make every budget explicit and every exit reportable.
• Tools are the security boundary: typed schemas, validation, output caps, timeouts.
• Assume injection; enforce safety with permission tiers and scoped credentials, not prompt language.
• Trace every step from day one; watch cost per task, not just success rate.
• Keep a golden-task eval suite and run it on every prompt or model change.

None of this is glamorous, and that's the point, the teams shipping agents that survive contact with users are the ones treating them as distributed systems with an unusual component, not as magic.