Skip to content

Observability

agent/otel adds OpenTelemetry spans and metrics to an agent. It is opt-in: nothing in agent imports it.

Enable tracing

a, err := agent.New(agent.Config{
    Provider:  provider,
    Tools:     tools,
    SubAgents: []agent.SubAgentDef{researcher},
    Metrics:   myMetrics, // optional; keeps receiving every record
}, otel.WithTracing(otel.Config{
    TracerProvider:     tp,
    MeterProvider:      mp,
    Redactor:           maskEmails, // optional
    CaptureToolContent: true,       // optional; see "Tool content"
}))
if err != nil {
    return err
}

WithTracing wraps the provider and tools. It also wraps the provider and tools of sub-agents and handoff members that are set when the option runs. Put WithSubAgents and WithHandoffs before it, or set them on the base config.

A Metrics sink set before the option keeps receiving every record next to the OTel bridge. If the meter cannot create its instruments, the error is logged and the existing sink stays in place.

Spans

Span Kind Attributes
chat {model} client gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.request.* options, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokens, gen_ai.response.model, gen_ai.response.id, gen_ai.response.finish_reasons, gen_ai.response.time_to_first_chunk, saige.route.*, saige.dials.*
execute_tool {name} internal gen_ai.tool.name, gen_ai.tool.description, saige.tool.capability; with CaptureToolContent, gen_ai.tool.call.arguments and gen_ai.tool.call.result
invoke_agent {name} internal gen_ai.agent.name

Usage is merged across the parts a provider streams and written once when the stream ends, so a provider that reports prompt tokens first and completion tokens last keeps both. Partial usage is kept when the stream fails.

saige.route.profile, .provider, .model, .experiment, .variant and .reason come from the last route a router or traffic split reported for the call. A profile built from a catalog preset also sets saige.route.preset, saige.route.config_hash and saige.catalog.revision.

Each route is also recorded as a saige.route.attempt span event with its route attributes and the attempt's effective gen_ai.request.* options. When the call ends, the span's gen_ai.request.* attributes are set from the serving attempt's effective options: the profile's configured options merged with the per-call overrides. Without a route that reports options, they reflect the per-call overrides only.

A call with dials also records how they compiled: saige.dials.requested lists every dial as name:value, saige.dials.mapped the mapped ones as name:from→to, saige.dials.dropped the dropped ones, and saige.dials.policy the policy in force. A router reports them per attempt on its route, so they appear on each saige.route.attempt event and, for the serving attempt, on the span. For a single adapter that declares its capabilities, the traced call compiles the dials itself and records the effective options it sends.

gen_ai.response.time_to_first_chunk is measured from before the provider call to the first text chunk, so it includes connection setup.

WithTracing also sets agent.Config.RunTracer, so each run opens an invoke_agent span. Every chat and tool span of the run, including those of its sub-agents, shares one trace under it, and the span is marked failed when the run fails. To trace a run of an agent built without WithTracing, set RunTracer: otel.NewAgentTracer(cfg) on its config.

Errors

A failed span gets an error status, an exception event, and these attributes:

Attribute Value
error.type The error kind for a classified error (rate_limit, context_length, auth, unavailable, truncated, ...), canceled or timeout for a context error, tool_error for a tool result flagged IsError, panic for a recovered tool panic, or the Go type name otherwise
saige.error.transient Whether a retry can succeed. Set for classified errors only
saige.error.retry_after_ms The delay the server asked for, when it sent one

Provider and tool errors can contain request bodies or user data. Set Config.Redactor to rewrite every message before it reaches a span status or exception event.

Tool content

Spans carry no message content by default. Set Config.CaptureToolContent to record each tool call's arguments and result on its execute_tool span:

Attribute Value
gen_ai.tool.call.arguments The call's arguments as JSON
gen_ai.tool.call.result The result text, including the text of a failed call. Absent when the tool returned none

Arguments and results often hold user data, so turn this on only where the telemetry backend may store it. Config.Redactor rewrites each value first, and each value is cut to otel.MaxToolContentBytes (8 KiB) at a rune boundary, ending with …. Prompts and model responses are never recorded.

Metrics

Instrument Unit Operations
gen_ai.client.operation.duration s chat, execute_tool, invoke_agent (keyed by gen_ai.operation.name)
gen_ai.client.token.usage {token} gen_ai.token.type is input, output, cache_read or cache_creation
gen_ai.client.operation.time_to_first_chunk s chat

The agent loop records input and output token usage, cache_read and cache_creation usage when the provider reports prompt caching, and one invoke_agent duration per run with error.type set when the run failed. A custom Metrics sink receives the cache counts and the run outcome by also implementing types.CacheUsageRecorder and types.AgentOutcomeRecorder; otherwise it gets RecordAgentInvocation.

Wrappers keep tool behavior

The agent loop looks for optional interfaces on a tool: markers for approval, RichTool for typed parts and citations, Cacheable, Configurable, and the handoff and sub-agent interfaces. otel.WrapTool keeps each of them:

Tool Result
*types.MarkedTool Re-marked around a traced inner tool. The approval prompt still fires and the span covers only the approved work
HandoffSignaler or SubAgentInvoker Returned unchanged
Configurable Stays Configurable; binding returns a traced instance
Already traced Returned unchanged
Anything else *otel.TracedTool, which is a RichTool and reports the inner tool's cache policy

*otel.TracedTool and *toolcache.Tool implement Unwrap() types.Tool, and *otel.TracedProvider implements Unwrap() types.Provider, so an interface behind a decorator can be found by walking the chain.