ObservabilityCAUTION

A framework for building agentic apps

The instruction file, verbatim from the audited commit — this is the text the model reads, and the surface the audit's instruction layer examines. Quoted here so you can judge it without cloning anything.

---
name: observability
description: >-
  Agent observability, evals, feedback, and experiments. Use when adding
  observability dashboards, configuring trace capture, setting up evals,
  creating A/B experiments, or collecting user feedback on agent responses.
scope: dev
metadata:
  internal: true
---

# Agent Observability

## Rule

The observability system auto-instruments every agent run with zero configuration. Traces, automated evals, and feedback collection work out of the box. All data lives in the app's own SQL database — no external services required. Templates can optionally export to Langfuse, Datadog, or any OTel-compatible platform.

## Five Pillars

### 1. Traces

Every `runAgentLoop()` call is automatically instrumented via `instrumentAgentLoop()` in `packages/core/src/observability/traces.ts`. It captures:

- **agent_run** span — top-level parent with total duration and cost
- **llm_call** span — model name, token counts (input, output, cache read/write), cost
- **tool_call** spans — one per action invocation, with duration and success/error

The run span is NAMED by what started it, because the trace list shows that name
and nothing else: a scheduled job is `background_automation_run:<job name>`, a
chat turn is `agent_run`, and a turn a feature sent on the user's behalf is
`agent_run:<usageLabel>` when the caller named it:

```ts
sendToAgentChat({
  message: "Enrich this record from the web",
  newTab: true,
  usageLabel: "crm:enrich-record", // → usage row label + `agent_run:crm:enrich-record`
});
```

Content (prompts, tool args, tool results) is **redacted by default**. Opt in through the declared `observability` config domain:

```ts
// server/plugins/config.ts
import { defineAppConfig } from "@agent-native/core/server";

export default defineAppConfig({
  observability: {
    enabled: true,
    capturePrompts: false,
    captureToolArgs: true, // capture action input args
    captureToolResults: false, // include results and full error text in local tool spans
    evalSampleRate: 0.05, // 5% of runs get LLM-as-judge eval
    inferredSentimentEnabled: false,
    inferredSentimentSampleRate: 0,
    inferredSentimentModel: "gpt-6-luna",
  },
});
```

Two things are recorded whatever the flags say, because a failure nobody can
classify is not observable:

- **A failed tool span always keeps a signature**: the first line of the error,
  redacted (credentials, emails, long opaque ids) and capped at 500
  characters. `captureToolResults` only decides
  whether the rest is kept. The span's metadata says which (`__tool_error_detail`:
  `full` | `signature`), and the read path returns it as `errorDetail`
  (`full` | `signature` | `withheld` | `unrecorded`) so "withheld on purpose" is
  never read as "nothing was recorded". `$ai_*` telemetry omits error text
  even when capture is enabled. Optional host-owned OTel run spans also use
  fixed code-derived messages and retain `agent.error_code` / `agent.error_cause`.
- **A stop that waits on the user is not an error.** An `input_required`
  outcome (question, approval, connection) records the `agent_run` span as
  `status: "paused"` with the reason in `terminal_code`, not `error`. Anything
  that counts failed runs must treat `paused` as non-error; an `error` event
  that follows the pause still makes the run an error.

#### From an error report to the failing run

A report is only useful if it names where it happened. Four pieces make that
true, so nobody has to ask the reporter for an example:

- **The failure packet.** Every server `captureError()` carries
  `extra.failureContext` (`app`, `route`, `actionName`, `automationName`,
  `threadId`, `runId`, `requestId`, `userScope`, `threadUrl`, build,
  environment, `errorCode`, `failureClass`); see the tracking skill for how it
  is derived. `threadUrl` is `https://<app host>/?thread=<chat_threads.id>`, and
  the Analytics issue page links it. The same fields (`$ai_trace_id` = run id,
  `thread_id`) are on the `$ai_trace` for the run.
- **A copyable report on the client.** `formatClientFailureReport()` from
  `@agent-native/core/client/failure-report` builds the plain-text packet (app,
  thread link, run, request, code, time, build, and the inspection call) for a
  "Copy details" button. The chat run-error card's existing "Copy debug info"
  button uses it; call it from any other error card. It needs only what the
  card already knows (`message`, `errorCode`, `runId`, `requestId`) and falls
  back to the open thread. The error-screen "Open GitHub issue" template adds
  the same thread link, build and time. A failed action carries `error.requestId` (the
  response's `x-agent-native-request-id`, also on the server capture's
  `failureContext.requestId`).
- **Read by id.** Dispatch's `get-agent-thread-debug` takes a `threadId` or the
  copied run id (`run-...`) and returns the run's `terminalReason` and terminal
  event, its events, trace spans (failed tools carry a signature, see above),
  feedback and checkpoints; `list-agent-run-failures` finds candidates. Both are
  owner-scoped. Without Dispatch, `GET /_agent-native/observability/traces/:runId`
  returns the same spans for the caller's own runs.
- **Alerts name examples.** The chat-health Slack alert lists the latest failed
  turns as `threadUrl (run id, code)`; if they cannot be read it says so.

#### Optional inferred sentiment

Self-hosted apps default to no inferred sentiment. First-party apps hosted on
`agent-native.com` automatically classify 100% of eligible user replies with
`gpt-6-luna`; an explicit stored `inferredSentimentEnabled: false` remains an
opt-out. Deployment overrides are `AGENT_NATIVE_INFERRED_SENTIMENT=on|off`,
`AGENT_NATIVE_INFERRED_SENTIMENT_SAMPLE_RATE=0..1`, and
`AGENT_NATIVE_INFERRED_SENTIMENT_MODEL=<model>`; `off` is always the emergency
kill switch.

Classification uses only the original visible user text, capped at 2,000
characters, with no tools, temperature 0, an eight-token output, and a five
s

CAUTIONgrade B · trust 89/100 Install with care. The audit found things worth knowing before you trust its output.

What the source does

Filesystem

none-observed

Network

none-observed

Shell

none-observed

Dependencies

pinned

Secrets in source

none-found

Findings (5)

.agents/skills/an

.agents/skills/content-product-development

.agents/skills/design-exploration

.agents/skills/turn-into-app

.agents/skills/visual-edit

Gates applied: no_behavioural_pass.

[![ModelsAtlas audit: Observability](https://modelsatlas.com/skills/builderio/observability/badge.svg)](https://modelsatlas.com/skills/builderio/observability/)

Not your listing, or something out of date? Claim or update your listing.