Observability and costs

Agent behavior is only trustworthy if you can check it. Everything here is on by default; there's nothing to instrument.

Traces

Every turn produces a trace where each tool call, delegation, and model step is an edge with duration and token counts.

sf api get "/apps/<appId>/agent-trace?agent=<route>&session=<sid>&traceId=<tid>&format=mermaid&detail=1"

format=mermaid renders a sequence diagram; detail=1 adds the user message plus previews of each tool's input and output. The same data appears in the console news feed and the desktop app's traces screen.

The habit worth building: verify claims against edges.

If an agent says it checked inventory and there is no check_inventory edge in the trace, it didn't. It produced a plausible sentence.

This is how you catch the failure mode that matters most — a confident answer with nothing behind it. It's especially important after building a delegation, where a missing child edge means the child never ran no matter what the parent reported. See Delegation.

Turns and spend

sf api get "/apps/<appId>/agent-turns?limit=100"

Per-turn model calls, tokens, and cost. A delegating tree bills as one root-labeled turn, so what you see is the true cost of the whole tree rather than a fragment of it.

When a turn is more expensive than expected, read its trace before changing the model. The usual culprits are a tool being called in a loop, a tool returning far more data than the model needs, or a system prompt that's grown into a document.

What's actually deployed

sf api get "/apps/<appId>/agent-describe"

Returns what each deployed class really runs: model, tools with their approval flags, workspace grants, pinned repositories, schedules, event types. The deployed code is the registry — there's nothing to keep in sync, and therefore nothing that can silently disagree with reality.

Use it to answer "what is production actually running?" without reading the branch and hoping it matches.

Transcripts

sf api get "/apps/<appId>/agent-messages?agent=<route>&session=<sid>"

The persisted conversation. Useful when a user reports something odd and you need the actual exchange rather than their summary of it.

Your app's own event feed

Traces are for you. Events are for people — a record of what your app did, in business language, in your own database and your own UI.

Declare the vocabulary in code:

// app/events.ts
import { defineEvents } from "@spikefrost/agents";
import { z } from "zod";

export default defineEvents({
  "promo.created": {
    audience: ["business"],
    severity: "info",
    title: "Promotion {name} created at {discount}%",
    fields: z.object({ name: z.string(), discount: z.number() }),
  },
  "guardrail.refused": {
    audience: ["business"],
    severity: "warn",
    title: "Refused: {rule}",
    fields: z.object({ rule: z.string() }),
  },
});

Emitting an undeclared type throws — the catalog is the contract.

Emit from domain code, so the same event fires whichever caller acted:

import { emitAppEvent } from "@spikefrost/agents/events";   // light subpath

await emitAppEvent(env, "promo.created", { name, discount }, { actor });

From inside an agent, await this.emitEvent(...) adds agent and conversation links automatically.

A doctrine that pays off: every write-path domain function emits its outcome, every scheduled job emits completion, every guardrail refusal emits. Then "what happened yesterday?" is a query rather than an investigation.

Reading the feed

Events land in _spike_events in your app's database, created on first emit:

SELECT * FROM _spike_events ORDER BY ts DESC LIMIT 100;

Columns: id, ts (epoch ms), type, severity, audience, title, actor, fields (JSON), links (JSON), indexed on ts and (type, ts). Render title, color by severity, parse fields and links for detail.

Two conventions: audience: ["business"] is what belongs on your dashboard, and titles name domain objects, never people — a person goes in fields, so the feed doesn't read like surveillance.

Choose event types a colleague would mention out loud. "Promotion created", yes. "Model returned 412 tokens", no.

Budget refusals

When a cap is reached, the model call fails with a typed 402:

• tree_budget_exhausted — this delegation tree's cap
• team_cap_exhausted — the team's overall cap

Surface it. Never retry-loop it. A retry loop against an exhausted budget is just a loop. Per-tree caps cover the whole tree, since children inherit the root's budget.

Agents that watch their own app

An agent can read its own app's logs, deployments, and analytics — but only when you declare it:

override tools = {
  app_logs: this.appLogsTool(),               // this app's worker logs
  app_deployments: this.appDeploymentsTool(), // deploy records and events
  app_analytics: this.appAnalyticsTool(),     // SELECT over this app's metrics
};

All three are read-only and pinned server-side to this app — the model cannot point them at another app.

Caveats worth knowing before you trust them:

• Logs lag live traffic by up to about five minutes because of log batching. This is a debugging tool, not real-time monitoring.
• Analytics is sampled. Weight aggregates by _sample_interval. Use it for trends and rates; use logs for exact events and the event feed for declared business outcomes.

Grant these to debugging, support, and ops-shaped agents. A customer-facing agent has no business reading your logs — decide that in the design, not by reflex.

Next

• Delegation — verifying a tree actually ran.
• Approvals — the events an approval queue is built from.
• Build and deploy — request logs and deployment events.