Monitoring Agents
Also known as: monitoring observability telemetry opentelemetry otlp analytics logs export grafana datadog elastic siem dashboard tracing
● Intermediate5 minBefore this: Audit Log, LoggingSpec 8.6
Two records say what an agent did: the audit log the runtime writes for every capability call, and the events the agent logs itself, which go on to the monitoring tools you already run.
use std.fs
use std.log
type PayError = Declined | Denied(reason: String)
capability Payments {
fn refund(order: Int, amount: Int) ! PayError
}
type Bank: Payments {
fn refund(self, order: Int, amount: Int) ! PayError {
}
}
type RefundIssued: LogEvent {
order: Int
amount: Int
}
type RefundBlocked: LogEvent {
order: Int
reason: String
}
agent Refunder {
settings AgentSettings(audit: Audit())
can Payments.refund
limit Payments.refund {
amount <= 500
}
on refund(order: Int, amount: Int) ! PayError {
Payments.refund(order, amount) catch e => {
log.warn(RefundBlocked(order: order, reason: "{e}"))
fail e
}
log.info(RefundIssued(order: order, amount: amount))
}
}
fn main() ! Error {
const bot = Refunder.start(Payments: Bank())
bot.refund(1, 400) catch _ => {}
bot.refund(2, 900) catch _ => {}
print("-- the audit log --")
for line in try fs.lines("agents/Refunder/audit.log") {
print(line.split("Z ").last() ?? line)
}
}The program ran and printed:
kavach run
INFO RefundIssued(order: 1, amount: 400)WARN RefundBlocked(order: 2, reason: "Denied(reason: \"limit broken: amount <= 500\")")-- the audit log --Refunder request#1 Payments.refund(order: 1, amount: 400) verdict: allowed result: okRefunder request#2 Payments.refund(order: 2, amount: 900) verdict: refused (limit broken: amount <= 500) result: denied: limit broken: amount <= 500
- The audit log, turned on by
Audit()in the agent's settings, is written by the runtime, so no call can be left out: each one with its arguments, the verdict and the result, refused calls included. - An
askin the audit log also names the model that answered and the tokens it took, so what each agent costs is there too: see Choosing the model. - The agent's own events,
RefundIssuedandRefundBlocked, say what each call meant for the business, in fields you choose.
Sending them to your tools
| Record | Holds | Goes to, today |
|---|---|---|
| The audit log | every capability call: arguments, verdict, result; secrets as <secret> | a file in the agent's folder, as text or as JSON lines, and with otlp: true also any OpenTelemetry collector |
| The agent's log events | records you define, one line each | the terminal, files that rotate by size, and any OpenTelemetry collector: Grafana, Datadog, Elastic, Honeycomb and others read from there |
To send the log events to a collector, name it in kavach.toml. The program's owner chooses where events go, not its code:
[log]
level = "info"
file = "logs/agents.log" # also kept on disk, a new file every 10 MB
[log.otlp]
endpoint = "http://localhost:4318" # OTLP over HTTP
service = "refunds"- Each event becomes an OpenTelemetry log record: its level is the severity, its printed form the body, and its type's name the attribute
event.name. - Events are sent in batches by a thread of their own, so a slow collector never slows the agent.
- Logging needs no capability, so an agent's
candoesn't change when you add it.
The audit log as JSON lines
A log shipper reads JSON without a parser written for it. With Audit(format: AuditFormat.JsonLines) in the agent's settings, each entry is one JSON object on a line of its own:
use std.fs
type Verdict {
/// true if the customer is unhappy
complaint: Bool
}
type PayError = Declined | Denied(reason: String)
capability Payments {
fn refund(order: Int, amount: Int) ! PayError
}
type Bank: Payments {
fn refund(self, order: Int, amount: Int) ! PayError {
print("refunded {amount} on order {order}")
}
}
agent Support {
model Model(name: "claude-haiku-5-5", thinking: false)
settings AgentSettings(audit: Audit(format: AuditFormat.JsonLines))
can Ai, Payments.refund
limit Payments.refund {
amount <= 500
}
on handle(order: Int, amount: Int, note: String) ! Error {
const v = try ask<Verdict>("Is this a complaint? {note}")
if v.complaint {
try Payments.refund(order, amount)
}
}
}
fn main() ! Error {
const bot = Support.start(Payments: Bank())
try bot.handle(7, 40, "My order never came.")
bot.handle(8, 900, "It broke on day one.") catch e => print("refused: {e}")
for line in try fs.lines("agents/Support/audit.log") {
// The time changes every run, so it is cut out here.
print("{line.slice(0, 9)}…{line.slice(29, line.len())}")
}
}The program ran and printed:
kavach run
refused: Denied(reason: "limit broken: amount <= 500"){"time":"…","agent":"Support","request":1,"call":"Ai.ask","shown":"ask(\"Is this a complaint? My order never came.\")","verdict":"dry run","model":"claude-haiku-5-5","result":"ok Verdict(complaint: true)"}{"time":"…","agent":"Support","request":1,"call":"Payments.refund","shown":"Payments.refund(order: 7, amount: 40)","verdict":"dry run","result":"simulated ()"}{"time":"…","agent":"Support","request":2,"call":"Ai.ask","shown":"ask(\"Is this a complaint? It broke on day one.\")","verdict":"dry run","model":"claude-haiku-5-5","result":"ok Verdict(complaint: true)"}{"time":"…","agent":"Support","request":2,"call":"Payments.refund","shown":"Payments.refund(order: 8, amount: 900)","verdict":"refused (limit broken: amount <= 500)","result":"denied: limit broken: amount <= 500"}[dry run] ask("Is this a complaint? My order never came.") -> Verdict(complaint: true)[dry run] Payments.refund(order: 7, amount: 40)[dry run] ask("Is this a complaint? It broke on day one.") -> Verdict(complaint: true)[dry run] Payments.refund(order: 8, amount: 900) refused: limit broken: amount <= 500
- Every entry has
time(UTC),agent,request,call,shown,verdictandresult. A refused call is there too, with the rule it broke. - An
askaddsmodel, and in a real runinput_tokensandoutput_tokens. A dry run, as here, has no token fields, and its verdict isdry run. - The setting is per agent: an agent without it keeps the text format, in the same run.
Textis the default format.
A log shipper (Vector, Fluent Bit, the OpenTelemetry Collector's file receiver) can carry the file on from there.
The audit log to a collector
With otlp: true in the agent's Audit, each entry also goes to the collector named in kavach.toml's [log.otlp], the same one the log events go to. Where the collector is depends on where the program runs, so it stays in kavach.toml:
settings AgentSettings(audit: Audit(otlp: true))[log.otlp]
endpoint = "http://localhost:4318"| In the log record | Holds |
|---|---|
event.name | AgentAudit, so audit records are easy to pick out from your own events |
severity | WARN for a refused call, INFO otherwise: an alert on WARN finds every refusal |
body | the call as shown in the file, such as Payments.refund(order_id: 2, amount: 150) |
agent, request, call, verdict, result | the JSON line's fields, as attributes |
model, input_tokens, output_tokens | for an ask, so cost per agent is a query |
service.name | service in [log.otlp], or the package's name |
- Records go in batches, on the thread that sends the log events, so a slow collector never holds up an agent.
- The file stays the record that counts. It is still written, and a collector that is down loses nothing from it.
- The setting is per agent, and
falseby default. - Without a collector to send to, the program is refused before it runs:
agent Support {
settings AgentSettings(audit: Audit(otlp: true))
can Console
on note(text: String) {
print(text)
}
}
fn main() {
Support.start().note("never runs")
}The compiler refused the program:
kavach run
error: `otlp: true` sends the audit log to a collector: add a `[log.otlp]` section with its `endpoint` to `kavach.toml` (spec 8.6, 14.12)For analytics
Because an event is a record, every field is a column you can query: refunds per day, the total refunded per customer, how often a limit stopped an agent, from RefundBlocked's reason. A secret field is written as <secret> wherever the event goes.
Keep the durable journal out of your analytics: it is state, and holds secret values as they are.