Skip to content
Kavachकवच

Monitoring Agents

Also known as: monitoring observability telemetry opentelemetry otlp analytics logs export grafana datadog elastic siem dashboard tracing

● Intermediate5 minBefore this: Audit Log, LoggingSpec 8.6

Two records say what an agent did: the audit log the runtime writes for every capability call, and the events the agent logs itself, which go on to the monitoring tools you already run.

use std.fs
use std.log

type PayError = Declined | Denied(reason: String)

capability Payments {
    fn refund(order: Int, amount: Int) ! PayError
}

type Bank: Payments {
    fn refund(self, order: Int, amount: Int) ! PayError {
    }
}

type RefundIssued: LogEvent {
    order: Int
    amount: Int
}

type RefundBlocked: LogEvent {
    order: Int
    reason: String
}

agent Refunder {
    settings AgentSettings(audit: Audit())
    can Payments.refund

    limit Payments.refund {
        amount <= 500
    }

    on refund(order: Int, amount: Int) ! PayError {
        Payments.refund(order, amount) catch e => {
            log.warn(RefundBlocked(order: order, reason: "{e}"))
            fail e
        }
        log.info(RefundIssued(order: order, amount: amount))
    }
}

fn main() ! Error {
    const bot = Refunder.start(Payments: Bank())
    bot.refund(1, 400) catch _ => {}
    bot.refund(2, 900) catch _ => {}
    print("-- the audit log --")
    for line in try fs.lines("agents/Refunder/audit.log") {
        print(line.split("Z ").last() ?? line)
    }
}

The program ran and printed:

kavach run

INFO RefundIssued(order: 1, amount: 400)WARN RefundBlocked(order: 2, reason: "Denied(reason: \"limit broken: amount <= 500\")")-- the audit log --Refunder request#1 Payments.refund(order: 1, amount: 400)  verdict: allowed  result: okRefunder request#2 Payments.refund(order: 2, amount: 900)  verdict: refused (limit broken: amount <= 500)  result: denied: limit broken: amount <= 500
src/main.kav: Kavach code, then what running it prints
  • The audit log, turned on by Audit() in the agent's settings, is written by the runtime, so no call can be left out: each one with its arguments, the verdict and the result, refused calls included.
  • An ask in the audit log also names the model that answered and the tokens it took, so what each agent costs is there too: see Choosing the model.
  • The agent's own events, RefundIssued and RefundBlocked, say what each call meant for the business, in fields you choose.

Sending them to your tools

RecordHoldsGoes to, today
The audit logevery capability call: arguments, verdict, result; secrets as <secret>a file in the agent's folder, as text or as JSON lines, and with otlp: true also any OpenTelemetry collector
The agent's log eventsrecords you define, one line eachthe terminal, files that rotate by size, and any OpenTelemetry collector: Grafana, Datadog, Elastic, Honeycomb and others read from there

To send the log events to a collector, name it in kavach.toml. The program's owner chooses where events go, not its code:

[log]
level = "info"
file = "logs/agents.log"      # also kept on disk, a new file every 10 MB

[log.otlp]
endpoint = "http://localhost:4318"   # OTLP over HTTP
service = "refunds"
kavach.toml: TOML code
  • Each event becomes an OpenTelemetry log record: its level is the severity, its printed form the body, and its type's name the attribute event.name.
  • Events are sent in batches by a thread of their own, so a slow collector never slows the agent.
  • Logging needs no capability, so an agent's can doesn't change when you add it.

The audit log as JSON lines

A log shipper reads JSON without a parser written for it. With Audit(format: AuditFormat.JsonLines) in the agent's settings, each entry is one JSON object on a line of its own:

use std.fs

type Verdict {
    /// true if the customer is unhappy
    complaint: Bool
}

type PayError = Declined | Denied(reason: String)

capability Payments {
    fn refund(order: Int, amount: Int) ! PayError
}

type Bank: Payments {
    fn refund(self, order: Int, amount: Int) ! PayError {
        print("refunded {amount} on order {order}")
    }
}

agent Support {
    model Model(name: "claude-haiku-5-5", thinking: false)
    settings AgentSettings(audit: Audit(format: AuditFormat.JsonLines))
    can Ai, Payments.refund

    limit Payments.refund {
        amount <= 500
    }

    on handle(order: Int, amount: Int, note: String) ! Error {
        const v = try ask<Verdict>("Is this a complaint? {note}")
        if v.complaint {
            try Payments.refund(order, amount)
        }
    }
}

fn main() ! Error {
    const bot = Support.start(Payments: Bank())
    try bot.handle(7, 40, "My order never came.")
    bot.handle(8, 900, "It broke on day one.") catch e => print("refused: {e}")
    for line in try fs.lines("agents/Support/audit.log") {
        // The time changes every run, so it is cut out here.
        print("{line.slice(0, 9)}…{line.slice(29, line.len())}")
    }
}

The program ran and printed:

kavach run

refused: Denied(reason: "limit broken: amount <= 500"){"time":"…","agent":"Support","request":1,"call":"Ai.ask","shown":"ask(\"Is this a complaint? My order never came.\")","verdict":"dry run","model":"claude-haiku-5-5","result":"ok Verdict(complaint: true)"}{"time":"…","agent":"Support","request":1,"call":"Payments.refund","shown":"Payments.refund(order: 7, amount: 40)","verdict":"dry run","result":"simulated ()"}{"time":"…","agent":"Support","request":2,"call":"Ai.ask","shown":"ask(\"Is this a complaint? It broke on day one.\")","verdict":"dry run","model":"claude-haiku-5-5","result":"ok Verdict(complaint: true)"}{"time":"…","agent":"Support","request":2,"call":"Payments.refund","shown":"Payments.refund(order: 8, amount: 900)","verdict":"refused (limit broken: amount <= 500)","result":"denied: limit broken: amount <= 500"}[dry run] ask("Is this a complaint? My order never came.") -> Verdict(complaint: true)[dry run] Payments.refund(order: 7, amount: 40)[dry run] ask("Is this a complaint? It broke on day one.") -> Verdict(complaint: true)[dry run] Payments.refund(order: 8, amount: 900) refused: limit broken: amount <= 500
src/main.kav: Kavach code, then what running it prints
  • Every entry has time (UTC), agent, request, call, shown, verdict and result. A refused call is there too, with the rule it broke.
  • An ask adds model, and in a real run input_tokens and output_tokens. A dry run, as here, has no token fields, and its verdict is dry run.
  • The setting is per agent: an agent without it keeps the text format, in the same run.
  • Text is the default format.

A log shipper (Vector, Fluent Bit, the OpenTelemetry Collector's file receiver) can carry the file on from there.

The audit log to a collector

With otlp: true in the agent's Audit, each entry also goes to the collector named in kavach.toml's [log.otlp], the same one the log events go to. Where the collector is depends on where the program runs, so it stays in kavach.toml:

settings AgentSettings(audit: Audit(otlp: true))
src/main.kav, in the agent: Kavach code
[log.otlp]
endpoint = "http://localhost:4318"
kavach.toml: TOML code
In the log recordHolds
event.nameAgentAudit, so audit records are easy to pick out from your own events
severityWARN for a refused call, INFO otherwise: an alert on WARN finds every refusal
bodythe call as shown in the file, such as Payments.refund(order_id: 2, amount: 150)
agent, request, call, verdict, resultthe JSON line's fields, as attributes
model, input_tokens, output_tokensfor an ask, so cost per agent is a query
service.nameservice in [log.otlp], or the package's name
  • Records go in batches, on the thread that sends the log events, so a slow collector never holds up an agent.
  • The file stays the record that counts. It is still written, and a collector that is down loses nothing from it.
  • The setting is per agent, and false by default.
  • Without a collector to send to, the program is refused before it runs:
agent Support {
    settings AgentSettings(audit: Audit(otlp: true))
    can Console

    on note(text: String) {
        print(text)
    }
}

fn main() {
    Support.start().note("never runs")
}

The compiler refused the program:

kavach run

error: `otlp: true` sends the audit log to a collector: add a `[log.otlp]` section with its `endpoint` to `kavach.toml` (spec 8.6, 14.12)
src/main.kav: Kavach code, then what running it prints

For analytics

Because an event is a record, every field is a column you can query: refunds per day, the total refunded per customer, how often a limit stopped an agent, from RefundBlocked's reason. A secret field is written as <secret> wherever the event goes.

Keep the durable journal out of your analytics: it is state, and holds secret values as they are.