Durable Agents

There is nothing to set: in a project, every agent keeps its journal in its folder, agents/<Name>/journal, beside kavach.toml. Here the bot pays a refund, then the power goes out before it sends the email:

use std.cli

type Receipt { id: String }
type PaymentError = Declined | Denied(reason: String)
type EmailError = NotSent | Denied(reason: String)

capability Payments {
    fn refund(order_id: Int) -> Receipt ! PaymentError
}

capability Email {
    fn send(to: String) ! EmailError
}

type Shop: Payments {
    fn refund(self, order_id: Int) -> Receipt ! PaymentError uses Console {
        print("[pay] refund {order_id}")
        Receipt(id: "r{order_id}")
    }
}

type Mail: Email {
    power_cut: Bool

    fn send(self, to: String) ! EmailError uses Console, Env {
        if self.power_cut {
            cli.exit(3)
        }
        print("[email] {to}")
    }
}

agent Bot {
    can Payments.refund, Email.send

    on request(order_id: Int) -> String ! Error {
        const r = try Payments.refund(order_id)
        try Email.send("order {order_id}: {r.id}")
        r.id
    }
}

fn main() ! Error {
    const power_cut = cli.args().contains("power-cut")
    const bot = Bot.start(Payments: Shop(), Email: Mail(power_cut: power_cut))
    if power_cut {
        const _id = try bot.request(7)
    }
    print("started")
}

Run it twice, first with the power cut, then normally:

[pay] refund 7

[email] order 7: r7
started

• The journal records each event, then the result of each capability call, then that the handler finished.
• At start, every event that never finished runs again from the beginning.
• A call already in the journal gives its recorded result without running: there is no second [pay] line, so the refund is paid once.
• The first call not in the journal, the email, runs for real.
• The first run's sender is gone, so no one gets the event's result. The outcome goes in the audit log if it is on, and an error nothing read is printed either way.
• Several events can be under way at once, each in its own task, as when one waits for a person while another runs. Each call goes in the journal under the event that made it, even a call from a task its handler spawned, so after a restart each event gets back its own results.
• When the agent starts, events that ended are dropped from the journal, so it stays small.
• A journal that can't be written, on a full disk, stops the program with a panic, as a database stops when its log fails: going on would break the promise.
• A call the crash cut off, before its result was written, runs again: each call runs at least once, and may run twice.

This works because a handler reaches the world only through capabilities, so a handler must be safe to run again from its start. If the program changed and a handler makes a different call than the journal recorded, the event stops with a panic. The journal holds secret values too, so keep it as safe as other secrets: the .gitignore that kavach new writes leaves agents/ out. journal: false in the agent's settings turns it off.

Time is replayed too

• A handler's clock reads (time.now_ms(), today(), now()) go in the journal, so a run after a crash sees the same time and makes the same calls.
• time.sleep records when the wait ends. After a restart it waits only for what is left, and never more than it was asked to, so a clock set back can't stretch it.
• every records each round's wait the same way. After a restart, rounds already waited for pass at once, and the next one waits only for what is left.
• A .timeout(d) records when it ends, so after a restart it times out when the first run would have, not a whole d later.
• Inside a capability's own code, the clock isn't journaled: the call's result is.

on remind(order_id: Int) ! Error {
    try Email.send("Your order {order_id} ships soon.")
    time.sleep(1.days)            // a crash an hour in: after the restart, 23 hours more
    try Email.send("Your order {order_id} has shipped.")
}

Each call has a key

A call the crash cut off runs again, so a service may see it twice. Each call in a handler has a key, the same in the run after a crash, and the capability's implementation reads it with agents.call_key(). Here the email crashes, then runs again with the same key:

use std.agents
use std.cli

type PaymentError = Declined | Denied(reason: String)
type EmailError = NotSent | Denied(reason: String)

capability Payments {
    fn refund(order_id: Int) ! PaymentError
}

capability Email {
    fn send(to: String) ! EmailError
}

type Bank: Payments {
    fn refund(self, order_id: Int) ! PaymentError uses Console {
        // A real bank would get the key, and drop a refund it has seen before.
        print("[pay] refund {order_id}, key {agents.call_key()}")
    }
}

type Mail: Email {
    crash: Bool

    fn send(self, to: String) ! EmailError uses Console, Env {
        print("[email] {to}, key {agents.call_key()}")
        if self.crash {
            cli.exit(3)
        }
    }
}

agent Bot {
    can Payments.refund, Email.send

    on request(order_id: Int) ! Error {
        try Payments.refund(order_id)
        try Email.send("order {order_id}")
    }
}

fn main() ! Error {
    const crash = cli.args().contains("crash")
    const bot = Bot.start(Payments: Bank(), Email: Mail(crash: crash))
    print("outside a call: {agents.call_key()}")
    if crash {
        try bot.request(7)
    }
}

outside a call: none
[pay] refund 7, key Bot-1-1
[email] order 7, key Bot-1-2

[email] order 7, key Bot-1-2
outside a call: none

• A key is the agent, the event's number and the call's place in it: Bot-1-2 is the second call of event 1.
• A service that is told the key can drop the repeat, as Stripe and other payment APIs do with an Idempotency-Key header.
• Outside a call from a handler, the key is none.

An event that never finishes

An event that stops the program each time it runs, from a bug, would stop the agent again after every restart. So after 3 starts it is set aside. Here the payment crashes the program 3 times, then the program starts normally and prints what the agent's folder holds:

use std.cli
use std.fs

type PayError = Declined | Denied(reason: String)

capability Payments {
    fn pay(order_id: Int) -> Int ! PayError
}

type Till: Payments {
    crash: Bool

    fn pay(self, order_id: Int) -> Int ! PayError uses Console, Env {
        if self.crash {
            cli.exit(3)
        }
        order_id
    }
}

agent Bot {
    settings AgentSettings(audit: Audit())
    can Payments.pay

    on request(order_id: Int) -> Int ! Error {
        try Payments.pay(order_id)
    }
}

fn main() ! Error {
    const crash = cli.args().contains("crash")
    const bot = Bot.start(Payments: Till(crash: crash))
    if crash {
        const _paid = try bot.request(7)
    }
    print("started")
    for line in try fs.lines("agents/Bot/set-aside") {
        print("set-aside: {line}")
    }
    for line in try fs.lines("agents/Bot/audit.log") {
        print(line.split("Z ").last() ?? line)
    }
}

started
set-aside: {"event":1,"handler":"request","args":[7]}
set-aside: {"again":1}
set-aside: {"again":1}
Bot set aside: event #1 of request (started 3 times without finishing)

• Each start of an unfinished event is written to the journal: {"again":1} is event 1 starting again.
• The 4th start moves the event's lines from the journal to set-aside in the agent's folder, writes it in the audit log, and starts the agent with the rest.
• A person can read the event there, fix the program and send it again. Message queues call this a dead-letter queue.
• A handler that panics counts too. Its send fails, but the event stays unfinished, so it runs again at the next start, until it is set aside.

One program per agent

An agent with a state file or a journal keeps them in its folder, and the program that starts it locks the folder (the file lock in it) while it runs. A second program that starts the same agent waits up to 5 seconds, so a restart can follow a program that is just ending. Then it stops with a panic, as a second database server refuses a data folder in use:

panic: the agent RefundBot already runs in another program: its folder agents/RefundBot is locked

A real run names the folder's full path. The lock is the operating system's, so a program that crashed holds none.