When an agent sends an email on your behalf, approves a vendor contract, or updates your CRM, you need to be able to answer: who authorized this, when, and why? Our audit system is the answer — and building it at the scale Harnyss operates required some interesting engineering choices.
Append-only by design
The audit log is an append-only event store. No record is ever updated or deleted. Every event is assigned a monotonic sequence number per agent, a wall-clock timestamp, and a cryptographic hash of its content chained to the previous event's hash — making any tampering detectable.
{
"seq": 40291,
"agentId": "cmo-03",
"action": "send_email",
"target": "partner@example.com",
"authorizedBy": "approval:8821",
"hash": "sha256:a3f9...",
"prevHash": "sha256:c12e..."
}Handling 200k events per minute
At peak, our fleet generates roughly 200,000 audit events per minute across all customer orgs. Writing each event synchronously to our primary database would create a write bottleneck. Instead, we write to an in-memory append-only log partitioned by org and agent, flush to object storage every 500ms, and index asynchronously for queryability.
This gives us sub-millisecond write latency on the hot path, with query latency of under 200ms for the most recent 1,000 events and under 2 seconds for arbitrary range queries across 30 days of history.