Small numbers, including our misses.
Everything here comes from our own replays and from test cases written by separate agents. Each set is counted on its first run, before we changed anything it found. Nothing is rounded up.
What worked
33/36
Bad transactions blocked on first contact with test cases written by separate agents (18 of 20 and 15 of 16).
What we got wrong
5/17
Normal transactions wrongly stopped on those same first runs. Our biggest problem is false alarms, not misses.
What we got wrong
1 in 4
Normal transactions still interrupted on real traffic of 5 public agents (30 of 120), under rules we had to guess. Fixing this comes first.
Replay
21
Transactions of the Bankr/Grok wallet replayed: oathline would have blocked the 4 May drain, and 3 normal looking transfers too, under a rule we had to guess.
What we got wrong, so far
- On one real agent that runs its own contracts, our first look blocked 11 of 30 new transactions, all apparently normal: oathline could not read those contracts, and a call it cannot read is at best asked about. Since 9 October your rules can name your own contracts; we have not measured this agent again on new traffic yet.
- In the Bankr replay, one normal transfer was counted as blocked because our replay failed, not because of the rules.
- The hosted copy of Base can be up to two minutes old, so a transaction that depends on something seconds earlier can be reported as “fails in simulation”.
- oathline is not externally audited yet. The signed log and the MIT licensed SDK are open to inspection; the cosigner signs for real money first for a few design partners who accept that it is not externally audited, and a full outside audit comes after we raise money.
Limits check amounts. Scanners check reputations. oathline checks the effect.
| Kind of check | Asks | Helps catch | Does not see |
|---|---|---|---|
| Spending limits and allowlists | Is it under the cap? Is the address allowed? | Oversized payments, unknown recipients | A normal looking transaction that does something else, a permission that drains later, a swap at a terrible price |
| Threat scanners | Is this address or token known to be bad? | Known scams, flagged contracts, obvious drains | A fresh attacker address, a token the attacker just made, a transfer that is simply against your wishes |
| oathline | Is this what the owner asked for? | The real effect of the exact transaction against your rules: what leaves, what enters, who gets permission | Anything outside the transaction it was shown, like a key stolen and used elsewhere |
oathline uses outside reputation data as one input and an AI judge only for grey cases. Fixed rules decide the clear ones.
Every figure has a run behind it.
- First contact test sets: two sets of scenarios written by separate agents (20 and 16 bad transactions, 7 and 10 normal ones), each run once before any fix. That is where 33 of 36 and 5 of 17 come from.
- Real traffic: past transactions of 5 public agents on Base, replayed on a copy of the chain at the state just before each one, under rules we had to guess for each owner. 30 of 120 normal transactions were interrupted.
- The Bankr/Grok wallet: all 21 outgoing transactions, including the drain of 4 May 2026, replayed the same way. oathline would have blocked the drain under the rule “pay only addresses the wallet had paid at least twice before”; the same rule would have blocked 3 transfers that look normal.
- The test network run: on 5 October 2026 a two of three Safe on Base Sepolia made one routine payment that oathline cosigned with its key in AWS KMS, and oathline refused four attacks, including a delegatecall to an unknown address. Anyone can look the Safe up on sepolia.basescan.org.
- Speed: 118 checks on the benchmark, 1 October 2026: median 1.2 seconds, slowest 10% at 7.7 seconds or more.
What comes next: Break the guard at Devcon (3 to 6 November), a public game where anyone tries to trick our agent into paying them, with the signed log as the scoreboard; the Safe cosigner on Base for a few design partners; faster checks shown live here; and a full outside audit after we raise money. Planned, not promised.
See your own numbers.
Request an invite. Where your wallet’s history is public, we can replay it first, so you see the false alarms before you install anything.
Request an invite