Most memory systems ingest asynchronously — a fact you save can take minutes to hours to become retrievable, or only lands at the end of a session. brainbox writes and indexes synchronously, so we benchmark the tier others can't enter: same-turn write→recall, against a corpus built to be hostile. Full method below; every number here comes from clean-room runs on an isolated tenant with a fixed seed.
Last run: 13 July 2026 · three clean-room runs · full method and per-run data below · public harness repo ships with the launch
Rank-1 precision under near-twin interference came back at 0.39 — our distractors (same job, different city; same event, different city) tied with the right answer and the tie broke randomly. Impossible queries never triggered an "I don't know." We built the benchmark to be adversarial; it was.
Live probes showed same-template candidates collapsing onto byte-identical relevance scores — a tie cluster — and cosine similarity scoring 0.88 for provably unanswerable queries, far above any usable threshold. An independent adversarial review of our first fix caught it attacking weights when the bug was the formula's shape.
Completeness-weighted term coverage plus deterministic tie-breaking took rank-1 from 0.39 to 0.90 on the identical corpus and seed — entity 0.84, preference 0.92, temporal 0.94 — while top-5 stayed perfect and latency improved. Same benchmark, same hostility, third clean room.
| Tier | Query mode | n | hit@1 | 95% CI | hit@5 | MRR | p50 | p95 |
|---|---|---|---|---|---|---|---|---|
| Fresh (same-turn) | semantic | 75 | 0.933 | [0.853, 0.971] | 1.000 | 0.967 | 1.36s | 1.96s |
| Fresh (same-turn) | keyed | 75 | 1.000 | [0.951, 1.000] | 1.000 | 1.000 | 1.40s | 1.93s |
| Interference (headline) | semantic | 150 | 0.900 | [0.842, 0.938] | 1.000 | 0.950 | 1.34s | 1.76s |
| Interference | keyed | 150 | 1.000 | [0.975, 1.000] | 1.000 | 1.000 | 1.35s | 1.94s |
| Cold (separate session) | semantic | 150 | 0.900 | [0.842, 0.938] | 1.000 | 0.950 | 1.04s | 1.15s |
| Cold (separate session) | keyed | 150 | 1.000 | [0.975, 1.000] | 1.000 | 1.000 | 1.03s | 1.15s |
Headline = interference tier, semantic mode: paraphrase queries, no shared identifiers with the target, after the entire corpus (golds + distractors) is written. Keyed mode (query contains the memory's unique token) is a fulltext-precision check — we report it, we don't headline it, because it's easy by construction. The fresh tier carries a recency advantage, so it's a freshness claim, not the headline. The cold tier runs hours later in a separate session: semantic rank-1 held at 0.900 — identical to warm — where the pre-fix worker had degraded to 0.23. Recall is not warm-cache theatre.
Our negative probes ask for combinations that don't exist ("the actuary in Porto" when the corpus has actuaries and Porto, never together). The system currently returns its closest matches instead of saying "nothing connects these." The detection signal exists (no candidate covers the full query), and a qualified-answer behaviour is on the roadmap — we ship the 0/30 disclosed rather than gaming the metric with a threshold tuned to this corpus.
Facts and paraphrases are generated deterministically (seed 42) with uniqueness guarantees, which makes runs reproducible but means query phrasing is systematic. A human-written blind probe set is being added before the public harness release.
No third party has verified these numbers yet. That's exactly why the harness ships with the method, seeds, and per-run state files — run it against your own tenant and check us. Industry-standard suites (LongMemEval) are planned next; their scores will be published here whatever they come out to be.
Corpus. 150 gold facts across three types (entities, preferences, dated events), each with a unique canary token and a semantic key that is unique among golds — every probe has exactly one correct answer. Each gold gets a near-twin distractor sharing half its key (same profession, different city), so rank-1 requires beating a near-twin, not an empty field. 30 negative probes ask for combinations absent from the corpus.
Probes. Every gold is probed two ways: keyed (contains the canary — fulltext check) and semantic (paraphrase, no canary). Three tiers: immediately after each write, after the full corpus is in, and in a later cold session. Rank is parsed from the API's structured output; hits require the canary in the returned text.
Discipline. Isolated evaluation tenant, wiped to zero nodes before every run. Fixed seed. Retrieval parameters at production defaults. Latencies measured client-side through the live public API, paced within production rate limits. Confidence intervals are Wilson at 95%.
Same-turn recall is the tier others can't enter. LongMemEval-S is the tier everyone competes on — 500 questions over long, multi-session chat histories, across six reasoning types. We ran the whole set through our unchanged recall pipeline and scored it with LongMemEval's official gpt-4o judge — the exact scorer Zep and Mem0 publish against. Not our proxy metric; their pipeline. An independent adversarial pass re-computed the number before it went here.
First subset came back at 29% — but retrieval hit the evidence session ~97% of the time. The reader was answering "I don't know" with the answer already in its context. Reader-limited, not memory-limited — the opposite of where most systems fail.
Adopting LongMemEval's own reader prompt (extract-then-reason, no reflexive "I don't know") and putting the session date into each memory took temporal reasoning from 0 to 62%. 60.4% on a 48-question subset — parity with Zep+gpt-4o-mini — with the fix isolated to the reader, not the retrieval.
A single eval node hit a database-churn ceiling at ~q114. We sharded the run across five ephemeral nodes, merged, and scored once. 65.3% on the full set — independently re-computed by an adversarial second pass before publication.
| System (reader) | Overall | ss-user | ss-asst | ss-pref | multi-sess | temporal | know-update |
|---|---|---|---|---|---|---|---|
| brainbox (gpt-4o) | 65.3 | 81.4 | 94.6 | 13.3 | 51.9 | 66.2 | 71.4 |
| Zep (gpt-4o) | 71.2 | 92.9 | 80.4 | 56.7 | 57.9 | 62.4 | 83.3 |
| Full-context (gpt-4o) | 60.2 | — | — | — | — | — | — |
| Mem0 | ~49 | — | — | — | — | — | — |
brainbox = openai/gpt-4o via OpenRouter, k=30, session granularity, seed 42, unchanged recall pipeline. Comparators are the published numbers from the Zep paper (arXiv:2501.13956, Jan 2025, Tables 2–3), which uses the same official gpt-4o judge. Bold = brainbox's category wins. We report judged accuracy, not evidence-hit rate — a fact existing in the retrieved set is not the same as answering the question, and only the second is honest to publish.
On single-session-preference the reader gives generically helpful advice without recalling and applying the user's stated preference, so the judge marks it wrong. It is nearly the entire 6-point gap to Zep — a reader failure, not a retrieval one (the evidence is retrieved). Lift this one category to even ~50% and the overall lands near 68%. It's the next fix, disclosed rather than buried.
We call openai/gpt-4o via OpenRouter; the Zep paper likewise reports "gpt-4o" without
a pinned snapshot. A few points of the gap could be reader drift rather than memory-system difference.
The judge is pinned (gpt-4o-2024-08-06, the official scorer) for both.
The brainbox row is our own run. The Zep, full-context and Mem0 rows are the figures published in the Zep paper, on the same dataset and the same official judge. No third party has re-run brainbox yet — which is why the harness ships with the method, shard seeds and per-run state files. Run it and check us.
Beta cohort 2 is five seats. You get an isolated tenant, the connector that works across Claude, Cursor, Codex and the rest of the verified list — and the harness to hold us to these numbers.
Request a seat What is brainbox?