Statewave tops a memory benchmark we didn't write.
An open-source, self-hosted memory runtime for AI agents. It clears mem0 OSS on both LoCoMo and LongMemEval, run on mem0's own harness at gpt-4o, same eval loop, their judge unchanged, and matches the paid mem0 cloud tier.
A single run. Not a reproduction of mem0's published gpt-5 + Qwen figures.
The scoreboard
Three systems, one eval loop
The margins are small on purpose: a fair fight, not an inflated one.
Click a system to isolate it · switch the axis for full scale
LoCoMo
+0.039
vs mem0 OSS
LongMemEval
+0.134
vs mem0 OSS
Conditions of the run
Shared stack
gpt-4.1 extraction · text-embedding-3-small · gpt-4o answer + judge.
Single run
One run, not an average. LoCoMo (n=1,540) is the robust read.
n=30 on LME
A 30-question matched set with wide error bars. Directional only.
Asymmetry
mem0 cloud's extractor/embedder isn't configurable, a product-inherent asymmetry.
Methodology
One loop, one swap
A fork of mem0's harness, not a rewrite. Every run travels the same path; only the memory backend changes.
statewave_client.py + a --backend statewave dispatch› gpt-4.1 extraction shared by Statewave & mem0 OSS› judge & scoring code untouched from upstreamThree fixes, all in mem0's favor
Bugs in mem0's own client code. Every one of them raises the mem0 backends' scores, and every one is applied in the fork rather than withheld — so the scoreboard above runs against their best config, not a strawman.
Cloud v3 add URL
Without it cloud ingested nothing. This fix is what lets it score at all.
applied to mem0 cloud
OSS v2 search-filter
Corrected so queries return the intended memories, not an over-filtered subset.
applied to mem0 OSS
OSS date grounding
Grounds the session date back into message content so time-anchored questions resolve.
applied to mem0 OSS
Reproduce it
Reproduce every number here
Copy-pasteable, straight from the harness README. LoCoMo shown; for LongMemEval swap in benchmarks.longmemeval.run with --per-type 5.
- 1
Clone & install
bash$ git clone https://github.com/smaramwbc/statewave-memory-benchmarks.git$ cd statewave-memory-benchmarks$ pip install -r requirements.txt$ export OPENAI_API_KEY=sk-... # answerer + judge - 2
Run the harness
$ export STATEWAVE_URL=https://your-instance$ export STATEWAVE_API_KEY=sw-...$ python -m benchmarks.locomo.run \--backend statewave \--answerer-model gpt-4o \--judge-model gpt-4o› Statewave Honors the harness top-200 retrieval request.
- 3
Check your numbers
Published result · Statewave
axis 0.80–1.00
- LoCoMon = 1,540
- 0.905
- LongMemEvaln = 30
- 0.967
The same figures charted on the scoreboard. Answerer and judge are both gpt-4o, so a rerun can land slightly either side of these.
Scope
What we claim, and what we don't
An honest benchmark is worth as much for where it stops as for what it shows.
Leads its open-source peer in this run: LoCoMo +0.039 at n = 1,540, LongMemEval +0.134 at n = 30.
Matches the paid cloud tier, 0.905 vs 0.899 on LoCoMo, while staying free and self-hosted.
Holds against mem0’s best config; our client fixes are applied to their backends, not withheld.
Reproduces from one public, Apache-2.0 code path with mem0’s judge unchanged.
Not a reproduction of mem0’s published gpt-5 + Qwen figures.
No win over the paid cloud tier: 0.905 vs 0.899 is inside single-run noise.
Not a multi-run average, so nothing here says a margin repeats on a rerun.
No category-level or per-type breakdowns beyond the aggregate scores.
No long-context BEAM score. The harness runs, but no number is claimed.
LongMemEval (n=30) is directional, not a significance test.
Beyond retrieval
What mem0 doesn't do
Retrieval is table stakes, and the run above puts Statewave level with the paid tier. Statewave's real difference is governance: the controls a memory layer needs before it touches production data.
| Capability | mem0 cloud | Statewave |
|---|---|---|
| Retrievalwhat the benchmark measures | ||
| LoCoMoAggregate score on mem0’s own harness, gpt-4o answerer and judge. | 0.899 | 0.905 |
| LongMemEvalSame harness, same judge, 30-question matched subset. | 0.933 | 0.967 |
| Governancewhat it doesn't | ||
| Access policiesScope what each agent and tenant can read or write, enforced at retrieval time. | Not offered | Included |
| Sensitivity labelsTag memories by sensitivity and keep classified content out of the wrong context. | Not offered | Included |
| Tamper-evident auditEvery write and read leaves a verifiable receipt you can replay after the fact. | Not offered | Included |
| ProvenanceTrace any retrieved memory back to the exact source turn it came from. | Not offered | Included |
mem0 cloud is the paid tier — their strongest showing on the retrieval rows, which carry the single run above.
FAQ
Questions you're right to ask
No spin. The awkward questions, answered directly.
Why gpt-4o and not gpt-5?
mem0's headline figures use gpt-5 + Qwen. We standardized on gpt-4o as a shared answerer and judge across all three backends so the only variable is the memory layer. A cleaner comparison, not a reproduction of their numbers.
Isn't n=30 too small on LongMemEval?
Yes, treat it as directional. It's a matched 30-question subset with wide error bars: one question moves the score by 0.033, so the whole spread between the three backends is a handful of answers. LoCoMo at n=1,540 is the robust signal, and the margin over mem0 OSS there is the only one wide enough to read as more than run-to-run noise.
Did you tune Statewave and handicap mem0?
The opposite. Three client fixes we shipped raise mem0's own scores; without the cloud v3 add-URL fix, cloud ingested nothing. The scoreboard runs against their best config, not a strawman.
Why run on mem0’s harness instead of your own?
So the framing isn't ours to bend. The judge and scoring code are unchanged from upstream; only the memory backend swaps. You can diff the fork against upstream line by line.
It's one run. Can I trust it?
Don't take our word for it. The harness is Apache-2.0 and copy-pasteable: clone it and re-run every number yourself. It's a single run, so read small margins as noise: a rerun can land slightly either side of these figures.
What is Statewave, exactly?
An open-source memory runtime for AI agents: the layer that ingests, stores, and retrieves what an agent needs to remember. These benchmarks measure that retrieval quality head-to-head.
Don't take our word for it. Fork the harness
Apache-2.0, one code path, mem0's own judge unchanged. Clone it, diff it against upstream, reproduce every result.