Every tool in this space leads with a metric — tokens saved, ingestion time,
query latency. EKOS has never published one. This is the first: a real repo, a real question,
raw grep vs the compiled ledger, both sides counted with the same standard tokenizer
(tiktoken, not a hand-rolled estimate). It doesn't win every case —
the one case grep wins is on this page too, not edited out.
$ ekos init && time ekos build
Build complete. Files observed (new): 2022
real 0m33.587s
A genuinely cold run — the workspace deleted and reinitialized, not a warm-cache
rerun — against a real, unmodified 2,022-file open-source repository (Plausible Analytics: Elixir
application code, two real SQL schemas, 500 real git commits, 14 real CI pipelines, 41 real
documents). The full observe → compile → ledger pipeline, including
identity resolution and commit, finishes in under two minutes total — see the full-loop deck for
the complete stage-by-stage timing.
| Tier | What it assumes |
|---|---|
| Best-case | The agent already knows the exact file and line to grep — -A15 around the one right spot. |
| Realistic | The agent greps the whole repo with context lines, the way an agentic coding tool actually does — every real match, not the lucky one. |
| Naive | The agent found the one correct file and just reads it in full. |
Publishing only the "best-case" number would be the easy, flattering benchmark —
and it's the one where grep wins (see §04). Every tier here is a real command
against the real repo, not a synthetic estimate. On the EKOS side, the same "doesn't know where
to look yet" reality applies: the token cost includes an ekos_search
lookup plus the ekos_state fetch, not just the final answer in
isolation.
| Path | Real tokens (tiktoken) |
|---|---|
| grep, realistic (32 real hits, -C3 context) | 10,357 |
| grep, naive (read the whole 366-line file) | 3,651 |
EKOS (ekos_search + ekos_state, real MCP) | 1,186 |
32 real files in this repo mention imported_browsers —
test fixtures, migrations, application code, the one real schema file. A search agent without a
compiled ledger has to sift through that; EKOS's ekos_search ranks the
real Table object first, and ekos_state
returns exactly its 10 real columns, typed, with evidence — nothing else.
$ grep -A 15 "CREATE TABLE plausible_events_db.imported_browsers" \
priv/ingest_repo/structure.sql
CREATE TABLE plausible_events_db.imported_browsers
(
`site_id` UInt64, `date` Date, `browser` String, ...
)
ENGINE = MergeTree ORDER BY (site_id, date, browser)
SETTINGS index_granularity = 8192, ...;
122 tokens. EKOS's full round trip: 1,186 tokens — 9.7× more.
This is real and stated plainly, not smoothed over: if the agent already knows the exact file and the exact line before it searches, a targeted grep is unbeatably cheap — there was nothing to discover. That's not a realistic starting condition for "what does this table contain?" on a repo the agent hasn't already memorized, which is why §03's realistic and naive tiers are the honest headline number, not this one. A benchmark that only showed this slide would be marketing, not evidence.
$ ekos ekl "FIND Object WHERE kind = 'Table' AND name CONTAINS 'public'"
id name kind
951e87a2-... public.api_keys Table
b26a68a7-... public.check_stats_emails Table
c5dfe0c6-... public.create_site_emails Table
...
42 row(s). 1,258 tokens
There's no grep pattern that cleanly lists "every real table" without also
matching migrations, tests, and ALTER TABLE statements naming the same
tables — the realistic raw-source path is reading the whole 2,738-line dump.
ekl answers directly from the already-compiled ledger.
| This benchmark | Does not claim |
|---|---|
| Real repo, real questions, real tiktoken counts | A comparison against any named competitor's own published numbers — none were reproduced here, none are claimed |
| 67-93% fewer tokens, realistic and naive tiers | A win in every case — the best-case grep tier (§04) genuinely costs less |
| Two representative real questions | A statistically broad sample — more questions, more repos, is exactly what "rough" means here |
| Token cost, tiktoken-measured | A latency or wall-clock-per-query benchmark — not measured in this pass |
The point isn't to win a leaderboard against tools this project hasn't measured — it's to exist, with real numbers a skeptic can rerun themselves against the same public repo.
$ git clone https://github.com/plausible/analytics $ cd analytics && ekos init && time ekos build $ ekos recover && ekos resolve && ekos compile && ekos commit $ ekos mcp serve --workspace . # try ekos_search + ekos_state yourself $ pip install tiktoken # count the tokens yourself
| Everything behind this deck | Where |
|---|---|
| Raw commands, grep outputs, MCP responses | docs/presentations/examples/token-benchmark/ |
| Token-counting script | count-tokens.py — tiktoken, cl100k_base, no hand-rolled estimate |