The harness found weak answers.
The diagnosis found a ledger bug.

RFC 0138 exists to catch exactly this. It did its job.

A previous session's most complete eval run (the pre-Phase-2, 32-scenario suite) came back Status: FAIL against every gate — 45.8% answer correctness against an 85% bar, 12.5% hallucination against a 5% ceiling. That was logged as an open finding, not silenced: the harness doing its job is not a bug. Two things were still undone: (1) the current 101-scenario, 7-category dataset had never been run end-to-end, only the older 32-scenario version had; (2) nobody had yet looked at why specific answers were scoring low — matcher strictness, a genuinely wrong answer, and a contaminated evidence set all produce the same low number, and only reading the real answers tells them apart.

This report's purpose is narrow and honest: run the full baseline, diagnose real failures instead of guessing, fix what diagnosis actually finds, and report the before/after numbers plainly — including the parts that didn't improve.

Five deterministic evaluators. No LLM judge. Same suite, twice.

Dataset: ekos-full, 101 scenarios across 7 categories — architecture (20), code (15), dependencies (12), lineage (12), history (12), security (12), adversarial (18). Agent: ollama llama3:latest, the model this workspace's own ekos.toml is configured with (no ANTHROPIC_API_KEY/OPENAI_API_KEY was set in this environment, so an A/B against a stronger provider stays an open follow-up, not part of this comparison). Gates: correctness ≥85%, groundedness ≥90%, completeness ≥80%, recall@10 ≥80%, hallucination ≤5%.

Procedure: (1) run the full suite cold against the existing ledger — the contaminated baseline; (2) reproduce individual low scorers live via ekos ask to find a real failure pattern; (3) fix what that diagnosis actually found; (4) wipe .ekos/ and rebuild the whole pipeline from scratch (init → build → recover → resolve --force → compile → commit); (5) re-run the identical suite, same agent, same gates — the clean baseline. Only the ledger's underlying data changed between the two runs.

The model cited its own exam paper as the evidence.

$ ekos ask "What module redacts secrets and PII before content reaches
             the artifact store or ledger?"

"...the module responsible for redacting secrets and PII is the one
mentioned in `evals/datasets/security.yaml`."

Sources:  evals/datasets/security.yaml  (should never be a source)
          ekos/crates/common/src/redaction.rs  (the real answer, present but not named)

Scenario sec-001 expects the word "redaction" in the answer. The real module (redaction.rs) was in the model's evidence set — but the final answer names the eval's own question file back at itself instead. Confirmed directly against the ledger, not inferred from the answer alone:

$ ekos query find "FooBarNonexistentAnalyzer" --mode lexical
1 result(s): evals/datasets/adversarial.yaml  ← an intentionally-fake
                                                       name that exists nowhere
                                                       in EKOS's real source

The adversarial scenario built around that fake name (adv-001 ) did correctly refuse to fabricate a description — no hallucination there — but its stated reason was circular: "the only available information... comes from a search match in the evals/datasets/adversarial.yaml file." A separate, architecturally unrelated question (arch-001, "what does EKOS compile enterprise systems into?") got a plausible-sounding but flatly wrong answer — "a runtime environment" instead of "the Canonical Knowledge Model" — with no evals/ path anywhere in sight, showing the two problems are genuinely separate.

Two missing exclusions. One ~34% smaller, cleaner ledger.

ekos.toml's [observe] ignore-patterns was missing two directories — the same contamination class already fixed once for doc/doc-sa/doc-objects (devlog_90):

+ "evals",       # RFC 0138's own scenario YAML + saved JSON reports
+ "test-runs",   # 2.3GB local, git-ignored leftover from the RFC 0111/0113
+              # E2E soak test (devlog_144) -- git-ignored is not the same
+              # as ekos.toml-ignored

test-runs/ was the larger of the two: a duplicated 95-partition Elixir workspace plus MySQL/MSSQL DB-script fixtures, fully re-observed on every build despite being correctly excluded from git. It alone caused a 376-conflict ekos resolve hard-stop (generic names like generate /validate/json colliding across Rust/Python/Elixir/JS symbols from that test corpus) and roughly a third of the ledger's total object count.

Some metrics moved. Answer correctness didn't.

Category pass counts, contaminated → clean: adversarial 7→10/18, architecture 9→8/20, code 8→7/15, dependencies 11→11/12, history 2→2/12, lineage 6→7/12, security 4→3/12. No gate passes in either run — both are still Status: FAIL.

* Correction (2026-09-07). The −10pp recall@10 line above was reported here as a regression. It is not one — it is noise, and the follow-up work says so plainly. Only 10 of the 101 scenarios carry expected_objects, so recall@10 is a mean over ten values and the "10-point drop" is exactly one scenario flipping from 1.0 to 0.0. The right response is to raise the metric's statistical power, not to tune ranking against a single data point — which is what RFC 0139 does. Left visible rather than quietly edited out, because a published number that turned out to be over-read is worth more as a correction than as a deletion.

Second correction (2026-09-08). The column headed Clean above is not clean. Both runs in this table were compiled from a corpus that was 94% not our code — a Python virtualenv and two scanner-output directories that neither exclusion covered. §07 has the measurements, and §09 has the run against a genuinely clean ledger. Kept in place for the same reason as the first correction: a published number that turned out to be over-read is worth more as a correction than as a deletion.

Worth fixing on its own merits. Not the dominant cause.

The honest conclusion: ledger self-contamination was real and worth fixing — a system should never cite its own exam paper as evidence — but it was not the dominant cause of weak answer quality. llama3:latest's own answer-generation quality (see arch-001, §03) remains the primary open gap. The original TODO's next steps are now the clearer priority: A/B a stronger provider once an API key is available, raise the recall@10 metric's statistical power, and only after that, revisit prompt/retrieval tuning.

What happened next (2026-09-07). Following that conclusion led to RFC 0139, and the conclusion above turned out to be half right. "The model is the primary gap" was the wrong headline: capturing what the model was actually shown revealed that lexical search returned nothing at all for 80 of 89 scenarios, and that 26 completely different questions were being answered from byte-identical evidence. The answer was usually not in the context to find. Relaxing the query lifted answer correctness 37.6% → 48.1% — and, exactly as that RFC predicted it might, briefly made fabrication worse (10 → 15) until a weak-evidence refusal guard was added. The lesson worth keeping: this page's own numbers were honest but under-diagnosed, and only capturing the transcripts settled it.

The ledger this deck called clean was 94% not our code.

§04 and §05 above are built on a run labelled clean because evals/ and a 2.3 GB test-runs/ directory had been excluded. Both exclusions were real. Neither was close to sufficient — and nothing in the metrics said so. What eventually said so was a sample of cited claims: every one pointed into web/api/.venv/lib/python3.13/site-packages/.

Both were self-inflicted, like evals/: they exist because this project runs its own tooling on itself. A compiler that reads its own workspace keeps eating its own exhaust.

The exclusions removed no knowledge. After dropping 9,130 files, recover produced identical counts: 2,623 Rust symbols, 1,734 Calls edges, 128 JS modules, 2,518 JS symbols. That equality is the check that separates a correct exclusion from an over-aggressive one — "fewer objects" on its own is indistinguishable from "lost knowledge."

Claims now cite a place in a file, not just a file.

Every claim the reasoner saw was a name and, at best, a file path. Measured across a full run: 0 of 1,289 rendered claims carried a line number; 26.4% carried any location at all. Analyzers had recorded a source_span per symbol since RFC 0088 — but never attached a file to it. A symbol knew it lived at lines 200-322 of an unnamed file.

The feature shipped, was verified, and did nothing — twice. Both failures presented as success, which is the part worth keeping:

Both now carry guard tests — each verified by breaking what it guards and watching it fail. For a bug class defined by looking correct, a passing test is not evidence.

Real gains. And one real regression.

The regression is stated, not buried. Six scenarios fabricated where three did before. A plausible mechanism: with third-party noise gone, an adversarial question now retrieves real EKOS objects, which read as more convincing evidence and make the model likelier to answer than refuse. That is a hypothesis awaiting the transcripts, not a finding.

The flattering baseline we did not use. Re-grading the original saved answers under the current ruler reports 20/101, which would let this deck claim 20 → 43. It is invalid: that report recorded a ranked result list for only 10 of 101 scenarios (today's records 101/101), so re-grading it penalises 91 scenarios that never captured the data the newer ruler asks for. The comparison above uses the post-RFC 0139 run instead — same ruler, same provider, so only the corpus differs.

Two measurement defects found in the harness itself while reading this run: the report line "Answered but uncited: 57 (excluded from groundedness)" is mislabelled — only 10 are excluded, 44 score 0.0, so the metric is more honest than its own caption claims. And recall@10 is still a mean over just 17 scenarios carrying expected_objects, so it remains too low-powered to compare across runs — the same statistical-power problem §05's correction named, only partly addressed.

Bit-identical headline scores. That's a good sign.

Five days later (2026-09-14), a further round of fixes shipped — RFC 0139 Phase 2 (a bareword "or" in a fallback search treated as a required literal word; recall@10 graded against the raw question instead of the query the REASON pipeline actually issues), RFC 0140 §3/§4 (on-demand source text; an opt-in LLM rerank, off by default), and RFC 0141 (a real signature property on Rust/Python/Elixir functions; attributes on Calls edges). Same procedure as every prior section: real recover/resolve --force/compile/commit rebuild, then the identical 101-scenario suite, same agent, same gates.

The bit-identical scores are a real finding, not a stalled measurement. Every LlmProvider implementation is contractually required to run at temperature: 0 (RFC 0008) — an unchanged prompt has no source of entropy left to produce a different completion from a local model. Landing on the exact same 8-significant-figure score twice confirms this comparison is measuring like-for-like, not noise.

* The recall@10 line is a correction, not a regression — the same class of finding as §05's. Diffing every scenario's own retrieval_recall between the two reports: exactly one scenario changed. code-002 flipped from 1.0 → 0.0. Before today, the harness graded recall against the raw question sentence — a different, unrelated query from what the REASON pipeline actually searches with — and that raw-question probe happened to retrieve the right object for this one scenario, reporting a perfect score for a search the pipeline never ran. RFC 0139 Phase 2 fixed the harness to grade the pipeline's real query instead, and it now honestly reports zero.

$ ekos query find "LlmProvider" --mode lexical
  a83c0280...  build_llm_provider   ← now ranks #3, was outside the top ten

$ ekos ask --explain "What function builds the LlmProvider used by ekos ask, ..."
query type: Structural, routing confidence: 1.00
plan: Compose[ Graph Dependents from LlmProvider (2 hops), Fact LlmProvider.* ]
Insufficient evidence. ... does not mention the specific function ...

code-002 still fails — and it is a new, separate, real finding, not a failed fix. RFC 0141's signature property demonstrably works: build_llm_provider moved from outside the top ten to #3 for a bare lexical search, verified directly above, independent of the eval score. But code-002's own question resolves "LlmProvider" as an exact-name entity match with confidence 1.00, so the planner routes to a direct dependents-graph and fact dump on the trait itself and never calls Search at all — the function the fix made rankable never gets a chance to surface, regardless of ranking quality. A retrieval-routing gap, not the retrieval-ranking gap RFC 0141 closed. The refusal shown above is honest and correct given what the model was actually shown.

13 scenarios flipped pass/fail (6 to pass, 7 to fail, net −1). Spot-checked the two adversarial flips: one answer opens "REFUSING." instead of the prompt's required exact opening ("Insufficient evidence.") — a real refusal in intent, graded as a fabrication on wording; the other hedges rather than committing. Neither traces cleanly to a specific fix shipped this round — read as local-model wording variance on borderline prompts, not concluded further here.

Not the routing fix. Three cheaper bugs, found by classifying every failure.

The code-002 routing gap from §10 looked like the next obvious fix. A multi-round classification of all 59 then-failing scenarios said otherwise: a bigger, cheaper, previously-undiscovered bug was hiding in plain sight — Search/Graph evidence claims rendered only an object's name, discarding the exact text BM25 had already matched against. Alongside it, a citation-block parser that rejected a bare claim position or an "evidence <id>"-wrapped uuid, and a refusal-wording mismatch where the model's own echoed prompt header ("REFUSING.") wasn't recognised as a refusal at all. Fixed all three, plus three grading-ruler defects that would otherwise have inflated the measured gain. The routing fix from §10 is re-sequenced to run after this, not before — a re-routed plan only helps if the evidence it finds can render its content, not just a name.

Measured category-by-category, not in one run. A single full-suite background run was OOM-killed twice by the host (real memory pressure, not a fluke — confirmed via free -h). Each of the 7 categories runs comfortably inside a single foreground call, and a fresh process per category releases memory before the next one starts — that's what actually produced this number, at the cost of manually diffing 7 report files afterward instead of one.

22 scenarios flipped to pass, 11 flipped to fail — both reported, not just the net. The code category alone went 3/15 → 8/15, the single largest jump, almost entirely the citation-parsing and evidence-content fixes landing on scenarios whose answers were already correct and only the citation mechanics were broken.

$ ekos eval run --dataset ekos-full --category code --agent ollama --save-answers
Scenarios: 15   Passed: 8   (was 3/15 before Phase 1)
Evidence groundedness: 75.0%   Recall@10: 57.1%

Two real side effects, measured and reported, not hidden. Richer excerpt text on weak/partial-overlap claims made the model fabricate on 4 adversarial false-premise questions it previously refused correctly — exactly offset by 4 different adversarial scenarios the wording fix corrected, net zero for that category but a real trade-off between generosity of evidence and adversarial safety. Separately, restyling the prompt's headers to a bracketed form coincided with 3 scenarios where the model wrote Cited evidence: [...] in prose instead of the required JSON block — history's only net regression (4/12 → 3/12). Neither pursued further this pass; both are open follow-ups, not swept under the rug.