We moved EKOS's answering model from a local 8B model on Ollama to DeepSeek V4 Flash through OpenCode Zen, then fixed how EKOS decomposes documents and plans a query. The same checked-in 101-question suite, graded deterministically, went from 53 to 87 passed. This deck separates what the model bought from what the engineering bought, and names what still fails.
ekos-full dataset · agent: deepseek-v4-flash via OpenCode Zen · RFC 0144 · RFC 0145 · devlog_183–186 · 2026-09-15llama3:latest (8B) on Ollamabars drawn to scale · full width = 101 scenarios
The part of EKOS that builds knowledge — observing repositories, databases and documents, recovering tables, symbols, call graphs and lineage, resolving identities, writing an append-only ledger — uses no LLM, and every fact carries the evidence it came from.
People don't ask a ledger questions in EKL, though. They ask “what depends on the redaction module?”. The LLM sits at the very end: it receives a small, typed evidence set the planner already assembled, writes the answer in plain language, cites the claims it used, and must say “Insufficient evidence” when the evidence doesn't answer. It never touches the source systems.
| Where | What the LLM does |
|---|---|
| ekos ask / REASON | Writes the final answer from the evidence set and cites it. This is what the 101-scenario eval grades. |
| ekos eval run | Drives the suite against a real workspace. Grading itself is deterministic keyword/id matching — no LLM judge. |
| document semantics · opt-in | Extracts Concept objects and their relationships from prose documents. |
| architecture reasoning · opt-in | Classifies crate roles; results are written as reviewable, never as fact. |
| llm-description · opt-in | Adds evidence-grounded ai_overview / ai_usage text to modules and symbols. |
| docs --prose · rerank · NL-to-SQL | Prose overviews in generated docs, an opt-in LLM rerank of evidence, and the gated ClickHouse query bridge. |
num_ctx, so Ollama used its small default window and dropped the start of longer prompts — where the instructions live. The largest prompt recorded under llama3 was 3,875 tokens, right at that ceiling. Fixed in RFC 0145 ([llm] context-window).cited_evidence block and refuse false premises. llama3 drifted into prose citations and fabricated on adversarial questions; hallucinated answers fell from 7 to 3.| Category | Before | After | Passed share | Change |
|---|---|---|---|---|
| Architecture | 9/20 | 18/20 | +9 | |
| Code | 8/15 | 13/15 | +5 | |
| Dependencies | 10/12 | 10/12 | 0 | |
| Lineage | 5/12 | 10/12 | +5 | |
| History | 3/12 | 11/12 | +8 | |
| Security | 7/12 | 10/12 | +3 | |
| Adversarial | 11/18 | 15/18 | +4 | |
| Total | 53/101 | 87/101 | +34 |
References edges — 7,377 doc→code and 3,420 RFC→RFC on EKOS's own repository. Ambiguous names never link.ekos_common::redaction resolve as one name; generic words (“crate”, “record”) no longer match unrelated objects.ekos commit since 10 September had left the search index stale for read-only queries; answers truncated at a lower token limit kept replaying from cache. Both fixed with regression tests.The compile side of EKOS needs no GPU, no network and no model. Answer quality tracks the class of model you can serve: a workstation-sized local model gets roughly the “before” column. Reaching the “after” column without a cloud API means hosting a much larger open-weights model on dedicated GPU servers, with a context window of at least 32k tokens and enough throughput to answer in seconds rather than minutes.
# hosted, or a self-hosted OpenAI-compatible server (vLLM, llama.cpp) inside the network [llm] provider = "openai" base-url = "https://opencode.ai/zen/v1" model = "deepseek-v4-flash" api-key-env = "OPENCODE_API_KEY" [ai] max-tokens = 8192 # reasoning model: hidden reasoning counts against this # local Ollama — set the window explicitly, never truncate silently again [llm] provider = "ollama" context-window = 32768
Another reason to self-host: prompts contain real source excerpts. EKOS redacts secrets and PII before anything reaches the ledger, but a hosted model still sees your code. Zen's paid models are zero-retention; its free models may use prompts for training.
| Cause | Scenarios | Why |
|---|---|---|
| Premise rejected, not in refusal wording | adv-004 · adv-011 · adv-015 | The model correctly rejects a false premise (“the evidence does not confirm…”, “EKOS is MIT-licensed, free of charge”), but the grader only accepts an explicit refusal. Widen the grader or tighten the prompt — an open decision. |
| Answer never reaches the top of the evidence | code-002 · lin-008 · dep-004 | The function that builds the LLM provider ranks below the provider trait; the cited_evidence convention isn't in the evidence retrieved for that question; Rust import-path objects don't link back to the crate consuming the redaction module. |
| Facts not recorded yet | code-004 · lin-007 · dep-005 | The Rust edition is set once in the workspace Cargo.toml and inherited; struct fields like KirEvidence.confidence aren't emitted as claims; no caller of plan_question reached the evidence, so the model correctly can't answer — scored as ungrounded. |
| Ranking, not answering | arch-009 · hist-007 | Retrieval-only scenarios graded on recall@10. Documents are now many sections, which outrank the crate or file these scenarios expect — also why recall@10 fell while answers improved. |
| Right in substance, wrong keyword | arch-020 · sec-002 · sec-010 | “Objects can't be deleted from the append-only ledger” without the word “tombstone”; “the security section is additive only” where the grader expects “extend”. |
$ export OPENCODE_API_KEY=...
$ ekos eval run --config ekos.zen.toml --dataset ekos-full --save-answers
$ ekos eval history # trend table across every saved run
| Everything behind this deck | Where |
|---|---|
| Document structure + doc links | ekos/docs/rfcs/0144-document-structure-and-links.md |
| LLM endpoints + context window | ekos/docs/rfcs/0145-llm-endpoints-and-context-window.md |
| Write-ups with measurements | devlogs/devlog_183.md (llama3 baseline), 185 (RFC 0144/0145, model comparison), 186 (planner fixes) |