Second full pass over the distributed storage stack (RFC 0111 / 0113), the
compiled-knowledge query engine (RFC 0118 / 0119–0126) and the MCP protocol
(RFC 0013 / 0115), run against a live [storage.partition] workspace plus a
two-worker distributed cluster and a local Ollama. Its purpose was to confirm the three
partitioned-store bugs the first run surfaced (F3, F5, F6) are closed, and that nothing
regressed. Every command was logged in full; results below come from the logs, not from
memory of what was expected.
cannot write: ledger opened read-only at distributed gateway is read-only —
append_evidence must go through ekos compile-worker (Service A), exit 1.
No silent no-op, no crash, no state corruption.
Committed in 78ffaae after the first run; this pass exercised each in a real end-to-end context.
PartitionedLedger::retrieve re-labelled every partition's ranked list as
Bm25, so a within-partition exact-name promotion only tied at rank 0
with a strong lexical hit elsewhere and lost the KirId tiebreak. A
cross-partition ExactName arm over the candidate union was added — matching
what DistributedLedger::search_ranked already did.
query find "customers" → schemas/ecommerce.sql #1
now → customers (the table) #1, ecommerce.sql #2
--workspace config
ekos mcp serve --workspace <dir> never loaded
<dir>/ekos.toml, so a server against a partitioned workspace failed
every call. A new resolve_config_path helper threads the
--workspace flag (and EKOS_WORKSPACE) into config resolution.
ekos status on partitions
ledger::status branched on uses_fact_engine (a
facts/manifest.json check) and fell through to the SQLite path for any
partitioned store. A partitioned / distributed branch now runs first via
open_store.
ekos commit first."
now (partitioned, RFC 0111) · 1 299 entries · 30 objects
cargo build --workspace (stock) and -p ekos --features distributed both exit 0.
object_store absent from the stock tree, v0.14.1 present with the feature — confirmed by cargo tree, not assumed.
Workspace: northwind.sql + ecommerce.sql + sample_project/ + sample_docs/, dimension = "entity-kind", time-bucket = "weekly".
Local queries: query object 0.07 s · diff 0.60 s · query find 0.10 s.
Same workspace committed a second time with segment-backend-url = "file://…" and the distributed binary.
Identical object / relationship / evidence counts (27 / 1 032 / 226); 11 partition manifests published to the file backend; 165 ledger unit tests pass. Behaviour did not change when the backend was swapped.
coordinator returned an error: partition main is already leased, exit 1Commit complete, released shard 'main', exit 0file:// location; shard main generation 1 315Two query workers + a [storage.distributed] gateway. Reads via the gateway match local:
query object 0.08 s, query find 0.06 s.
Failover: kill -9 query-worker 1 → gateway logs
WARN query worker unreachable — failing over → the query completes via worker 2, exit 0, same result. Clean, no error surfaced to the client.
F6 over the gateway: ekos status →
Ledger: distributed cluster @ 127.0.0.1:7801 (distributed cluster, RFC 0113), 1 315 entries.
For the shared term customer, the local partitioned FactLedger and the distributed
gateway's cross-shard RRF merge return a byte-identical top-7. No shard-local-IDF divergence at this corpus size.
All 22 Criterion bench functions run, exit 0. Coverage audit (grep): no bench references
PartitionedLedger, DistributedLedger, SegmentBackend or the coordinator — the
distributed path still has zero micro-benchmark coverage, verified rather than assumed.
9/9 ledger + 7/7 runtime retrieval unit tests. Default arms_run = {bm25: true, vector: false} — the Phase-0 contract.
The MCP server was reached with --workspace only (no --config) for this and every subsequent Part B/C step — F5 in daily use.
F3 confirmed: query find "customers" on the partitioned store now ranks the
customers table #1 and schemas/ecommerce.sql #2. The --explain
plan shows the query routed to a Fact lookup at routing confidence 1.00.
ekos_retrieve "what does the customers table connect to" → query_type: Structural,
resolved_entities: [(customers, 1.0)]. (Last run's F7 miss does not recur — the merged table is
named exactly customers, so the mention resolves cleanly.)
ekos_query "what depends on the customers table" returns the real dependents —
orders, CustomerCustomerDemo, payments, "Order Details",
order_items — each source-traceable; ekos_dependents on the id agrees on the subset it scopes to.
ekos_query response also carried three
fact claims about customers itself (name, kind,
columns) from the Compose plan — noise for a "what depends on X"
question. The dependents are all correct; the extra facts look like planner scoping over-reach.
Plan: Compose[ Search + Graph Neighborhood from #customers (1 hop) ] → a two-item evidence set.
The answer (Ollama llama3) names exactly CustomerCustomerDemo and orders —
every answer table is in the evidence set, none fabricated.
--classic path, on the same question and model, produced an empty
cited_evidence block; EKOS printed
treat this answer as ungrounded even though it parsed cleanly. The grounding
safeguard held where the classic path did not.
FIND Object SEMANTIC 'customer' LIMIT 5 → 5 rows (lexical BM25 fallback, no vector index, no error, no vector: true).ekos_search { limit: 3 } → exactly 3 matches.query find --explain prints the compiled plan.On a non-partitioned workspace with [embeddings] enabled=true, provider="ollama", model="nomic-embed-text":
ekos commit → 30 embedded, 0 errors, 768-dim;
vectors/{meta.json, ids.bin, vectors.f32, tombstones.bin} created.
| semantic query | lexical | vector — top hit |
|---|---|---|
| money owed and received | 0 hits | payments — shares no query word |
| the people we sell to | 0 hits | payments, schemas/ecommerce.sql |
| moving goods to the buyer | 0 hits | order_items, schemas/ecommerce.sql |
cargo bench --bench retrieval_eval → baseline: OK (tol 0.02).
by type — Lookup 1.000 · Lexical 0.625 · Conceptual 0.938 · Structural 0.250 · Aggregate 1.000
Protocol 2025-03-26; 17 tools — all 8 legacy tools + ekos_query + ekos_retrieve;
ekos_search schema carries limit and mode.
isError: false with real data.ekos_identity_review { decision: "rejected" } → { status: "recorded" }; a re-scan reports written: 0, skipped: 39 — the decision persisted.Handshake over nc. Ten ekos_status calls: 3.020 s alone vs
2.983 s with a second client holding a stalled connection — within noise. 10/10 responses both runs;
per-message (not per-connection) locking holds.
Server logs no authentication, trusted network only — RFC 0115's stated v1 scope; only 127.0.0.1 was ever bound.
One JSONL line per call (4 / 4). Within a live server session, an identical repeat of an Expensive call is served from cache:
| tool | class | call 1 | call 2 |
|---|---|---|---|
| ekos_impact (max_hops 6) | expensive | 44 ms | 0 ms — cache_hit |
| ekos_architecture_evaluate | expensive | 227 ms | 0 ms — cache_hit |
Reads from a partially-degraded cluster (query-worker 1 still dead): ekos_status
(distributed cluster @ 127.0.0.1:7801, 1 315 entries) and ekos_search → all
isError: false via automatic failover.
ekos identity scan against the gateway →
Error: cannot write: ledger opened read-only at distributed gateway is read-only —
append_evidence must go through ekos compile-worker (Service A), exit 1.
Explicit, clean, no corruption.
Turn 1: ekos_status + ekos_ekl … SEMANTIC 'payments' → the payments id.
Turn 2: that id through ekos_query, ekos_retrieve, ekos_state,
ekos_dependents, ekos_neighborhood, ekos_diff — all 6
isError: false. The KirId round-tripped through every tool with no
malformed-id rejection (the exact class the project's own session test guards against).
| # | state | finding |
|---|---|---|
| F3 | fixed | Cross-partition ExactName promotion lost in PartitionedLedger::retrieve. Fixed & re-verified: query find "customers" → the table ranks #1. New test retrieve_promotes_an_exact_name_match_across_partitions. |
| F5 | fixed | ekos mcp serve --workspace ignored the workspace's ekos.toml. Fixed & used flag-only throughout Part B/C. New test config_path_resolution_precedence. |
| F6 | fixed | CLI ekos status / ekos ledger status claimed "not initialised" on partitioned workspaces. Fixed & re-verified local and over the gateway. New test in ledger.rs. |
| F2 | watch | ekos diff --from <very-old-ts> returned an empty diff last run. Did not reproduce this run (Versions written: 1106). Needs a deterministic repro before it can be closed. |
| F4 | open | arm_timings (RFC 0126) is [] on partitioned + distributed stores — only FactLedger::retrieve emits it. Documented design; a telemetry blind spot on exactly the multi-partition deployments. |
| F7 | n/a | Inflected entity-mention resolution miss. Did not reproduce — the merged table is named exactly customers, so the mention resolves at confidence 1.0. The original case was customer (singular) vs Customers. |
| B4 | new | ekos_query "what depends on X" also returns fact claims about X itself, from the Compose plan. Dependents are correct; the extra facts are planner scoping over-reach. |
northwind + ecommerce SQL + sample_project + sample_docs), 27 objects — chosen for tractability across 22 acts, not scale.file:// only; no real S3 / MinIO in this run (prior run devlog_144 covers real MinIO).llama3 for ekos ask, nomic-embed-text for embeddings. Structural checks verified, not answer quality.DefaultResolver (RFC 0007) merged northwind.Customers with ecommerce.customers (and orders / products / categories) on exact normalized-name match — expected, and why the canonical name is lowercase customers.