Autonomous verification  ·  re-run

EKOS Full-Stack Test Run

run 20260901T172928Z commit a948bed 2026-09-01 mode partitioned + distributed + Ollama plan EKOS_FULL_TEST_PLAN_v2
Result All 22 acts PASS  ·  F3 / F5 / F6 confirmed fixed  ·  no BLOCKER  ·  3 findings still open

Second full pass over the distributed storage stack (RFC 0111 / 0113), the compiled-knowledge query engine (RFC 0118 / 0119–0126) and the MCP protocol (RFC 0013 / 0115), run against a live [storage.partition] workspace plus a two-worker distributed cluster and a local Ollama. Its purpose was to confirm the three partitioned-store bugs the first run surfaced (F3, F5, F6) are closed, and that nothing regressed. Every command was logged in full; results below come from the logs, not from memory of what was expected.

§1

Verdict matrix

Part A — Distributed storage
RFC 0111 / 0113
0 · env & buildpass
1 · local baselinepass
2 · backend seampass
3 · coordinator + fencingpass
4 · read path + failoverpass
5 · distributed searchpass
6 · benchmarkspass
Part B — Query engine
RFC 0118 / 0119–0126
B1 · retrieval seampass
B2 · RRF + ExactNamepass
B3 · query understandingpass
B4 · QUERY surfacenote
B5 · REASON + citationspass
B6 · surfacepass
B7 · vector armpass
B8 · eval harnesspass
Part C — MCP protocol
RFC 0013 / 0115
C1 · handshake + tools/listpass
C2 · tool round-trip + writespass
C3 · TCP transportpass
C4 · usage log + cachepass
C5 · MCP over the gatewaypass
C6 · multi-turn sessionpass
Safety-critical check (C5). A write-capable operation issued against the read-only distributed gateway fails with an explicit error — cannot write: ledger opened read-only at distributed gateway is read-only — append_evidence must go through ekos compile-worker (Service A), exit 1. No silent no-op, no crash, no state corruption.
§2

The three fixes, re-verified

Committed in 78ffaae after the first run; this pass exercised each in a real end-to-end context.

fixedF3 cross-partition exact-name

PartitionedLedger::retrieve re-labelled every partition's ranked list as Bm25, so a within-partition exact-name promotion only tied at rank 0 with a strong lexical hit elsewhere and lost the KirId tiebreak. A cross-partition ExactName arm over the candidate union was added — matching what DistributedLedger::search_ranked already did.

was  query find "customers" → schemas/ecommerce.sql #1 now  → customers (the table) #1, ecommerce.sql #2

fixedF5 mcp --workspace config

ekos mcp serve --workspace <dir> never loaded <dir>/ekos.toml, so a server against a partitioned workspace failed every call. A new resolve_config_path helper threads the --workspace flag (and EKOS_WORKSPACE) into config resolution.

was  every tool call → "time-bucket weekly vs monthly" now  used flag-only for all of Part B and C — zero failures

fixedF6 ekos status on partitions

ledger::status branched on uses_fact_engine (a facts/manifest.json check) and fell through to the SQLite path for any partitioned store. A partitioned / distributed branch now runs first via open_store.

was  "Ledger not initialised. Run ekos commit first." now  (partitioned, RFC 0111) · 1 299 entries · 30 objects
§3

Part A — Distributed storage & query

0 · Environment & buildcargopass

cargo build --workspace (stock) and -p ekos --features distributed both exit 0. object_store absent from the stock tree, v0.14.1 present with the feature — confirmed by cargo tree, not assumed.

1 · Local baselineRFC 0111 Phase Apass

Workspace: northwind.sql + ecommerce.sql + sample_project/ + sample_docs/, dimension = "entity-kind", time-bucket = "weekly".

11
partitions
27
objects
1 032
relationships
226
evidence
3.3 s
build
14.8 s
commit

Local queries: query object 0.07 s · diff 0.60 s · query find 0.10 s.

2 · Backend seamSegmentBackendpass

Same workspace committed a second time with segment-backend-url = "file://…" and the distributed binary. Identical object / relationship / evidence counts (27 / 1 032 / 226); 11 partition manifests published to the file backend; 165 ledger unit tests pass. Behaviour did not change when the backend was swapped.

3 · Coordinator + Service A fencingRFC 0113 B3pass
worker B
rejected — coordinator returned an error: partition main is already leased, exit 1
worker C
after the 10 s TTL, acquired lease token 2, ran the full pipeline, Commit complete, released shard 'main', exit 0
recovery gap
~12.0 s (TTL + agent scheduling latency; not a pure coordinator-reclaim number)
coordinator
11 partitions registered, each with a file:// location; shard main generation 1 315
4 · Read path — workers + gatewayRFC 0113 B4pass

Two query workers + a [storage.distributed] gateway. Reads via the gateway match local: query object 0.08 s, query find 0.06 s.

Failover: kill -9 query-worker 1 → gateway logs WARN query worker unreachable — failing over → the query completes via worker 2, exit 0, same result. Clean, no error surfaced to the client.

F6 over the gateway: ekos statusLedger: distributed cluster @ 127.0.0.1:7801 (distributed cluster, RFC 0113), 1 315 entries.

5 · Distributed searchRFC 0113 B5pass

For the shared term customer, the local partitioned FactLedger and the distributed gateway's cross-shard RRF merge return a byte-identical top-7. No shard-local-IDF divergence at this corpus size.

6 · BenchmarksCriterionpass

All 22 Criterion bench functions run, exit 0. Coverage audit (grep): no bench references PartitionedLedger, DistributedLedger, SegmentBackend or the coordinator — the distributed path still has zero micro-benchmark coverage, verified rather than assumed.

§4

Part B — Compiled-knowledge query engine

B1 · Retrieval seamRFC 0119pass

9/9 ledger + 7/7 runtime retrieval unit tests. Default arms_run = {bm25: true, vector: false} — the Phase-0 contract. The MCP server was reached with --workspace only (no --config) for this and every subsequent Part B/C step — F5 in daily use.

B2 · RRF fusion + ExactNameRFC 0120pass

F3 confirmed: query find "customers" on the partitioned store now ranks the customers table #1 and schemas/ecommerce.sql #2. The --explain plan shows the query routed to a Fact lookup at routing confidence 1.00.

B3 · Query understandingRFC 0121pass

ekos_retrieve "what does the customers table connect to"query_type: Structural, resolved_entities: [(customers, 1.0)]. (Last run's F7 miss does not recur — the merged table is named exactly customers, so the mention resolves cleanly.)

B4 · QUERY surfaceRFC 0122note

ekos_query "what depends on the customers table" returns the real dependents — orders, CustomerCustomerDemo, payments, "Order Details", order_items — each source-traceable; ekos_dependents on the id agrees on the subset it scopes to.

New observation. The same ekos_query response also carried three fact claims about customers itself (name, kind, columns) from the Compose plan — noise for a "what depends on X" question. The dependents are all correct; the extra facts look like planner scoping over-reach.
B5 · REASON + citation checkRFC 0123pass

Plan: Compose[ Search + Graph Neighborhood from #customers (1 hop) ] → a two-item evidence set. The answer (Ollama llama3) names exactly CustomerCustomerDemo and ordersevery answer table is in the evidence set, none fabricated.

The --classic path, on the same question and model, produced an empty cited_evidence block; EKOS printed treat this answer as ungrounded even though it parsed cleanly. The grounding safeguard held where the classic path did not.
B6 · SurfaceRFC 0124pass
  • FIND Object SEMANTIC 'customer' LIMIT 5 → 5 rows (lexical BM25 fallback, no vector index, no error, no vector: true).
  • ekos_search { limit: 3 } → exactly 3 matches.
  • query find --explain prints the compiled plan.
B7 · Vector armRFC 0125pass

On a non-partitioned workspace with [embeddings] enabled=true, provider="ollama", model="nomic-embed-text": ekos commit30 embedded, 0 errors, 768-dim; vectors/{meta.json, ids.bin, vectors.f32, tombstones.bin} created.

semantic querylexicalvector — top hit
money owed and received0 hitspayments  — shares no query word
the people we sell to0 hitspayments, schemas/ecommerce.sql
moving goods to the buyer0 hitsorder_items, schemas/ecommerce.sql
B8 · Eval harnessRFC 0126pass

cargo bench --bench retrieval_evalbaseline: OK (tol 0.02).

0.841
Recall@10
0.739
MRR
0.745
nDCG@10
0.862
intent acc

by type — Lookup 1.000 · Lexical 0.625 · Conceptual 0.938 · Structural 0.250 · Aggregate 1.000

§5

Part C — MCP protocol

C1 · Handshake + tools/listRFC 0013pass

Protocol 2025-03-26; 17 tools — all 8 legacy tools + ekos_query + ekos_retrieve; ekos_search schema carries limit and mode.

C2 · Tool round-trip + write toolsRFC 0013pass
  • 12 read tools, one call each → all isError: false with real data.
  • ekos_identity_review { decision: "rejected" }{ status: "recorded" }; a re-scan reports written: 0, skipped: 39 — the decision persisted.
  • The two write-capable tools are the project's own documented exception to the read-only surface — recorded as design, not flagged.
C3 · TCP transport + concurrencyRFC 0115pass

Handshake over nc. Ten ekos_status calls: 3.020 s alone vs 2.983 s with a second client holding a stalled connection — within noise. 10/10 responses both runs; per-message (not per-connection) locking holds.

Server logs no authentication, trusted network only — RFC 0115's stated v1 scope; only 127.0.0.1 was ever bound.

C4 · Usage log + cacheRFC 0114pass

One JSONL line per call (4 / 4). Within a live server session, an identical repeat of an Expensive call is served from cache:

toolclasscall 1call 2
ekos_impact (max_hops 6)expensive44 ms0 ms — cache_hit
ekos_architecture_evaluateexpensive227 ms0 ms — cache_hit
C5 · MCP over the distributed gatewayA ∩ Cpass

Reads from a partially-degraded cluster (query-worker 1 still dead): ekos_status (distributed cluster @ 127.0.0.1:7801, 1 315 entries) and ekos_search → all isError: false via automatic failover.

Negative assertion (the single most safety-relevant check). ekos identity scan against the gateway → Error: cannot write: ledger opened read-only at distributed gateway is read-only — append_evidence must go through ekos compile-worker (Service A), exit 1. Explicit, clean, no corruption.
C6 · Multi-turn sessionmcp_session.rs patternpass

Turn 1: ekos_status + ekos_ekl … SEMANTIC 'payments' → the payments id. Turn 2: that id through ekos_query, ekos_retrieve, ekos_state, ekos_dependents, ekos_neighborhood, ekos_diff — all 6 isError: false. The KirId round-tripped through every tool with no malformed-id rejection (the exact class the project's own session test guards against).

§6

Findings status

#statefinding
F3fixed Cross-partition ExactName promotion lost in PartitionedLedger::retrieve. Fixed & re-verified: query find "customers" → the table ranks #1. New test retrieve_promotes_an_exact_name_match_across_partitions.
F5fixed ekos mcp serve --workspace ignored the workspace's ekos.toml. Fixed & used flag-only throughout Part B/C. New test config_path_resolution_precedence.
F6fixed CLI ekos status / ekos ledger status claimed "not initialised" on partitioned workspaces. Fixed & re-verified local and over the gateway. New test in ledger.rs.
F2watch ekos diff --from <very-old-ts> returned an empty diff last run. Did not reproduce this run (Versions written: 1106). Needs a deterministic repro before it can be closed.
F4open arm_timings (RFC 0126) is [] on partitioned + distributed stores — only FactLedger::retrieve emits it. Documented design; a telemetry blind spot on exactly the multi-partition deployments.
F7n/a Inflected entity-mention resolution miss. Did not reproduce — the merged table is named exactly customers, so the mention resolves at confidence 1.0. The original case was customer (singular) vs Customers.
B4new ekos_query "what depends on X" also returns fact claims about X itself, from the Compose plan. Dependents are correct; the extra facts are planner scoping over-reach.
§7

Environment & limitations

workspace
Fixture set (northwind + ecommerce SQL + sample_project + sample_docs), 27 objects — chosen for tractability across 22 acts, not scale.
object store
file:// only; no real S3 / MinIO in this run (prior run devlog_144 covers real MinIO).
LLM
Local Ollama — llama3 for ekos ask, nomic-embed-text for embeddings. Structural checks verified, not answer quality.
criterion
Reduced sampling (warm-up 1 s, measurement 2 s, sample 10) to bound wall-clock. The point — benches run, zero distributed coverage — is unaffected.
fencing gap
~12 s includes inter-command scheduling latency, not a pure coordinator-reclaim number. TTL (10 s) and the clean token-2 takeover are the real signals.
merges
The DefaultResolver (RFC 0007) merged northwind.Customers with ecommerce.customers (and orders / products / categories) on exact normalized-name match — expected, and why the canonical name is lowercase customers.
artifacts
test-runs/run-20260901T172928Z/ — logs/ (≈40 files), metrics/*.json, mcp-*.ndjson, client-ekos.toml, 6 workspace copies