Distributed Storage
Under Fire

Four services, one KnowledgeStore trait

EKOS is a compiler for enterprise knowledge: it observes source systems, compiles the observations through deterministic passes into a Canonical Knowledge Model, and stores the result in an append-only, evidence-carrying ledger. Distributed mode splits that ledger across many independent fact-segment partitions in object storage and puts a metadata coordinator, a compile worker, query workers, and a read-only gateway around it — every existing reader keeps working because the gateway implements the same trait the local ledger does.

Component map

Seven acts, executed sequentially, everything logged

Each run works through an executable test plan act by act. Every command's stdout, stderr, and exit code is captured to disk; metrics land in metrics/*.json; a REPORT.md is generated from the logs, not from memory. A step failure is recorded as-is — no softening, no "fixing" a result before recording it.

The two runs

What got harder, and what broke because of it

Run 1's fixes made the object-store path reachable through the CLI at all. Running that path against a real S3 endpoint, a 95-partition workspace, an 8-second lease, and concurrent RPC then exposed a second layer of defects that unit tests had never touched.

Eight defects, nine fixes, one commit

Every fix ships with a regression test. Branch fix/distributed-storage-issues, commit 2896481 — 23 files, +885 / −92. cargo test --workspace, clippy -D warnings, and fmt --check all pass.

Every feature, how it was exercised, in which run

"PASS (new)" marks behaviour introduced by this campaign's fixes. "PARTIAL" marks a feature exercised but not fully — noted, not hidden.

What this campaign did not prove, and what's still on the list

Commands, config, artifacts

The distributed CLI surface exercised

# write side — Service A, under a coordinator
ekos coordinator serve --listen 127.0.0.1:7801 --state coord.json --ttl-seconds 8
ekos compile-worker run --coordinator 127.0.0.1:7801 --shard main --workspace WS --force
ekos coordinator status --coordinator 127.0.0.1:7801

# read side — Services B & C
ekos query-worker serve --coordinator 127.0.0.1:7801 --listen 127.0.0.1:7811 --cache CACHE
EKOS_CONFIG=client-ekos.toml ekos query object <id>
EKOS_CONFIG=client-ekos.toml ekos query find "Subscription"
EKOS_CONFIG=client-ekos.toml ekos diff --from 2020-01-01T00:00:00Z --to <now>

[storage.partition]
dimension = "entity-kind"
time-bucket = "weekly"
segment-backend-url = "s3://ekos-partitions/run2c"

[llm]
provider = "openai"
model = "gpt-4o-mini"

[document-semantics]
enabled = true

[storage.distributed]
coordinator = "127.0.0.1:7801"
query-workers = [
  "127.0.0.1:7811",
  "127.0.0.1:7812",
]

# env for the object store (MinIO)
AWS_ENDPOINT=http://127.0.0.1:9100
AWS_ACCESS_KEY_ID=…  AWS_SECRET_ACCESS_KEY=…
AWS_ALLOW_HTTP=true

Test totals — post-fix

Artifact tree

test-runs/
├── run-20260831T212115Z/        # Run 1 — stock e8e1ca3, Pentaho, file://
│   ├── REPORT.md
│   ├── logs/{00-environment … 61-search-rs-test}.log
│   └── metrics/{baseline,fencing,latency_comparison,search_ordering,…}.json
└── run-20260831T222159Z/        # Run 2 — + fix diff, Plausible/Elixir, MinIO
    ├── REPORT.md
    ├── logs/00b-bugfix-diff.patch    # the 18-file fix
    ├── logs/{10-baseline … 70-criterion-bench}.log
    └── metrics/{baseline,fencing,latency_comparison,search_ordering,
                 benchmark_summary,criterion_distributed_coverage}.{json,txt}

Both runs carry a generated REPORT.md built from the logs. Narrative and rationale: devlogs/devlog_144.md.