EKOS, documented by the thing it documents

The compile pipeline

EKOS is a compiler for enterprise knowledge, not a database or document store. Five verbs, always run in this order, each one a real compiler stage that reads the previous stage's artifacts and writes its own:

Key invariants, enforced by review, not just convention: the Observation Layer never interprets meaning; the ledger is append-only; the Runtime is read-only; every semantic conclusion traces to Evidence; every compiler pass is deterministic and side-effect-free; every artifact is content-addressable; secrets/PII are never observed or stored.

Connectors

Each connector implements the same Observer trait — pull raw facts, no interpretation. Real and tested unless noted.

Knowledge recovery

A Pentaho step, a SQL SELECT, a VIEW, and a stored procedure are all the same underlying concept — sources flowing through filter/join/aggregate/calculate operations to a sink. ekos recover compiles all of them into one shared Transformation IR (Source/Filter/Join/Aggregate/Calculate/Sink/Unmapped), so legacy ETL logic can be diffed against a newly drafted pipeline without hand-reading XML. Unmapped is deliberate: anything unparseable is still recorded as evidenced fact, never silently dropped.RFC 0027 / 0028 / 0029

Beyond raw SQL DDL, the Python analyzer recognizes real SQLAlchemy declarative models (__tablename__ present) and compiles them into the same Table shape a CREATE TABLE produces — real columns, best-effort types, same-file ForeignKey resolution — plus real class inheritance as Extends edges.RFC 0091 / 0092

An opt-in document semantic memory pass reads local documents through an LLM to extract real Concept objects and their relationships, so the same idea mentioned across different documents becomes one findable, linkable thing — queryable through the exact same MCP tools as everything else, no new surface.RFC 0025 / 0026

A dbt project analyzer extracts real Table objects from a dbt project's own checked-in metadata only — never manifest.json/catalog.json (both are dbt/target/ build artifacts, gitignored in every real project inspected while designing this) and never a live warehouse connection, since dbt itself can point at any database. One Table per models/**/*.sql file (a model exists the moment its file does, regardless of YAML documentation) plus one per declared sources[].tables[] entry; ref()/source() macro calls become real DependsOn edges, with an unresolvable reference (e.g. into a gitignored dbt_packages/) honestly skipped rather than fabricated.RFC 0117

Identity resolution

The same real-world entity observed under different names across systems — Informix cust_mstr, Postgres customers, Databricks gold.dim_customer — can be linked. ekos identity scan scores candidate cross-system matches (column overlap, naming-pattern similarity, type compatibility) and writes them as unconfirmed relationships for human review via ekos_identity_review — never a silent auto-merge.RFC 0029 / 0063

Same-source duplicates (two Table objects both literally named customers) auto-merge only on an exact normalized-name match during resolve/compile; anything fuzzy goes through that same review flow instead — no confidence threshold on the underlying scoring formula reliably separates correct fuzzy merges from incorrect ones, so an irreversible auto-merge was judged unsafe as a default.

Storage & the ledger

Four backends, each a drop-in KnowledgeStore — every command, the MCP server, and docs generate work unchanged regardless of which one a workspace uses.

ekos ledger status --storage (or the shorter top-level ekos status --storage, RFC 0116), ekos ledger migrate, and ekos artifact repack shrink an existing workspace in place (both verify before touching anything and leave backups).

Documentation generation

Renders the compiled ledger straight into Markdown/HTML — zero LLM calls, every claim traceable to real compiled evidence, unless --prose is explicitly added.RFC 0035 / 0037 / 0042 / 0090 / 0094 / 0095

--prose (opt-in) layers an LLM-written overview onto each object page or the findings memo's executive summary, reusing ekos ask's exact grounding+citation pipeline, with a token-cost estimate shown before any call — currently wired for --layout objects and --layout solution-architect only; combined with --layout curated it fails with a clear error rather than silently doing nothing (a real bug found and fixed live during this project's own self-hosted benchmark — see §15).

Architecture reasoning & investigation loop

ekos architecture investigate runs an agentic loop: broad collection, deterministic crate-topology extraction, one batched LLM call classifying each crate's architectural role, a deterministic evaluator scoring completeness and evidence coverage (no LLM), and a targeted second pass for any crate still unclassified — stopping early once a quality threshold is met, always ending with a curated-docs run.RFC 0065 / 0066 / 0067

Because an LLM-classified role is a judgment call, not a deterministic fact, two follow-ons treat it accordingly: ekos architecture diff --since <t> does a real id-set comparison between two points in time (not a fuzzy match), and ekos architecture review lets a human confirm or reject a pending classification — both also exposed over MCP.RFC 0107 / 0108 / 0109

LLM-backed compile-time descriptions

Unlike --prose above (render-time, re-spent every call), [llm-description] runs at commit time and persists real, evidence-grounded ai_overview/ai_usage properties straight into the ledger — queryable through EKL/ekos ask/MCP like any other compiled fact, not just rendered once. An ai_comment_check property flags a real discrepancy between an existing doc comment and what the code actually does (consistent/stale/incomplete) — never silently trusted or overwritten. Opt-in and cost-gated: a real call-count estimate is shown, and confirmation is required before any spend.RFC 0088

Hierarchical rollups

Every other context-saving mechanism in EKOS is retrieval-limiting — fewer raw facts, never a synthesized higher-level one. ekos commit also synthesizes deterministic, zero-LLM Rollup objects — one per directory subtree (crate-level by default) or per project in a multi-project estate — each carrying real member counts, a kind breakdown, and boundary-relationship counts, linked to every member via the same Contains relationship everything else uses. An agent asking about a whole subsystem gets one condensed, evidence-linked object instead of personally synthesizing meaning from dozens of raw facts.RFC 0044

AI agent access (MCP)

The single read-only surface every AI agent — Claude Code among them — is meant to use instead of touching raw enterprise systems directly. ekos mcp serve --workspace <dir> exposes the Runtime over stdio (RFC 0013), with a per-server-session cached read-only ledger handle that never blocks a concurrent writer.RFC 0097

--tcp 127.0.0.1:7331 additionally serves the same protocol over plain NDJSON-over-TCP, so more than one MCP-speaking tool (PyCharm's AI chat, another agent host, a second Claude Code session) can connect to one already-running server instead of each spawning its own process. One OS thread per connection, each with its own independent cache — not shared across connections, so N concurrent clients means N independent ledger opens, not one reused handle. Stdio stays the default and is unaffected when --tcp is omitted. No authentication or TLS — loopback or a trusted network only.RFC 0115

Every read tool (and ekos ekl run from the CLI) appends one line to .ekos/query-log.jsonl; a static heuristic classifies each call cheap/expensive from its own arguments and opportunistically caches an expensive one's result for an identical repeat while the workspace hasn't changed underneath it.RFC 0114

ekos ask "<question>" is the LLM-backed CLI counterpart. It now compiles the question — a rules planner routes it to fact lookups and graph traversals, executes them into a flat set of atomic, source-cited claims, and the model's job shrinks to "explain this evidence, cite each item." --explain prints the plan and the evidence set alongside the answer; --classic (implied by --stream) selects the older retrieve-and-dump-JSON path.RFC 0118 / 0123 / 0124

Retrieval is lexical by default (BM25 + exact-name). The opt-in [embeddings] table in ekos.toml (provider ollama/openai/mock) adds a vector arm: ekos commit embeds every compiled object — its name, kind, and ai_overview prose if present — into a derived, rebuildable <ledger-dir>/vectors/ index, and ekos query find --mode vector|hybrid (or ekos_search {mode}) fuses cosine-nearest hits with the lexical arms via the same Reciprocal Rank Fusion as RFC 0120. So "the thing that sends welcome emails" finds a function named dispatch_signup_notification. Off by default (no table → nothing embedded, byte-identical lexical path); embeddings are cheap and disk-cached so there is no spend prompt; single-node this phase, and a vector search with no index yet degrades to lexical with a visible note.RFC 0125

Retrieval quality is CI-gated. ekos_runtime::retrieval_eval holds a checked-in graded query set (~30 queries across all five query shapes) and a hand-built reference estate; a normal workspace test fails the build if Recall@10 / MRR / nDCG@10 or intent-classification accuracy falls more than 2% below the recorded baseline, and cargo bench --bench retrieval_eval prints the current scoreboard. Every fused retrieve also carries arm_timings — per-arm wall-clock and candidate counts — surfaced on the ekos_search / ekos_retrieve results, written into .ekos/query-log.jsonl, and shown by ekos query find --explain.RFC 0126

Enterprise Knowledge Language (EKL)

A small purpose-built query language over the compiled ledger — ekos ekl "FIND Object WHERE kind = 'Table' AND name CONTAINS 'order' ORDER BY name LIMIT 10" — with point-in-time AS OF <timestamp> queries, COUNT/GROUP BY aggregation, and SEMANTIC 'text' [LIMIT k] — start from a ranked retrieval candidate set instead of a full scan, then filter/project/order it like any other FIND Object (mutually exclusive with FROM / AS OF / COUNT). Every result row carries the evidence it was derived from, the same as any other read path.RFC 0010 / 0096 / 0124

Marketing agent

ekos marketing publish [devlog] turns a devlog_N.md into a human-approved X (Twitter) release announcement: classifies the devlog's importance, drafts a tweet through the same LlmProvider used elsewhere, validates it, asks for Y/N/E approval, and publishes via a real OAuth 1.0a-signed API call — with a posted-history file preventing the same devlog from ever being posted twice. Off by default; requires explicit Twitter credentials and an explicit opt-in flag.RFC 0030

World Engine simulation (experimental, opt-in)

Auxiliary tooling built on the same ledger, kept deliberately separate from the compiler pipeline above: multi-agent scenarios with beliefs and goals, deterministic round-based decision-making with seed-reproducible conflict resolution, a VirtualForum (channels, replies, likes, follows, shares), a durable replayable event log, and real document ingestion into a scenario's starting world. ekos simulate scenario.yaml runs it; ekos replay scenario.yaml reads it back, read-only. Writes to a dedicated per-scenario ledger by default — never the real workspace ledger, since the ledger has no delete/tombstone mechanism and simulated agents/events are fictional.RFC 0047–0055

ClickHouse connector

Two independent pieces. Compiled metadata: ekos build/recover observe a configured ClickHouse database's live schema over its stock HTTP interface and compile every table into a real, searchable, cross-system identity-resolvable Table object. Live NL-to-SQL (ekos clickhouse ask "…"): the one path in EKOS that intentionally crosses the "AI never touches raw systems directly" invariant — an LLM builds a ClickHouse SELECT, it's parsed and hard-rejected unless it's exactly one read-only statement, then run live, redacted, and returned, with every call recorded as an Evidence/Event pair for audit. The matching ekos_clickhouse_query MCP tool is off by default, gated behind an explicit ekos.toml opt-in.RFC 0056

Live, generated evidence: this repo, compiled by itself

Everything below this line is rendered straight from ekos docs generate --layout curated and --layout solution-architect's own output, run against EKOS's real, currently-committed ledger — the same commands and output shape any other project gets. Nothing here is hand-typed.

Architecture summary

"Purpose: not yet computed — no real EKOS source for a project's stated purpose today."
"Architecture style: not yet computed — requires reasoning EKOS doesn't perform yet."
— verbatim from the generated page. An honest placeholder, not a fabricated 100%.

Concentration risk

Real compiled dependency fan-in, not a guess — the top of a 133-entry Risk list.

• ekos_kir::KirId55 dependents
• std::path::Path54 dependents
• ekos_kir::KirObject53 dependents
• std::collections::HashMap48 dependents
• ekos_kir::ObjectKind47 dependents

Top technologies, the same way:

• serde_json36 dependents
• thiserror31 dependents
• tokio27 dependents
• tracing26 dependents
• async-trait24 dependents

Solution-architect layout · real findings

Onboarding guide · excerpt

A real repository-layout table from the actual Cargo.toml graph, not a hand-maintained architecture doc that drifts: