The complete capability reference · kept current after every devlog

EKOS, documented by the thing it documents

Every capability below is real and shipped — grounded in the same RFCs and source that implement it, cross-checked against README.md and CLAUDE.md. Sections marked live, generated are also rendered straight from ekos docs generate's own output against this repository's real, currently-committed ledger — nothing hand-typed there. Where EKOS doesn't have a real answer to something yet, this page says so, the same way EKOS's own generated docs do, instead of guessing.

00

Install & first run

One command, no Rust toolchain. The installer resolves the latest release, verifies the download against that release's SHA256SUMS before unpacking it, installs a single binary into the EKOS home (~/.ekos/bin; an earlier install in ~/.local/bin is upgraded where it is), and never uses sudo.

curl -fsSL https://raw.githubusercontent.com/alexeyban/EKOS/main/install.sh | sh

Prebuilt binaries

Linux x86_64 (glibc and fully static musl), Linux aarch64, macOS on Apple Silicon and Intel, Windows x86_64. Every release is also a plain archive on the releases page for anyone who would rather not pipe a script into a shell: download it, check it against SHA256SUMS, unpack, put the binary on PATH — the README has the exact commands per platform, including the Gatekeeper xattr step macOS needs for an unsigned binary downloaded through a browser. Built from a toolchain pinned by rust-toolchain.toml, so a release rebuilt months later is the same compiler. Nothing in the dependency graph resists static linking: SQLite is bundled from source, TLS is rustls, and the one native tool (tesseract, for OCR) is shelled out to and degrades gracefully when absent rather than being linked.RFC 0153

Setup wizard and the console launcher

The full package — the ekos command, the web console and a pinned standalone Python to run it — installs on Linux, Windows 11 and macOS with nothing else needed. install.sh / install.ps1, or setup.sh / setup.command / setup.cmd in the unpacked archive, open ekos setup: a wizard in the browser, served by the binary on loopback with a one-time token, that asks where EKOS goes, local or team mode (OIDC issuer checked, client secret to the OS keychain), the port and MCP limits, a first project, which agents to register it with (Claude Code, Claude Desktop, VS Code — files backed up, other entries kept, a commented file left alone), and whether to start at login (systemd user unit, launchd agent, logon task; no administrator rights). It lists every step before doing any. ekos setup --answers answers.toml does the same without a browser. ekos console start|stop|restart|status|open|logs runs the console under a supervisor that restarts it when asked (the Settings page's button) or after a crash; open signs in locally with a one-time code. ekos uninstall undoes exactly what setup recorded and keeps the projects unless --purge --yes; ekos uninstall --ui, or the console's Settings → Uninstall EKOS in local mode, shows the same on a page (each step, the projects and their size, a typed confirmation) served by the binary, which keeps answering while it removes the console (RFC 0188). A project's top bar has Init & compile: init, build, recover, resolve (conflicts reported, not stopping), compile, commit and coverage as one run, each stage allowed up to run_stage_timeout_seconds (6 hours by default, on the Settings page) (RFC 0189). Released as v1.1.0 (2026-10-14): full packages for Linux x86_64 and aarch64, macOS on Apple silicon and Windows; Intel Macs get the command only. install.sh replaces a running ekos safely. The release workflow assembles the full package for five targets on Linux from each target's binary.RFC 0185

Offline install package

For a machine without internet access, or a commit that is not released yet: scripts/package-release.sh builds dist/ekos-<version>-release.tar.gz on a Linux x86_64 machine from the committed tree only (uncommitted changes are left out). It holds the glibc build (glibc 2.34 or newer) and the fully static musl build (made in the pinned Rust image, so the host needs no musl toolchain), packed exactly as a release packs them, one SHA256SUMS, install.sh and an INSTALL.md covering installing by hand, the first run, agents, upgrading and removing. The script installs the package into a throwaway home before it finishes. On the target: EKOS_VERSION=v<version> EKOS_BASE_URL="file://$PWD" sh install.sh from the unpacked directory, which verifies and falls back to the static build just as an online install does. A team server without internet access loads the console image with docker save / docker load (docs/deployment.md).RFC 0179

ekos init --detect

Writes an ekos.toml that reflects what is actually in the repository, instead of the inert seven-line default: the SQL dialect its schema is written in (scored from real markers across sampled files — a tie yields generic rather than a coin flip), exclusions for every third-party or generated directory found, and an inventory of the source kinds present. Every generated line carries the evidence behind it as a comment. Refuses to overwrite an existing config without --force; --dry-run previews and writes nothing.RFC 0152

Projects outside the sources

ekos project add shop --source <dir> creates a project in the EKOS home (~/.ekos, EKOS_HOME or --home): its ekos.toml and all compiled data live under data/<tenant>/<project>/, and the sources are only read. --project shop then runs any command against it from any directory, including ekos mcp serve. Each project has its own ledger, and --tenant separates tenants: two tenants can each have a project of the same name and share nothing. ekos project remove moves the data to a trash directory; deleting needs --purge --yes. A workspace used in place, with .ekos/ in the repository, works as before. Users and roles are in the web console (RFC 0180).RFC 0178

Several sources in one project

A project can be several repositories and directories, each with a name: ekos project add shop --source api=/repos/api --source handbook=/srv/handbook. They compile into one ledger, so identity and links span them, and a file of the same name in two sources stays two objects. Every object records its source (and sources when found in more than one), which EKL can filter on. ekos source add|list|remove edit the project config and keep its comments; ekos build --source <name> re-observes one source. A source declared git without a .git is reported, where it used to yield no commits silently. Confluence and GitHub are still configured once per project, and a project compiled without source names cannot gain them later.RFC 0178

Container image

One image holds the ekos binary, the console API and the built console UI, and serves both on port 8000. docker compose -f deploy/docker-compose.yml up -d --build starts it with one volume, the EKOS home, and your source directories mounted read-only at /sources. In the console, a project is created from a source directory: its config and compiled data go to the volume and nothing is written into the sources. A project may only read directories under /sources, at creation and when its config is edited. The console's own database and run logs are in the same volume, so a restart or an upgrade keeps everything. The container runs as a non-root user and listens on loopback over plain HTTP; a reverse proxy supplies TLS. Tagged releases publish ghcr.io/alexeyban/ekos.RFC 0179

Tenants, users and roles

The web console decides who may see and do what. Each tenant has its own projects and members; a project or tenant you have no role in answers 404, as if it did not exist. Four roles are granted per tenant or per project in the console's Admin page: viewer reads, reviewer also decides business semantics, conflicts and the source-model review and signs it off, operator also runs commands and edits the project config, admin manages the tenant's projects, members, groups and tokens. Platform owners, named in EKOS_CONSOLE_OWNERS, create tenants and set which host directories each may read. Sign-in is OIDC; a role can be granted to an e-mail address before that person first signs in, and a group can follow a value of the provider's group claim. Agents use API tokens, shown once and stored only as a hash; a token can view or operate and can never review or administer. Each tenant has an audit log of every request that changes something and every refusal. EKOS_CONSOLE_PRODUCTION=true makes the console refuse to start on development credentials.RFC 0180

Access inside a project

A project is one ledger over several sources, so a limit inside a project is applied where the ledger is read. A read scope is a list of allow and deny rules by source, by path pattern inside a source (secrets/**) or by object. Deny always wins, an object found in two sources needs both, and an object the compiler could not attribute to a source is hidden from a limited reader. A relationship shows only when both its ends do, and counts cover only what is visible. In the console a viewer can be limited to part of one project or have part of it hidden; they get the graph, search, object pages and impact, filtered, and the pages that show more than the ledger (dashboard, run logs, configuration, semantics, conflicts) are closed to them. Operators and admins cannot be limited. From the command line, --read-scope scope.json works with ekos mcp serve, ekos ekl and ekos query; under a restricting scope the MCP server offers only the tools that read solely through the ledger. A rule hides objects; it does not rewrite the text of a visible document that mentions a hidden one.RFC 0181

MCP for agents, through the console

An agent connects to a project at /mcp/<tenant>/<project> on the console with an API token, over MCP Streamable HTTP. The console sits in front of the project's own server: it lists only the tools the token's role allows (the two review tools need a signed-in reviewer, so no token sees them), attaches the token's read scope to every call and drops any scope the client sends itself, answers 404 for a project the token has no role in, and records every call in the tenant's audit log without its arguments. A project's server starts on first use, stops after fifteen idle minutes and at most eight run at once, so a console with many projects does not hold all of them in memory. Not built: OAuth for MCP clients and rate limits.RFC 0182

Backup and restore

ekos home backup writes the whole EKOS home as one zstd-compressed tar: every project's config and compiled data, and the console's database and run logs. --tenant limits it to one tenant's projects. A project is copied under its ledger's write lock, so a pipeline stage that is writing makes the backup stop with that project's name and leave no file, while readers carry on; the console database is copied as a consistent snapshot while in use. ekos home restore replaces nothing that exists and changes nothing if anything would be replaced; with --force each existing project is first moved to the home's trash. An archive with a path that would leave the home, or with a link in it, is refused. The sources are not in a backup, because they are not in the home. With --to s3://…, gs://… or https://<account>.blob.core.windows.net/… the archive goes to object storage with the machine's cloud identity, and restore --from takes one backup or the newest under a location (the container image has this; RFC 0232). Not built: encryption beyond the bucket's.RFC 0183

Azure, Google Cloud and AWS

Terraform in deploy/terraform deploys the image to any of the three clouds as a VM (Docker, a data disk, Caddy for HTTPS), a serverless container (Container Apps, Cloud Run, ECS Fargate over an NFS share) or Kubernetes (AKS, GKE Autopilot, EKS Auto Mode with the deploy/helm/ekos chart). Secrets live in Key Vault, Secret Manager or Secrets Manager and reach the console through the platform or as mounted files; one identity may read them and write backups, nothing else; with OIDC and an owner the console starts in production mode. A git sync beside the console keeps /sources at each repository's branch tip, with the token only in a request header. The console backs its home up every night to the deployment's bucket, and an instance lease keeps one console writing a shared home while a platform overlaps revisions. A console that starts over an empty home restores the newest backup there first, and refuses to start if it cannot read the location (an empty start would make the next backup the newest). With a domain, GKE gets a Google-managed certificate on a reserved address and EKS an Auto Mode ALB with the ACM certificate; AKS takes the operator's ingress. exposure = "private" keeps the console's address inside the network (internal Container Apps environment, Cloud Run internal ingress, internal load balancers). Each module is planned in CI with mocked providers (terraform test). Applied on Azure as a VM and on Container Apps, through Azure Pipelines (a revision rollout keeps one console: a restarted old one waits while the new one holds the lease); Google Cloud, AWS and AKS not yet.RFC 0232

Chat with a project

A project's Chat tab answers questions with a configured model — local Ollama, any OpenAI-compatible server, or Anthropic, called directly over HTTP — that uses the project's EKOS tools until it can answer, each call shown with its arguments and result. The model is offered only tools that read and that the person may call through the MCP gateway's own check, with their read scope attached, so a restricted reader's chat cannot surface what they cannot see; review tools are never offered to it (a test scans the chat code), and every tool call is audited. A hosted model is paid per call and answers only in a tenant whose metered calls an owner turned on; a model on this machine always may. The key is named by a variable and read from the environment or the OS keychain. Conversations are kept per person. ekos setup has a Chat step that lists the installed Ollama models and checks a model through the provider's free model list, never a paid call.RFC 0187

Every command and setting from the console

The binary describes itself: ekos commands --json lists every command with its arguments (flags, values, server paths, choices, defaults, help) and a class from one reviewed table — read, write, review, admin, external, local — and ekos config schema --json prints the JSON schema of ekos.toml from the Rust types, with each setting's documentation. A project's Commands tab builds a form for any command, shows the command line before it runs, and runs it as the project's job: the console builds the argv itself (only listed arguments, positionals after --, server paths kept inside the project and the tenant's roots, the reviewer of a review set to the signed-in person) and never a shell. Rights follow the class; destructive commands and those reaching live systems need an admin and the command typed; servers and the installer are not run from the web; a test fails when a command has no class. The config page's Settings form edits every section field by field and applies the changes to the file's text with its comments kept, before the usual validation and save.RFC 0186

Console settings, local mode, keychain

A platform owner changes the console's own settings on its Settings page — sign-in, owners, MCP limits, ports — saved in <home>/control/console.toml. A variable in the environment wins over the file and is shown read-only, so a container configured by its environment is unchanged; changes apply at the next start, with a restart button when a launcher or a restart policy starts the console again. Secrets typed there, and model keys stored under a variable name, go to the OS keychain (Windows Credential Manager, macOS Keychain, Secret Service) and are never shown again; a stored key reaches only the commands of a tenant granted that name while its metered calls are on. Local mode is for one person on their own machine: no identity provider, loopback only with a check on the Host header against DNS rebinding, and sign-in by a one-time code written into the home, so only the install's owner can sign in.RFC 0184

Model credentials per tenant

A project's ekos.toml names the environment variable holding its model key, never the key. In the console a platform owner lists, for each tenant, the variable names its projects may use and switches metered calls on. Every command the console runs for that tenant, its pipeline stages and its MCP server included, then receives those variables and no other tenant's, and none of the console's own settings. With the switch off it receives none, so a hosted provider fails for want of a key while a local model server keeps working. The step-by-step guide for running EKOS for a team is docs/deployment.md.RFC 0183

ekos coverage

Joins inputs present in the tree against objects compiled into the model, per source kind. A kind with files and zero objects is a named finding with its likely cause — not silence. Runs automatically as a one-line tail on ekos compile (and prints nothing at all when everything is healthy), as a doctor check, and as ekos coverage --strict for CI, which exits non-zero. --json emits the same verdict for machines.RFC 0152

Why this exists. Five separate real codebases compiled to nothing while every stage reported success: a wrong SQL dialect made a 103-table schema vanish behind one buried SQL001; an [observe] paths list of subdirectories made the git observer find zero commits with no error; a missing .venv exclusion made 50.9% of this project's own compiled objects describe numpy and scipy internals. The information needed to catch all of it was already in the workspace — nothing joined it. Now something does.

01

The compile pipeline

EKOS is a compiler for enterprise knowledge, not a database or document store. Five verbs, always run in this order, each one a real compiler stage that reads the previous stage's artifacts and writes its own:

ekos build
→
ekos recover
→
ekos resolve
→
ekos compile
→
ekos commit

build

Runs every configured connector's Observer, writes raw, content-addressable ObservationArtifacts. No interpretation — facts only. Secrets/PII are redacted at this boundary, unconditionally (RFC 0043). So is every later re-read of a workspace file for a model or the ledger (symbol source for [llm-description], crate doc comments for architecture reasoning, view definitions for migration lowering), which also refuses an excluded path, and each git blame commit subject recorded as rationale.

recover

Knowledge-recovery passes turn raw artifacts into typed KIR (Knowledge Intermediate Representation) — one analyzer per source kind (SQL, Pentaho, Python, Rust, git, GitHub, Confluence, documents, ClickHouse, crypto, crate/CI topology).

resolve

Identity resolution merges same-source-kind duplicates when the match is an exact normalized name; anything fuzzy becomes a reviewable unconfirmed relationship instead of a silent, irreversible merge.

compile

KIR + resolved identities → the Canonical Knowledge Model (CKM) — language/storage/AI-provider independent, the shared shape every backend and query surface reads.

commit

Writes the CKM into the append-only Semantic Knowledge Ledger. Nothing is ever modified in place; every conclusion carries the evidence it was derived from.

Key invariants, enforced by review, not just convention: the Observation Layer never interprets meaning; the ledger is append-only; the Runtime is read-only; every semantic conclusion traces to Evidence; every compiler pass is deterministic and side-effect-free; every artifact is content-addressable; secrets/PII are never observed or stored.

02

Connectors

Each connector implements the same Observer trait — pull raw facts, no interpretation. Real and tested unless noted.

File

Any local filesystem tree, respecting [observe] ignore-patterns — which is the only filter: .gitignore excludes nothing from the observation walk, so a git-ignored virtualenv or tool cache is still fully ingested unless named here. Use ekos config preview-scan to see what would actually be observed before compiling.

Git

Full commit/contributor history as real KirEvents.

GitHub

Issues and PRs, live-verified against a real repo — 1,600 real items in one run.RFC 0020 / RFC 0062

Confluence

Space/page content via the REST API.

Local documents

PDF/DOCX/text/Markdown/HTML/email — text, tables, and image OCR.

Pentaho Kettle

.ktr/.kjb jobs compiled into the shared Transformation IR.RFC 0027

Python / PySpark

Real AST parsing; DataFrame chains recovered into the Transformation IR; SQLAlchemy ORM models and class inheritance.RFC 0038 / 0040 / 0091 / 0092

Rust

Real AST parsing and a real function-call graph (Calls edges).RFC 0041

JavaScript / TypeScript

.js/.jsx/.ts/.tsx/.mjs/.cjs source, structurally parsed.RFC 0085

Elixir

.ex/.exs modules and functions.RFC 0081

Perl

.pl/.pm/.t/.psgi source, plus .cgi gated on a real perl shebang. Packages and subs; use/require dependencies (pragmas excluded); use parent/use base/@ISA inheritance; POD descriptions, source spans and signatures. Hand-written structural scanner — Perl is not statically parseable, and shelling out to PPI/perl -MO=Deparse would execute BEGIN blocks from observed code. Not a call graph: Perl dispatch is fully dynamic. Measured on LedgerSMB's lib/: 272 packages, 1,333 subs, 1,154 dependency edges, 34 inheritance edges, 1,313 source spans, 823 sub and 272/272 package descriptions, zero LLM calls.RFC 0147

Java

.java source. Types (classes, interfaces, enums, records, annotation types, nested), methods and constructors with spans, signatures, visibility and Javadoc; imports and inheritance resolved the way Java resolves names, and only to types in the observed tree — a library type is listed, never invented. SQL built in method bodies (String/StringBuilder, constants through the supertype chain, DB.* calls, new Query(…)) is folded, parsed and linked to the tables it reads and writes, each statement labelled parsed / partial / unparsed. IDEMPIERE-NNNN ticket ids anchor to the method or type they sit in. For the iDempiere/ADempiere family, the generated I_*/X_* classes become ModelTables: every column with its business name, description, list values and labels, mandatory flag and default, updatability and code-declared foreign key — a claim by code, kept apart from the database’s own Table. Hand-written declaration parser (no C toolchain). Measured on iDempiere release-13: 4,542/4,542 files parsed, 4,820 types, 31,434 methods, 763 tables with 16,660 columns and 2,882 code-declared references, 3,207 embedded SQL statements (1,992 parsed, 996 partial, 219 unparsed), zero LLM calls.RFC 0209

Compiled .NET & JVM binaries

.dll/.exe (managed CLI assemblies) and .class/.jar/.war/.ear — for programs whose source is gone. Recovers types, methods, signatures, fields, the real call graph, branch structure, string and numeric constants, and external I/O boundaries (database / HTTP / file / messaging / process), each carrying a provenance locator down to the bytecode offset or metadata token plus the binary's SHA-256. In-process, pure Rust, #![forbid(unsafe_code)], no .NET SDK or JRE required, nothing executed — a customer's untrusted DLL is only ever read as bytes. Format is decided by magic bytes, so the overwhelmingly common native .dll is reported as skipped rather than as an empty assembly. Measured on pdfbox.jar + mscorlib.dll: 3,717 types, 33,323 methods, 18,861 fields, 187 I/O boundaries, zero LLM calls. On a real lost-source Windows CE app (12 assemblies: handheld client, desktop sync server, vendor SDK wrappers): 888 types, 8,327 methods, 10,956 resolved call edges, each joined inside the assembly its metadata names — so two binaries built from shared source, or two copies of one exe, never borrow each other's methods — and assembly references and base types land on the real binary when it was observed alongside. I/O boundaries are concrete types only: a database provider or connection/command, never the in-memory DataSet/DataTable model. Ships as a separately licensed extension build, not in the open-source repository: it plugs into the public CLI through the extension seam (extra observers, recover passes, a post-commit step and MCP tools), and a ledger it wrote still resolves and queries in the open-source build.Statement-level recovery (RFC 0150, .NET): each method body is decoded to if/loop/switch/try statements with the IL offset of every line, labelled per method as structural, control_flow or statements — never higher than it reached. Checked, not asserted: on wine-mono's mscorlib (24,526 bodies, 96.0% fully structured), Newtonsoft.Json 13.0.3, and the TSD application, every structured method executes like its control-flow graph and no body of any fidelity loses a statement; against the published source of the same builds, decision points agree within ±1 in 98–100% of methods and string constants are recovered at 99.6–100%. ekos_binary_explain with method returns a per-method spec (pseudo-code, I/O calls with arguments, SQL and its tables, exceptions, callers/callees) and a callees-first migration order; ekos_binary_migration_check compares a Python rewrite against the original. Obfuscated binaries carry a trust warning and packed ones are refused. Characterization tests (ekos-characterize) go further than the static check: they run the original assembly — under wine-mono, inside a bubblewrap sandbox with no network and no access to the host — on recorded inputs, store what it returned, threw, printed and wrote, and check a Python rewrite against that record; the check step needs no wine and no untrusted code, so it can run in CI from a committed golden file.RFC 0148 / 0149 / 0150

ClickHouse

Live HTTP schema metadata plus an opt-in live NL-to-SQL query engine — see §14.RFC 0056

Crypto / DeFi export

DeFi Sentinel Parquet exports.RFC 0017

Scaffolded (proof-of-concept only)

Salesforce, SAP, Oracle, Microsoft Fabric, Snowflake — real API shapes, mock-tested, none yet exercised against a live account. PostgreSQL, SQL Server, and Jira remain planned.

Extensions add connectors, passes, tools and commands. An out-of-tree crate implements EkosExtension and builds its own binary with main_with: extra observers for build, passes for recover, a post-commit step and MCP tools (RFC 0149), and since RFC 0191 whole command groups of its own. Every leaf command an extension adds needs a class (read / write / review / admin / external / local), so it appears in ekos commands --json and the console runs it under the same rules as a built-in. A name that is already taken, or a command without a class, stops the binary at start-up. An extension's MCP tool is refused to a reader with restricted access unless the extension declares that the tool reads only through the store it is handed, which the read scope filters. EKOS Lakehouse (RFC 0190–0196) is built this way, in its own repository: PostgreSQL → Apache Iceberg pipelines (Trino, Polaris, dbt-trino, Dagster) generated from the compiled source — bronze reconciled against the source on every run with a planted control, tested and lineage-checked silver, source schema changes classified into nine classes and applied or proposed as branches with their impact, gold marts drafted from written descriptions under the anti-invention check, and per-table health served to people, webhooks, OpenTelemetry and MCP agents.RFC 0149 / 0191

Extensions add console pages, and install in one command. An extension can declare workspace pages for the web console (RFC 0198) — as data, never code: each panel shows the JSON one of its read commands prints, as a table, fields or text with status badges; each button runs one of its catalogue commands as a job, with the command's own class, role and typed confirmation. ekos extensions --json checks every page against the command catalogue, and the console checks again before each panel request, so a page can never make the console run a write. EKOS Lakehouse uses it for its Lakehouse tab: health per table and layer, proposals waiting for a merge, load modes, refused tables, the stack, and doctor / sync / check / plan / generate. It installs with curl … install.sh | sh: a verified full package (its binary, the console and the Python that runs it, assembled by EKOS's own packaging under the binary's name), then ekos-lakehouse lakehouse install starts the Docker stack, creates a project in the EKOS home — the sample shop, or --source host:port/db — runs its pipeline once, starts Dagster on its schedule and opens the console. A connection's password can now come from a file only its owner can read (ConnectionRef::secret_file), which is what a console job or an orchestrator container has instead of your shell's variables; ekos setup installs and starts the console under whichever binary runs it.RFC 0198 / 0199

Pentaho packages become silver and gold dbt models. EKOS Lakehouse reads a whole Kettle package — transformations, jobs, repository exports, shared.xml, kettle.properties, plugin jars — into a typed model with variables resolved and links between files followed, lowers every step into a relational IR with Kettle's semantics made explicit (null-as-value comparisons, empty string as null, left-to-right conditions, round-robin distribution, consecutive-row grouping), executes JavaScript and Java code steps symbolically into SQL, and splits each transformation into silver models (the cleansing) and gold models (what it writes: tables, merges, slowly changing dimensions as dbt snapshots, Abort steps as tests). Every step is labelled exact, rewritten (with the rule named), llm-drafted or gap (with its original text, and every model that needs it disabled); Trino plans every model before a branch is proposed, and merging it is the approval. A model can draft the code steps the translator cannot follow, but a draft counts only if it parses and reads nothing but the step's own inputs and literals; a paid model needs --allow-metered and, from the console, the tenant's permission. The default route is Claude Haiku 5.5 with GPT-6 Luna as fallback, chosen by a bake-off: on 15 steps the deterministic translator also handles, Haiku's drafts matched it row by row on Trino in 11, at about $0.035 a run; nearly every disagreement is on the null row, which is why drafts stay labelled for review. Compiled plugin steps are decompiled only by the private build, through EkosExtension::java_decompiler(): it recovers JVM bytecode to statements (RFC 0206 — 94% of pdfbox.jar's method bodies, with NaN-exact float comparisons), and the plugin's processRow then goes through the same translator as a User Defined Java Class. Whether translations return the same rows as Pentaho on real estates is open for the community to check (docs/pentaho-verify.md). Connection passwords never leave the files — Kettle's reversible Encrypted … values are now part of the built-in redaction. Console pages gained selections (pick a package, then a step), side-by-side code, graphs and Markdown (RFC 0207), which the Lakehouse Pentaho → dbt tab uses to show a step's Kettle text next to the SQL it became.RFC 0200–0205 / 0207

03

Knowledge recovery

A Pentaho step, a SQL SELECT, a VIEW, and a stored procedure are all the same underlying concept — sources flowing through filter/join/aggregate/calculate operations to a sink. ekos recover compiles all of them into one shared Transformation IR (Source/Filter/Join/Aggregate/Calculate/Sink/Unmapped), so legacy ETL logic can be diffed against a newly drafted pipeline without hand-reading XML. Unmapped is deliberate: anything unparseable is still recorded as evidenced fact, never silently dropped.RFC 0027 / 0028 / 0029

File-based SQL needs a dialect. .sql files parse under the generic ANSI dialect unless ekos.toml's [[recover.sql.dialect-rules]] (first path-glob match wins) routes them to postgres/clickhouse/etc. This is load-bearing for any real schema dump (pg_dump, a ClickHouse structure.sql): parse_ddl_structural parses the whole file in one pass, so one statement the dialect can't handle (CREATE TYPE … AS ENUM, ENGINE = MergeTree, CODEC(...), …) discards every table in it — reported only as a buried SQL001: no tables found warning, not an error. The postgres/clickhouse dialects carry the preprocessing (RFC 0057/0058/0059) that makes real dumps parse; generic does not.RFC 0031 / 0057 / 0058 / 0059

Trino. A trino dialect parses the SQL dbt-trino models and Iceberg pipelines are written in: catalog-qualified names, WITH (...) table properties, lambdas, ROW types, MERGE. Three Trino constructs no parser dialect reads are blanked before parsing, keeping line numbers: Iceberg time travel (FOR VERSION|TIMESTAMP AS OF, the table itself kept for lineage), ALTER TABLE … EXECUTE maintenance statements and TABLESAMPLE. The dbt analyzer falls back to it, so a dbt-trino model that reads an Iceberg snapshot keeps its filters and column lineage.RFC 0190

Oracle, PL/SQL and schema history. An oracle dialect reads Oracle schemas and SQL*Plus scripts the way SQL*Plus does: SET/PROMPT/@file lines and whole PL/SQL units (ending at a / line) are set aside, and Oracle-only clauses — ENABLE NOVALIDATE, ADD (…)/MODIFY (…)/DROP (…) column lists, MODIFY c DEFAULT x, VARCHAR2(10 BYTE), FORCE views, q'[…]' quoting, (+), MINUS, physical attributes — are rewritten, keeping every line number. ekos init --detect writes one dialect rule per engine-named directory, every Table records the dialect it was read under, and definitions are compared for conflicts only within one dialect: an Oracle script and its PostgreSQL twin describe two databases. PL/SQL functions, procedures, triggers and packages are read into the same procedural IR as PL/pgSQL and written as the same Procedure/ProcedureStatement objects with a computed fidelity label; an Oracle trigger’s inline body is classified like any trigger function. Migration scripts named <yyyyMMddHHmm>_<description>.sql become SchemaChangeSets (time, release, engine, tickets) holding one SchemaChange per operation, linked to its table at commit — claims, not proof that a change was applied. On iDempiere release-13: Oracle migration statements rejected 5,317 → 9 of 62,251; 60 of 61 db/oracle routines recovered completely (991 of 992 statements); 2,154 change sets with 26,060 changes, none unrecognized; zero LLM calls.RFC 0210

Live Oracle catalog and code ↔ database conflicts. A project can name an Oracle database in [[databases]] (connect string, user, owners, and the name of the variable holding the password — never its value). ekos build reads the catalog through ora-live: settings and time zone pinned, a call timeout, every read inside SET TRANSACTION READ ONLY, a write probe that must be refused, DBA_* views (through ALL_* a read-only user silently sees none of the functions), and each kind reconciled against the database’s own object counts, so missing visibility is an error rather than a smaller answer; text is redacted before it leaves the crate. ekos recover writes ordinary Table/View/Procedure objects marked origin: live, which never merge with script-defined tables and are used for name links only where no file defines the name. ekos commit compares the code’s claims with them and writes conflicts citing both sides — model_table_missing, table_without_model, model_column_missing, mandatory_but_nullable, routine_text_differs — with an explanation when the source tree accounts for the difference: a migration newer than the database (by date and by release, since scripts are dated when written, not when shipped) or a translation table; ekos conflicts show prints it. Built with --features oracle; needs Oracle Instant Client at run time. On iDempiere release-13 against its release 12 seed on Oracle XE: schema changes linked 11,748 → 28,916 of 31,620; 444 conflicts, 239 explained, all 173 pending-migration explanations confirmed against the database’s own AD_MigrationScript; the rest include leftover tables of removed modules, a model column no script created, and an installed procedure that differs from its file; zero LLM calls.RFC 0211

Database profiles without row values. [[databases]] profile chooses how much ekos build measures: p0 (the default) reads optimizer statistics only; p1 counts each table’s rows; p2 runs one read-only aggregate scan per table for exact null counts, number and date bounds and text lengths. A text column is never the argument of a minimum or maximum, number and date bounds are withheld for columns named like personal data (the rules pg-live uses, now shared), and a live test seeds a marker row and checks that none of its text reaches the snapshot. A mandatory model column that is nullable in the database then carries the number of rows breaking the rule and the query that re-counts them (ekos conflicts show), and the catalog records archive-log and supplemental-logging state for change data capture. A database is read on every build, independent of whether files changed; a catalog read replaces earlier reads at compile, and an object dropped from the database leaves the model. On the iDempiere seed: 903 tables in 6 s; of 116 mandatory-but-nullable columns 11 hold nulls and 105 none, each count re-checked against the database.RFC 0211

Write behaviour per table. ekos commit writes a WriteBehaviour for every table: an update class (po_only, raw_sql_sets_updated, raw_sql_skips_updated — a raw UPDATE that leaves Updated alone, which an incremental copy keyed on it would miss — or no_updated_column), every way rows disappear (deleted_by_app, deleted_by_users, deleted_by_cascade, purged, soft_delete_only, never_deleted_in_code) and whether deletes are change-logged, each with citations to the method, routine, foreign key or flag behind it. The inputs are the columns each UPDATE sets and how each DELETE chooses rows (by age, all rows, or selectively), PO delete calls resolved from declarations in the calling method, live cascading foreign keys, and — opt-in with [[databases]] dictionary = "idempiere" — the application dictionary’s IsDeleteable and IsChangeLog flags. Statements in migration scripts are ignored. On iDempiere: 144 of 903 tables have a raw update that skips Updated; 754 are deletable by users, a fact only the dictionary holds.RFC 0212

Ticket anchors. With [tickets] keys set (ekos init --detect proposes keys that name many files, such as IDEMPIERE in 202409031411_IDEMPIERE-6203.sql), every ticket a commit message names becomes a Ticket object — its commits, dates and authors, a title taken from the first commit’s subject, and Touches links to the files it changed — and ekos commit links it to every migration change set and Java type or method that cites it. Keys are matched exactly; a generic pattern would match UTF-8. [observe] git-max-commits sets how much history is read (default 500). On iDempiere’s 16,260 commits: 4,597 tickets, linked to 3,074 change sets and 426 Java types and methods, so a schema change reaches the commit that explains it.RFC 0219

Run logs. build, recover, resolve, compile and commit each write a JSON-lines run log — one line per step with its counts (connectors scanned, each Oracle dictionary query with rows and milliseconds, the reconciliation, a pass summary, analyzer stats, facts written), the warnings logged (redacted, the first 200 copied and the rest counted), and two closing lines: the model calls the command made, then its outcome. The file is EKOS_RUN_LOG when set, otherwise .ekos/runs/<timestamp>-<command>.jsonl; the console gives each run one file and shows it as a Steps table on the run page (GET /api/runs/{id}/events). On iDempiere with the live Oracle testbed every command logged zero model calls.RFC 0220

Run logs for every command that changes something, and the demo book. Every command whose catalogue class is write, review, admin or external writes the same run log, named after its path (<timestamp>-lakehouse-marts-build.jsonl), with the stage lines an extension adds (a mart merged, a build launched, each measure's two totals, an extraction change). docs generate --layout estate records its latest run in .ekos/docs/estate.json, and step 2 of the workflow bar reads it. ekos-lakehouse's tools/demo-book walks the console through the seven steps — screenshots, the facts the pages show, every console and terminal run log — and renders a book per step: the screen, what the operator clicks, the run log, what happens behind it, what they get, the caveats. Numbers, logs and screenshots are regenerated on every render; narration lives in manual sections a render keeps; what the capture lacks fails the render. The iDempiere book (8 pages, a 21-page PDF) is in the demo repository.RFC 0218

Documentation repositories and wiki snapshots. A Markdown documentation site is a project source like code. Front matter becomes the document’s properties (title, description, and for a wiki snapshot source_url, source_revision, licence) rather than a section’s text, and each section records the tickets it names, which ekos commit links to the ticket’s commits, migrations and code. On iDempiere’s code plus docs repository, 1,743 docs sections were linked to tickets. scripts/wiki-snapshot.py turns MediaWiki pages into Markdown with provenance, from a browser-saved Special:Export file or from the API, which it stops using at a 403 or a bot challenge.RFC 0221

Sources in the console. A project's Sources tab is step 1 of the workflow: add repositories and folders, add an Oracle, PostgreSQL or ClickHouse database (connect string, a read-only user, the schemas, and for Oracle the profile tier), Test connection — a read-only catalog read that checks the server refuses a write and puts the database's census next to what was compiled — then Init & compile. The password goes to the server's keychain and reaches only that tenant's processes; ekos.toml holds the variable's name, and an API token cannot enter one. The same from a terminal: ekos database add|list|test|remove. On iDempiere, from an empty project in a browser, census and compiled counts were equal for tables, views, routines and triggers. Identity candidates now have to agree word for word (a table and its child, or two packages of one scope, are no longer proposed): iDempiere's identity review went from 25 candidates to 1.RFC 0223 / 0224

PostgreSQL and ClickHouse databases. [[databases]] kind is oracle, postgres (connect = "host[:port]/database", owners are schemas) or clickhouse (connect = "http[s]://host[:port]", owners are ClickHouse databases); ekos database add --kind and the Sources tab's Kind selector set it. PostgreSQL is read through pg-live’s read-only session and reconciled against the server’s own counts; every ClickHouse catalog query carries readonly=1. Both run a write probe that must be refused and emit the same catalog artifacts as Oracle, so one analyzer compiles all three: tables with keys and foreign keys, views (ClickHouse sorting and primary keys as table properties, never as a unique key), PL/pgSQL routines and triggers through the readers files use, a PostgresCatalog / ClickHouseCatalog summary per schema, and a dropped object leaves the model. ekos database test now counts compiled objects from the latest model rather than the append-only ledger, so a dropped object no longer shows as a difference (for Oracle too), and the integrity check knows ClickHouse’s built-in functions. Against local PostgreSQL 16 and ClickHouse 24.8: census equal to compiled for tables, views, routines and triggers; references 100% resolved.RFC 0235

Oracle as a lakehouse source. EKOS Lakehouse reads Oracle as well as PostgreSQL: source-engine = "oracle" with source-database naming the project's [[databases]] entry, so steps 1–7 share one connection. lakehouse plan maps each column with an Oracle → Iceberg type map (a DATE keeps its time as timestamp(6); an unconstrained NUMBER lands at the scale its data uses, a column with more than 38 digits is refused with the reason) and lakehouse reconcile compares bronze with Oracle itself at V1–V3: the canonical form is rendered in Oracle SQL (an AL32UTF8 database), a wide row is hashed over column digests because an Oracle string stops at 4000 bytes, and a decimal with more digits than its scale is never rounded into agreement. On iDempiere: 901 of 903 tables planned, 12 tables loaded and reconciled green with the planted control firing.RFC 0216

Incremental loads and deletes from write behaviour. With application-watermark = "UPDATED", EKOS Lakehouse reads EKOS's write-behaviour facts: a table whose code always sets Updated loads incrementally on it (the plan says what that cannot see — an update made outside the application — and reconciliation catches it the next run), and each such table gets a delete plan from how it is deleted: checked every run, or audited weekly where the code never hard-deletes. lakehouse deletes compares 256 key buckets on the source and on bronze and reads keys only where a bucket differs; when more than 5 % of a table's keys (or 10,000) vanish at once it applies nothing and waits for a person (--accept), because a lost permission or a wrong database looks the same as a mass delete. Purges can be retained in the lakehouse. On iDempiere: 739 of 901 tables incremental; single, re-inserted, mass and purged deletes each reconciled green.RFC 0225

The extraction plan, reviewed and approved. Step 5 of the workflow is a decision a person makes. The lakehouse plan applies step 3's decisions from every review subject with a valid sign-off — tables out of scope are not planned, confirmed PII columns are masked in silver, write-behaviour overrides change a table's mode and delete plan — and saves the result as a proposal. The console's Extraction page (or lakehouse extraction show|change|approve) shows each table's load mode, delete plan, type mappings and masks with their evidence; the operator changes a table with a typed reason and approves the plan, which records who approved which plan (its digest) in the ledger. With require-plan-approval = true, generate builds only the approved plan and, when the source, step 3 or a change has moved since, refuses and names the tables. Extension page actions can now take typed text (Action::inputs). On iDempiere: 849 tables planned, 53 out of scope, one column masked.RFC 0226

Silver from the source model. The lakehouse plan carries, per column, what the database declares (Oracle check constraints such as IsActive IN ('Y','N')) and what the application's code declares (EKOS's application model: boolean accessors, reference lists, logical foreign keys). Silver turns Y/N flags into booleans — testing on bronze, for a flag only the code declares, that nothing else is stored — fails the build on a value outside a declared list, and warns on values outside the application's lists or on logical keys with no parent, since the database enforces neither. Rows the source purges but the lakehouse keeps carry the time the purge was detected, and history closes a row deleted at the source with the time the delete was seen. lakehouse run starts the Dagster job (the console's Run the pipeline). On the iDempiere testbed a closed subset of 165 tables ran end to end in Dagster: every bronze table reconciled with Oracle, 165 silver models built, 3,664 of 3,667 tests passing and three warnings on real data. A change to how a column is loaded now requests a reload of the incremental tables it affects. RFC 0227

Requirements to mart metadata (step 4). On the console's Requirements page an operator describes a mart in plain words; the model routed to the requirements task (local Ollama unless a metered route is allowed) drafts its grain, dimensions, measures and filters from an excerpt of the plan, and EKOS resolves every term deterministically against the approved plan — names in any case, at most four foreign keys per join, one path per table, and … via table.column to say which path when there are several. Whatever does not resolve, or could mean more than one thing, is a question for the operator, answered by changing the term, never by the model. Approving commits marts/<mart>.md and records a ledger fact; with require-plan-approval, gold is built only from approved requirements. CLI: lakehouse requirements draft|edit|show|approve. On iDempiere, the plan's own example request drafted by llama3 was answered in nine edits (three join paths chosen among 3, 36 and 19) and approved, and its gold mart was proposed with nothing invented.RFC 0215

Marts (step 7). The console's Marts page takes each approved requirement to a gold table: Propose drafts the dbt model on a branch with nothing invented, Merge is the person's approval of the generated model (a ledger fact), Build runs only that mart and its tests through Dagster, and Reconcile shows each measure's total in the mart next to the same aggregate over silver. The page puts the SQL beside the requirement, marks each term as compiled exactly or drafted by a model, and draws the mart's lineage. CLI: lakehouse marts show|propose|merge|build|reconcile. On iDempiere four marts went from approved to reconciled with every total equal, the fourth entirely on the page.RFC 0228

Source-estate documentation. ekos docs generate --layout estate writes a database estate's documentation from the ledger, ready for a wiki: functional areas with ER diagrams, a page per in-scope table (columns and business names, the database's foreign keys beside the application model's references, check constraints, readers and writers, write behaviour, schema history, open conflicts), a glossary, where each document type's DocAction methods start, routines with nested statements, schema history by release, and findings. The lakehouse-docs publishers send it to a GitHub wiki, Confluence or an Azure DevOps wiki; notes sections survive regeneration. Scope comes from the review. No data value is ever shown (profiles give counts only), and text is used only from sources with a known licence — detected from the source's licence file or set in [docs-estate] licences. iDempiere: 1,060 pages in 26 seconds.RFC 0214

Documentation (step 2). The console's Documentation page generates those pages and publishes them to the store a project names in [extensions.lakehouse.docs] — a GitHub wiki, an Azure DevOps wiki or Confluence — with Preview publish (what would change, nothing sent) before Publish. The lakehouse-docs publishers are compiled into ekos-lakehouse; credentials stay in the publishers' own environment variables. Step 2 is done only when the latest generation has been published. CLI: lakehouse docs show|generate|publish [--dry-run].RFC 0230

Review and sign-off. ekos review computes six subjects from the ledger — tables in scope (classed by name rules, [review.scope]), PII candidates, code ↔ database conflicts, identity matches, write behaviour per table, the glossary — each item with the facts a reviewer signs and measurements shown only. A person decides items (review scope|pii|write-behaviour; conflicts, identity and glossary keep their own commands) and signs a subject off with ekos signoff approve <subject> --digest, which is refused if the review changed since it was shown or while unexplained conflicts or unconfirmed matches remain. Validity is recomputed on every read: on the iDempiere testbed a column added in Oracle made the PII sign-off stale, naming the new column, while the scope sign-off stayed valid. No MCP tool can decide or sign. ekos workflow status shows the seven-step operator workflow (analyse, documentation, review and sign-off, requirements, extraction, bronze/silver, marts): step 1 from source freshness, step 3 from the sign-offs, the rest from an extension such as ekos-lakehouse; steps 4–7 are blocked until every subject is validly signed, and their commands check the same gate before writing (ekos-lakehouse with require-signoff = true). In the web console the step bar heads every project page and the Review tab does all of it in a browser: a tab per subject, a filterable table of items, decisions for the selected ones (identity matches too, also ekos review identity on the CLI), and a sign-off that shows the digest and what changed since a stale one, and is recorded only after the subject’s name is typed. Deciding and signing need the reviewer role, which no API token can hold.RFC 0213

iDempiere eval. evals/idempiere/ is a 25-question dataset (idempiere-25) about the iDempiere ERP — Java code, migrations and PL/SQL, git history and tickets, the documentation site, write behaviour, and six questions with no grounded answer that a correct answer declines — every expected fact checked in the source first. ekos eval run --datasets-dir evals/idempiere --dataset idempiere-25 grades it; --evidence-only grades retrieval without a model. On code + docs with and without the live Oracle database: expected facts reached the evidence for 9 of 18 answerable questions in both; answered by Claude Haiku 5.5, 14 and 15 of 25 passed with no hallucination for about 2 US cents a run. Each miss is attributed; most are documentation prose outranking the code that implements an answer.RFC 0222

One bad statement costs one statement. When a file does not parse as a whole, each top-level statement is tried alone; one that fails is skipped and the rest of the file is recovered with its real line numbers. SQL005 lists what was skipped (line, first words, parser error) and calls out a skipped CREATE TABLE as a lost table. SQL001 now says why: when a file states a CREATE TABLE and none was recovered, it names the dialect it was parsed under, the first error, and [[recover.sql.dialect-rules]]. Foreign keys written the way pg_dump writes them (ALTER TABLE … ADD CONSTRAINT … FOREIGN KEY) become edges. On LedgerSMB this recovered 13 more tables and 11 more foreign keys.RFC 0176

Hand-written schemas need more than the right dialect. The postgres dialect had been tuned entirely against pg_dump output, which emits a narrow, mechanical subset of PostgreSQL. A human-maintained schema does not. Measured on LedgerSMB (a Perl/PostgreSQL accounting ERP, 260 .sql files), a correctly-configured dialect = "postgres" still recovered 0 of 158 tables: the file's first COMMENT ON TABLE … IS $$…$$ failed the whole-file parse and took every table with it, reported only as a buried SQL001. The dialect now preprocesses COMMENT ON, INHERITS, SECURITY DEFINER, RETURNS SETOF, := named arguments, DO blocks, CREATE RULE, psql meta-commands and COPY … FROM stdin payloads out of the way. Same repo, same config: 8 → 192 Table, 1 → 218 ForeignKey, 8 → 470 data-lineage links.RFC 0146

Business logic out of a binary, without pretending it is certain. Structural recovery from bytecode is deterministic and never generative. On top of it, an opt-in [binary-reconstruction] stage asks an LLM to infer business-meaningful names and input → condition → outcome rules from those facts alone. The guard is mechanical rather than rhetorical: every rule must cite evidence locators, and each locator is checked against the exact set its own prompt contained — invented ones are dropped and counted, and a rule left with none is discarded outright. Confidence multiplies the model's own self-report by a hallucination penalty and by cross-reference agreement across call sites, and can only lower it, never promote. Anything below the threshold, or contradicted by another reconstruction of the same locator, is written status = "unconfirmed" rather than as a fact. Inferred facts carry extractor = "llm-reconstruction-v1", never the deterministic readers' ekos-jvm-classfile/v1 / ekos-cil-metadata/v2, so no query can conflate the two. Every method also compiles into the shared Transformation IR (RFC 0027), so recovered binary logic is diffable against a drafted rewrite through the existing ekos_transformation_diff.RFC 0148

Stored procedures, statement by statement. A PL/pgSQL routine is imperative — ordered statements, branches, loops, exception handlers — so it gets its own procedural IR rather than being forced into the dataflow one. ekos recover parses every routine in a postgres-dialect file with a deterministic in-process parser (no database, no LLM) into a Procedure object carrying a fidelity label computed from the parse (statements, partial with exact counts and the lines missed, or signature for an unparsed language), plus one ProcedureStatement per statement at every depth whose evidence is its own source text and line, linked by Contains edges that record the branch (then:0, else, handler:1). On LedgerSMB, 212 of 212 installed routines recover completely. Each statement’s embedded SQL is parsed as real PostgreSQL to record what it reads, writes and calls, and ekos commit links routines to tables (ReadsFrom/WritesTo per statement, DependsOn per routine, so impact analysis on a table finds the routines that touch it) and to the routines they call (Calls) — only where a name matches exactly one object, never a guess.RFC 0163

Triggers, classified by what they do. Every CREATE TRIGGER becomes a Trigger linked to its table and function, and is classified from its function’s recovered IR — never its name — as Validation, DerivedColumn, Audit, Cascade, Mixed (a human decides) or Unknown (no recovered body), always with its reasons. A function redefined across migrations gets a class only when every definition agrees, with RETURN NEW placeholders set aside openly. Never auto-translated.RFC 0163

Views are objects, not strings. Every CREATE [MATERIALIZED] VIEW in an observed SQL file becomes a View with its columns, its exact source and line, and what its query reads and calls from a real parse — parsed one definition at a time, so an unparseable view is still recorded with the reason. A view DependsOn the tables and views it reads, and routines and transformations that read a view link to it, so impact analysis on a table reaches the views built on it. Routines and views are documented like code: one curated page each, carrying the author’s own COMMENT ON FUNCTION|VIEW text as its description; views listed among the Data Stores with the routines that use them; routines in API.md under their file; opt-in LLM descriptions from their real source.RFC 0169

The schema's own documentation is evidence, not filler. COMMENT ON TABLE/COMMENT ON COLUMN used to be discarded outright — and an LLM was then asked to invent descriptions for the same tables. That text now becomes a real description on the Table object and the matching column, each with a SourceLocation line. Author-written text outranks generated text: description holds the human's words, the model's version is kept alongside as llm_description rather than discarded, and sql_comment records the provenance so a consumer can tell an observed fact from an inferred one. On LedgerSMB: 127 of 182 described tables are in their authors' own words, plus 106 described columns.RFC 0146

Enrichment budgets fit the schema, and partial coverage is reported. SqlAnalyzerPass had a hardcoded 4,096-token ceiling no configuration could reach ([ai] max-tokens governs only the read-side ekos ask runtime), so a 158-table file came back empty or covering a fraction of the schema with no diagnostic at all. Each file's budget is now computed from its own table and foreign-key count, [llm] max-tokens overrides it for compiler-pass calls, and a new SQL004 warning says how many tables were named versus declared — naming the ceiling explicitly when the response hit it. Reasoning models such as DeepSeek V4 Flash spend hidden tokens from the same budget, which is what made the old ceiling fail silently.RFC 0146

Beyond raw SQL DDL, the Python analyzer recognizes real SQLAlchemy declarative models (__tablename__ present) and compiles them into the same Table shape a CREATE TABLE produces — real columns, best-effort types, same-file ForeignKey resolution — plus real class inheritance as Extends edges.RFC 0091 / 0092

Documents keep their structure. Markdown is split per heading (code-fence aware): every Section is named by its heading path (docs/rfcs/0013-mcp-server.md § RFC 0013 — MCP Server › Motivation), carries its real line range, a doc_type (rfc/devlog/readme/claude_md/doc) and, for an RFC, rfc_number/rfc_title/rfc_status; its whole text is indexed (previously only the first 1,200 of 2,500 characters). compile then derives deterministic doc links: an RFC NNNN mention becomes a References edge to that RFC (tagged depends_on/supersedes/mentions), and a backticked identifier that names exactly one code symbol becomes an edge to it — an ambiguous name never links. On EKOS itself: 7,377 doc→code and 3,420 RFC→RFC edges. Questions naming "RFC 0013" resolve to that RFC by number; doc mentions are navigable neighbours but never counted as dependents in impact analysis.RFC 0144

Business semantics from technical traces, exported as LinkML (experimental, opt-in [semantics]). Meaning nobody wrote down survives as traces: WHERE NOT p.obsolete, entity_class = 2, the lookup row (2,'Customer'), a COMMENT ON COLUMN legend, the commit that last changed the line. Recovery records every column-vs-literal predicate in views, routines and CHECKs, normalized so status IN (1,3) and status = 1 OR 3 = status are one predicate, plus key columns and lookup seed rows — and PL/pgSQL IF NEW.col … conditions (resolved to the trigger’s table), Pentaho FilterRows, standalone analyst SELECTs and dbt schema.yml tests (accepted_values becomes a declared domain, relationships a lookup link). Opt-in llm-definitions adds a cited, cite-or-drop AI summary to undocumented concepts — a reading aid, never the definition. dbt model SQL is rendered (ref/source/var) so a model’s filter defines a concept; application constants (Perl use constant EC_CUSTOMER => 2, Python Enum classes) name codes when they agree with known labels or the column’s name; SQL inside Perl strings is read; filters through CTEs, derived tables and dbt model columns are traced back to the source column; marked glossaries document concepts and codes, and an unmatched term becomes a gap; [semantics] ontology suggests mappings from your own vocabulary (annotations only). ekos semantics gold-template gives a domain expert a blank, structure-only gold file to fill in before seeing any output; eval flags any gold set not marked as written blind. ekos commit then synthesizes hypotheses — BusinessConcept, EnumMeaning, ConstraintCandidate, SemanticGap, RationaleLink (git blame -w) — each with path:line evidence, deterministically and with no LLM. ekos semantics list|show|gaps|eval read them. Only a human promotes one: ekos semantics confirm|reject|edit is CLI-only (a source-scan test keeps MCP from reaching it); a review holds only while the item’s assertion and evidence are unchanged, otherwise it becomes needs_review, and a confirmed item whose traces vanish is flagged, never dropped. Threshold disagreements on one column become ConceptConflicts. ekos export linkml writes LinkML with expert names and labels (default --status confirmed, so hypotheses never leave as facts by accident), and ekos import linkml turns an expert’s edits to that YAML back into review decisions — the console’s Semantics tab wraps both in a viewer and editor. Agents read the results over MCP — ekos_semantics_lookup and ekos_semantics_gaps — with every answer’s status spelled out and rejected definitions left out; no MCP tool can confirm one. EKOS feeds LinkML; LinkML generates the rest. On LedgerSMB (SQL, Perl and UI observed): 47 concepts, 222 coded values (179 explained), 649 predicate sites (458 resolved to a table column) — demo/ledgersmb-linkml/run.sh reproduces it end to end; the export passes linkml-lint with 0 errors. No trace, no recovery: meaning defined only by column-to-column or date comparisons is not recovered.RFC 0170

An opt-in document semantic memory pass reads local documents through an LLM to extract real Concept objects and their relationships, so the same idea mentioned across different documents becomes one findable, linkable thing — queryable through the exact same MCP tools as everything else, no new surface.RFC 0025 / 0026

A dbt project analyzer extracts real Table objects from a dbt project's own checked-in metadata only — never manifest.json/catalog.json (both are dbt/target/ build artifacts, gitignored in every real project inspected while designing this) and never a live warehouse connection, since dbt itself can point at any database. One Table per models/**/*.sql file (a model exists the moment its file does, regardless of YAML documentation) plus one per declared sources[].tables[] entry; ref()/source() macro calls become real DependsOn edges, with an unresolvable reference (e.g. into a gitignored dbt_packages/) honestly skipped rather than fabricated.RFC 0117

04

Identity resolution

The same real-world entity observed under different names across systems — Informix cust_mstr, Postgres customers, Databricks gold.dim_customer — can be linked. ekos identity scan scores candidate cross-system matches (column overlap, naming-pattern similarity, type compatibility) and writes them as unconfirmed relationships for human review via ekos_identity_review — never a silent auto-merge.RFC 0029 / 0063

Same-source duplicates (two Table objects both literally named customers) auto-merge only on an exact normalized-name match during resolve/compile; anything fuzzy goes through that same review flow instead — no confidence threshold on the underlying scoring formula reliably separates correct fuzzy merges from incorrect ones, so an irreversible auto-merge was judged unsafe as a default.

A same-name group spread over separate namespaces is not a conflict: the database (Table/View/Procedure/Trigger) and each language’s code (Perl, JavaScript, Python, Rust, Elixir modules and symbols) name things after each other all the time — a Perl package LedgerSMB::GL beside a table gl. resolve accepts such a group when each namespace’s share is unremarkable on its own; a Table beside a same-named View still conflicts. LedgerSMB with sql/, lib/ and UI/ observed: 16 → 0 conflicts.RFC 0147 (b)

When sources disagree, both claims are kept. A table created in two files with different columns, a column type an exact-name merge would have dropped, two labels for one code from different sources: each becomes a ConflictingEvidence item — every claim with its file and line, what EKOS kept, and a Disputes link to the object — instead of the last one silently winning. Values are normalized first (INT/INTEGER/SERIAL are one type), a fact only one side states is unknown rather than a disagreement, and prose is never compared. ekos conflicts list|show|resolve; resolving (--pick N or --both-valid --note) is human-only, holds while the claims are unchanged and reopens when one changes. Agents read them through ekos_conflicts and open_conflicts on ekos_state. Answers say so too: an ekos_query / ekos_retrieve claim about an attribute with an open conflict carries disputed: true and the conflict ids, because its value is only the side EKOS kept, and ekos ask tells the model the same in the evidence it reads (RFC 0173). A label conflict is also linked to the table its column belongs to, so a question about the table surfaces it. On LedgerSMB: 3 — user_preference, re-created by a migration, and 2 order-class labels where an API query names the counterparty (customer) rather than the order type. A CASE result counts as a code’s label only when it depends on that code alone and does not map the column back into its own codes, so a sign flip like 'A'→'L' is no longer read as a meaning. In EKOS Migrate, ekos migrate assess also checks documentation against the data: where a column comment, or a sentence in a Markdown or Confluence page that names exactly one table.column in code formatting, makes a checkable claim (“never null”, “unique”, “one of …”, an A=asset,L=liability legend, “between 0 and 100”, “positive”), it counts the contradicting rows — counts only — records a DQ.CONSIST.DOC finding, and a contradicted claim becomes a doc_vs_data conflict with the comment’s path:line on one side and the measured count and query on the other.RFC 0172 / 0158

Broken references across systems. ekos check refs resolves every name a routine, a view, a Kettle step, SQL built in a Java method or a dbt source() points at against every compiled relation (DDL, live catalog, application model), by the rule lineage links by. A miss is an UnresolvedReference, several matches an AmbiguousReference: with the file and line, the closest names, the evidence, a confidence and a fingerprint that survives recompiles. Relations a routine creates at run time (CREATE TEMP TABLE, EXECUTE 'CREATE VIEW …'), table functions and system catalogs are not findings; a name in a schema EKOS compiled nothing from is a coverage gap. --scope cross-system|code|all, --min-confidence, --format table|json|sarif (SARIF 2.1.0 for code scanning), --fail-on error|warning and --baseline <time | report.json> for CI on legacy code. ekos commit writes the report to <state>/integrity/current.json, never to the ledger; [xref] sets the confidence floor and per-kind severities, and xref-ignore.toml suppressions each need a reason. Migration history counts too: a table the scripts alter but no compiled DDL defines is a coverage gap, one they drop is an error naming the script, and a migration script's own references are history. On LedgerSMB: 2 errors in 1,366 references, both real (a routine deleting from two tables no file creates); on iDempiere's code without its database, 0 errors and 609 info findings in 11,966 references.RFC 0233

Calls, imports, cycles and dead code. The same check covers references inside one language: a SQL routine calling a function no file defines and no engine provides (per-dialect built-in lists; dbms_* and pg_catalog are built in); a Perl use or Elixir alias of a package under the workspace's own namespace that no file declares; a relative JavaScript import with no file behind it; a Python import under an in-tree package. An import into a directory ignore-patterns excludes is a coverage gap naming the file. Module dependency cycles (one finding per cycle) and tables nothing uses are reported too. Reachability runs through a per-language contract: SQL routines reached from views, triggers, Java SQL text or [xref] entry-points are live; an unreached one is UnreachableSymbol, or DynamicReferenceUnknown when code that may call it by name exists, and the dynamic share is always shown beside the coverage. Languages whose analyzer records no call graph are reported as unsupported, with the reason. --reachability lists those findings and --from-index serves the last commit's report. Agents call ekos_xref_check (read-only; name checks one table or routine name before they write SQL). The web console has an Integrity tab and dashboard tiles; the estate docs' Findings page gains a Reference integrity section. ekos check explain asks the configured model for the likely target or caller and keeps the answer only when it names and cites an object it was shown; suggestions are printed, never stored, and a model that is not a local Ollama one needs --allow-metered. On LedgerSMB: 4 errors, all real (two missing tables, a misspelled function call, a call to a function that exists only under another name), and 95.7 % of routines dynamic, so no dead-code claim.RFC 0233

04b

DAO treasury compliance

Answers "was this payment approved by governance?" with evidence, never a verdict. EKOS_TREASURY_ADDRESS + EKOS_TREASURY_CHAIN_ID observe a treasury's outgoing transfers via an Etherscan-family explorer (every sub-transfer of a multisig batch is its own payment), read in pages so a history past the explorer’s 10,000-row cap is complete, and refused outright if it cannot be read whole; EKOS_SNAPSHOT_SPACE observes a Snapshot space's proposals and outcomes. ekos treasury scan scores payment↔proposal candidates on recipient address, amount, text reference and timing, writes each as an unconfirmed AuthorizedBy relationship with its evidence, and lists payments with no candidate. A payment made before its approval is flagged and heavily penalised; a rejected proposal is never offered as an approval; nothing is auto-confirmed — a person reviews each candidate with ekos treasury confirm|reject (human-only: no MCP tool can confirm a payment authorization). Verified against mock clients and a full-pipeline fixture. The Snapshot client is additionally live-verified against hub.snapshot.org — which surfaced that a renamed or misspelled space id returns an empty proposal list with HTTP 200 and no error, so both observers now warn when they observe nothing rather than let every payment silently read as unapproved. The real explorer client has still not been run live, and the scoring weights are untuned.RFC 0032

04c

Reading while another process writes

A long-lived reader — in practice ekos mcp serve, held open for a whole agent session — picks up commits made by a separate build → recover → resolve → compile → commit process without locking, restarting, or reopening the store. refresh_snapshot() costs one HEAD read plus one stat when nothing has changed, and decodes only the newly appended tail when the active segment has grown; the search index moves forward through a reader-side tantivy reload() that never touches the writer lock. A seal, a manifest or dictionary change, or a shrunken file still falls back to a full cold reopen, because those are not pure appends.RFC 0112

Measured on 5,000 objects: 27.7 µs when nothing changed, against 103.9 µs for the previous whole-store walkdir fingerprint; 109.5 µs after one appended batch, against 832 µs for the cold reopen it replaces. This applies to the fact-engine store; SQLite readers already get WAL isolation natively, and partitioned/distributed stores keep the fingerprint check.

05

Storage & the ledger

Four backends, each a drop-in KnowledgeStore — every command, the MCP server, and docs generate work unchanged regardless of which one a workspace uses.

SQLite (legacy)

The original backend. Any pre-existing SQLite workspace keeps serving from SQLite forever unless explicitly migrated — never switched implicitly.

Fact-segment engine — default for new workspaces

EAV facts, immutable signature-verified segments, tantivy full-text search, mmap'd reads. A real cross-process write lock (fs4), self-healing search index, ledger repair, and version-chain checkpoints for fast point-in-time reads.RFC 0016 / 0103–0106

Partitioned storage — opt-in

Splits data across many independent fact-segment ledgers keyed by kind + time bucket, with a persisted catalog and run-file index so a reopened store resolves any object/relationship with zero partition scan; aged partitions tier to cold. A drop-in for the single-ledger backend — ekos status reports it with real counts, and retrieval fuses each partition's ranked hits plus a cross-partition exact-name arm (RFC 0120) so an exact-name query still wins across the partition boundary.RFC 0111 Phase A

Distributed (multi-machine) — opt-in

A coordinator hands out fencing-tokened write leases with a TTL the compile worker's heartbeat adapts to; ekos compile-worker run [--force] [--retry-lease-seconds N] runs the real pipeline under lease and publishes each partition — including its active (unsealed) segment — to object storage (S3 / Azure / any S3-compatible store such as MinIO; credentials from the standard AWS_* / AZURE_* env vars), so a query worker serves it from the URL alone. --retry-lease-seconds (0 default, unchanged fail-fast behavior) keeps retrying every 3s if the shard's lease is already held by another worker, instead of failing on the first attempt — it only ever retries that one "already leased" conflict, never a genuinely failed run. Query workers + a pooled, concurrent, pruned DistributedLedger gateway fan reads across the cluster, merge them (distributed BM25 search included), and fail over to another worker when one is unreachable. ekos coordinator status shows the committed generation per shard. Feature-complete at v1 scope.RFC 0113 Phase B

ekos ledger status --storage (or the shorter top-level ekos status --storage, RFC 0116), ekos ledger migrate, and ekos artifact repack shrink an existing workspace in place (both verify before touching anything and leave backups). ekos status --json (RFC 0127) emits the same information as one machine-readable object — entry/object/relationship/evidence counts, backend tag, per-component storage breakdown, mtime-proxy last_write — with the text output left unchanged. ekos doctor --json (RFC 0129) does the same for the environment checklist — where an absent optional thing (no LLM provider configured, no Rust toolchain on PATH) is reported as a fact, not a failure, so a prebuilt binary or the console's own container never reports a healthy workspace as broken — and ekos ledger timeline --bucket day|week|month (RFC 0129) emits cumulative object/relationship counts bucketed by mint time — the growth series behind the console dashboard. ekos ledger audit <id> [--json] (RFC 0135) shows the write history of one object/relationship with the pipeline run id, stage, and source artifact behind each version — ekos build/ekos commit now stamp every write with that provenance (SQLite: three nullable entries columns; fact engine: a provenance.jsonl sidecar; both additive, no format bump); for a committed object/relationship the source artifact is its own recovered KnowledgeArtifact id(s) (a merged identity carries all of them), falling back to the run's CKM content hash only for objects compile synthesizes itself, like rollups and concentration risks. ekos config validate [--json] (RFC 0130) parses ekos.toml and flags [observe] mistakes — most usefully an ignore-patterns entry like *.lock that looks like a glob but is matched as a bare file or directory name and so matches nothing (a whole file name such as mysql-data.sql does match: every walk, observers and recover alike, skips a file of that name); ekos config preview-scan [--json] counts what ekos build would observe (files, by extension, directories each ignore pattern pruned) without reading anything. Every --json and export command routes its logs to stderr so stdout stays a clean document.

ekos graph export (RFC 0127) writes the whole compiled graph — every object and relationship — as one JSON or NDJSON document: the first non-per-object, non-LIMIT-capped read path in EKOS. Object/relationship-kind and minimum-degree filters, an optional --level aggregate that collapses to one super-node per kind or path prefix, and degree-descending truncation that is reported in a truncated block rather than silently returning a prefix. --as-of <rfc3339> reconstructs the graph as it stood at a past instant and --first-seen stamps every node/edge with when it entered the ledger (RFC 0134). Deterministic modulo its generated_at timestamp; the ekos_graph_export MCP tool exposes the same function to agents.

Web console (web/) — a browser surface over one or more compiled workspaces. A FastAPI app (web/api/) plus a Vite + React dashboard (web/ui/). Phase 1 (RFC 0129): a persisted SQLite workspace registry; an McpSupervisor that spawns and restarts one ekos mcp serve --tcp per workspace on its own (fresh random bearer token per process, readiness probe, exponential-backoff restart); and a statistics dashboard — entry/object/relationship/evidence counts, storage breakdown, objects by kind, a cumulative growth timeline, query-log stats, and doctor status. Phase 2 (RFC 0130) adds an ekos.toml editor: GET/PUT /config, POST /config/validate, POST /config/preview-scan. PUT runs ekos config validate, writes an ekos.toml.bak, and warns when you narrow [observe] that already-compiled data stays in the append-only ledger. Phase 3 (RFC 0131) runs EKOS pipeline commands from the browser: a hardcoded command allowlist (build/recover/resolve/compile/commit, a chained pipeline, doctor, ekl, ledger repair, docs generate, …), a per-workspace job queue with SIGTERM→SIGKILL cancellation and per-stage status, and an SSE log tail. That first browser mutation brings the read/write role split: auth is OIDC (Authorization Code + PKCE, a claim → the write role) or, when OIDC_ISSUER is unset, two static tokens — CONSOLE_TOKEN (read) and CONSOLE_WRITE_TOKEN (read + write). Both modes end in a signed session cookie. Phase 4 (RFC 0132) adds scheduled runs: a Schedule (workspace + command + cron/interval trigger + a required notify_url) fires the same job runner; APScheduler is rebuilt from the SQLite table on start, and a non-succeeded run POSTs a JSON body to the notify_url. Phase 5 (RFC 0133) is the graph view (2D react-force-graph, lazy-loaded): an overview as one super-node per object kind, click-to-expand one kind into its 500 most-connected objects, kind/relationship-kind filters, search-with-fly-to, and an object panel showing ekos_state — properties, relationships, and one evidence row per claim (path · analyzer · confidence · fragment). Phase 6 (RFC 0134) is a timelapse slider under the graph: drag it back and the diagram redraws as the knowledge existed at that instant, and a clicked node's panel shows the evidence it had then. Backed by ekos graph export --as-of <rfc3339> --first-seen (and the same as_of / include_first_seen on ekos_graph_export); the browser fetches the latest graph once with per-element first-seen stamps and filters it client-side against a frozen layout, so scrubbing never refetches and nodes never move. Ticks come from ekos ledger timeline. The slider is only as deep as the ledger's retained history. Graph v2 (RFC 0136) adds: neighbourhood isolation (an object's real BFS sub-graph — both objects and relationships, not just ids — at a depth of 1-3, via the existing, unmodified ekos_neighborhood tool); impact mode (ekos_impact's hop-distance results rendered as node coloring source-outward, plus highlighted edges between two impacted nodes already on screen — RFC 0127's own "differentiating screen," the visual form of the evidence-chain claim); a server-side ForceAtlas2 layout (Python networkx + fa2_modified, cached per graph structure) for graphs past ~2,000 nodes; and one-click PNG/glTF export of the current view, entirely client-side. Zero Rust-side changes were needed for any of it — both MCP tools already existed. Phase 7 (hardening) closed out the console's last named phase: every route now code-splits on its own (react-router-dom's per-route lazy), shrinking the production build's main chunk and isolating the dashboard's charting dependency to its own chunk, fetched only when that route loads; the graph view supports deep links (?as_of=&focus=, plus a one-click "copy link" that shares the current time-travel position and selection); and docker-compose.yml dropped configuration dead since Phase 1. Distributed workspaces (RFC 0113) also gained a real evidence_count RPC alongside this pass — ekos status/ekos_status report a real evidence count there now instead of null. An Evals tab (RFC 0138) reads every saved ekos eval run report straight from evals/reports/*.json — a history table plus a full per-scenario detail view (all eleven metrics: the five headline scores plus tokens/latency/cache-hit/RSS/CPU) — with a new run triggerable from the existing Run tab (eval-run added to the same command allowlist every other pipeline command uses, so it needed no bespoke trigger UI). A Semantics tab (RFC 0170) is the human review surface for business-semantics hypotheses: a review queue with each item’s evidence lines, links and rationale commits, Confirm / Reject / Edit, and bulk confirm/reject (all or nothing); a gaps-and-conflicts view; and a LinkML schema viewer and YAML editor — edits round-trip through ekos import linkml (Validate shows the decisions they stand for, Apply records them all or nothing; a side-by-side diff shows the edits against the current export). A Conflicts tab (RFC 0174) lists where sources disagree about one fact, with every claim’s file and line; a write-role user picks the right claim or marks both valid, through the human-only ekos conflicts resolve. The graph draws a ring around every object with an open conflict. Decisions need the write role and run the human-only CLI, never MCP, attributed to the OIDC user or a typed token:<name>. Reloading any client-side route now serves the app instead of a 404.RFC 0127 / 0128 / 0129 / 0130 / 0131 / 0132 / 0133 / 0134 / 0136 / 0138 / 0170

Is the ledger still current? The ledger describes the source as it was at the last ekos commit. ekos build records every observed file’s size and modification time, in the same walk that computes its cache fingerprint; a successful commit promotes that list to “what the ledger reflects”. ekos status and ekos doctor then say whether the source has moved since (and the git commit it was compiled at); ekos freshness lists the changed, added and removed files and, for each, the compiled objects whose evidence cites it — “these facts may be stale”. ekos compile/commit warn FRESH001 when the source moved after build; over MCP, ekos_status carries the same report and read tools add a one-line note while the ledger is behind. Metadata only — no file is read, so no new raw-content path; a file that was only touched counts as changed, and the output says so. With [freshness] content-hash = true (off by default) ekos build also records a content hash per file, and a file whose time moved but whose bytes did not is no longer reported; files excluded as secrets are never hashed. [freshness] enabled = false turns it off.RFC 0171

06

Documentation generation

Renders the compiled ledger straight into Markdown/HTML — zero LLM calls, every claim traceable to real compiled evidence, unless --prose is explicitly added.RFC 0035 / 0037 / 0042 / 0090 / 0094 / 0095

--layout objects (default)

One page per significant compiled object, plus an index and an entity-relationship diagram.

--layout curated

README.md/Architecture.md/API.md/SequenceDiagrams.md — the shape a developer actually expects, with a real parsed crate/workspace dependency graph, CI/CD pipelines, an Architecture Confidence score, and Concentration-Risk findings.

--layout solution-architect

DependencyRiskReport.md/OnboardingGuide.md/FindingsMemo.md — a team-handoff bundle: declared-version risk, a first-day repository map, and an actionable findings list.

--prose (opt-in) layers an LLM-written overview onto each object page or the findings memo's executive summary, reusing ekos ask's exact grounding+citation pipeline, with a token-cost estimate shown before any call — currently wired for --layout objects and --layout solution-architect only; combined with --layout curated it fails with a clear error rather than silently doing nothing (a real bug found and fixed live during this project's own self-hosted benchmark — see §15).

07

Architecture reasoning & investigation loop

ekos architecture investigate runs an agentic loop: broad collection, deterministic crate-topology extraction, one batched LLM call classifying each crate's architectural role, a deterministic evaluator scoring completeness and evidence coverage (no LLM), and a targeted second pass for any crate still unclassified — stopping early once a quality threshold is met, always ending with a curated-docs run.RFC 0065 / 0066 / 0067

Because an LLM-classified role is a judgment call, not a deterministic fact, two follow-ons treat it accordingly: ekos architecture diff --since <t> does a real id-set comparison between two points in time (not a fuzzy match), and ekos architecture review lets a human confirm or reject a pending classification — both also exposed over MCP.RFC 0107 / 0108 / 0109

08

LLM-backed compile-time descriptions

Unlike --prose above (render-time, re-spent every call), [llm-description] runs at commit time and persists real, evidence-grounded ai_overview/ai_usage properties straight into the ledger — queryable through EKL/ekos ask/MCP like any other compiled fact, not just rendered once. An ai_comment_check property flags a real discrepancy between an existing doc comment and what the code actually does (consistent/stale/incomplete) — never silently trusted or overwritten. Opt-in and cost-gated: a real call-count estimate is shown, and confirmation is required before any spend.RFC 0088

09

Hierarchical rollups

Every other context-saving mechanism in EKOS is retrieval-limiting — fewer raw facts, never a synthesized higher-level one. ekos commit also synthesizes deterministic, zero-LLM Rollup objects — one per directory subtree (crate-level by default) or per project in a multi-project estate — each carrying real member counts, a kind breakdown, and boundary-relationship counts, linked to every member via the same Contains relationship everything else uses. An agent asking about a whole subsystem gets one condensed, evidence-linked object instead of personally synthesizing meaning from dozens of raw facts.RFC 0044

10

AI agent access (MCP)

The single read-only surface every AI agent — Claude Code among them — is meant to use instead of touching raw enterprise systems directly. ekos mcp serve --workspace <dir> exposes the Runtime over stdio (RFC 0013), with a per-server-session cached read-only ledger handle that never blocks a concurrent writer.RFC 0097

--tcp 127.0.0.1:7331 additionally serves the same protocol over plain NDJSON-over-TCP, so more than one MCP-speaking tool (PyCharm's AI chat, another agent host, a second Claude Code session) can connect to one already-running server instead of each spawning its own process. One OS thread per connection, each with its own independent cache — not shared across connections, so N concurrent clients means N independent ledger opens, not one reused handle. Stdio stays the default and is unaffected when --tcp is omitted. No TLS — loopback or a trusted network only.RFC 0115

--http 127.0.0.1:7331 instead serves MCP's Streamable HTTP transport at one endpoint, POST /mcp — for the many clients (VS Code / Copilot agent mode, Visual Studio 2022, ChatGPT Developer Mode, mcp-remote) that only take a URL, not a spawn command or a raw socket. The POST response is application/json, or a single-shot text/event-stream when the client's Accept header asks for it (ChatGPT's connector requires the SSE form) — same JSON-RPC bytes either way. No server push: GET /mcp is 405, no sessions. --http and --tcp are mutually exclusive; each replaces stdio. One worker thread serializes all HTTP requests (a slow tools/call blocks the next, same as the stdio loop), keeping the RFC 0097/0114 caches alive across requests. The Origin header, when present, must be loopback or an explicit --http-allow-origin (DNS-rebinding defence); a request without one is allowed. ChatGPT is cloud-hosted and needs a public HTTPS tunnel in front; its Deep Research connector (fixed search/fetch tool names) is incompatible — only Developer Mode is.RFC 0143

--token-file <path> (or the EKOS_MCP_TOKEN env var; --tcp-token-file is a back-compat alias) adds an optional bearer token to --tcp and --http. Over TCP, every connection's first message must be an initialize carrying a matching params._meta.token; over HTTP it is an Authorization: Bearer header on every request. Checked in constant time; a mismatch returns -32001 unauthorized (401 over HTTP) before any tool is reachable. Token-less network transports are unchanged; stdio is never gated. A plaintext bearer token — defence against a casual local process, not a wire attacker.RFC 0128 / 0143

Limits on every transport. Message lines are capped at 4 MB, and --tcp serves at most 64 connections; both hold before any token is read. A request that panics gets -32603 and the server keeps serving (before, a panic killed stdio and wedged --http's single worker for good). --http queues at most 64 waiting requests, then answers 503 with Retry-After. Out-of-range depth, max_hops and max_objects are refused with the limit named, never silently clamped. An argument name a tool does not declare is refused with the accepted names listed, never silently ignored.

ekos_search
Keyword search over compiled objects (limit, RFC 0124); mode lexical/vector/hybrid adds semantic matching and returns arms_run (RFC 0125).
ekos_query
Compiled fact + graph answer to a natural-language question — a typed list of source-cited claims, no LLM (RFC 0124). A claim about an attribute sources disagree on is marked disputed: true with its conflict ids (RFC 0173).
ekos_retrieve
"Show your work": the compiled query plan, the evidence set, and the query understanding — no LLM (RFC 0124).
ekos_ekl
Run an Enterprise Knowledge Language query — see §11.
ekos_neighborhood
BFS graph traversal from an object. depth 0–3; stops at max_objects (default 500) and reports truncated: true.
ekos_state
Reconstruct an object's full state — object, relationships, evidence.
ekos_dependents
Single-hop impact analysis. Each entry carries the other end’s status; an unconfirmed claim at the other end (a business-semantics hypothesis, a migration proposal) is not listed as a dependency fact until a person confirms it.
ekos_impact
Directed, kind-filtered, multi-hop impact tracing. Like ekos_neighborhood, it does not step onto an unconfirmed claim.
ekos_graph_export
The whole compiled graph as nodes + edges in one call — kind/degree filters, optional super-node collapse, degree-descending truncation reported in the payload (RFC 0127).
ekos_diff
Raw ledger-entry changes since a timestamp.
ekos_audit
The write history of one object/relationship — every version with the pipeline run id, stage (build / commit / commit:rollup / …), and source artifact behind each write (RFC 0135). The source artifact is the object's own recovered KnowledgeArtifact id(s) where tracked, else the run's CKM hash. Read-only.
ekos_status
Ledger health: entries, object/relationship counts, path — plus a freshness block: does the ledger still match the source (RFC 0171)? When it does not, every read tool’s answer carries one extra note saying so.
ekos_transformation_explain / _diff
Explain or diff a Transformation IR pipeline.
ekos_binary_migration_check
Compares a Python rewrite (passed as text) with the compiled type it replaces, per method: I/O kinds, SQL tables, messages and constants, calls, exceptions, complexity. Evidence, not a score. Ships with the separately licensed binary extension.
ekos_binary_explain
Business logic recovered from a compiled .NET assembly or JVM class: members, per-method fidelity and migration readiness, complexity, constants and external I/O boundaries; with method, the full per-method spec (RFC 0150); plus any LLM-reconstructed rules with their confidence — kept in separate, separately-attributed sections, with the provenance chain back to a bytecode offset and the binary's SHA-256.
ekos_architecture_evaluate / _drift / _diff
Completeness/evidence-coverage score, documentation drift, and a real architecture-level diff over time.
ekos_semantics_lookup / _gaps
Listed when [semantics] is on (RFC 0170). Lookup: the recovered business meaning of a term, table, column or code value — definition, evidence lines, review status spelled out (hypothesis / confirmed / needs_review), rejected definitions left out, no_semantics_found when nothing matches. Gaps: open questions and conflicts. Read-only — no MCP tool can confirm or reject.
ekos_signoff_status
The seven-step workflow bar and whether each review subject is signed off — valid, stale, revoked or none (RFC 0213). Read-only: only a person signs off.
ekos_conflicts
Where sources disagree about one fact — every claim with its file and line, what EKOS kept, and whether a person has decided (RFC 0172). Read-only: an open conflict is to be reported as a disagreement, never one side as fact.
ekos_identity_review
Write-capable: confirm/reject a candidate identity match (SameAs). A treasury payment authorization is not reviewable here: that is ekos treasury confirm|reject, for a person. Optional reviewer and note are recorded, the note redacted, on the evidence the decision event cites.
ekos_architecture_review
Write-capable: confirm/reject an LLM-classified crate role claim. Takes the same optional reviewer / note.
ekos_session_note
Opt-in ([session-memory], RFC 0151), inbox-only: redacts, caps and appends a note to a local file — never opens the ledger, never a confirmed fact.
ekos_session_recall / _brief
Opt-in, read-only: recall and a token-budgeted, untrusted-envelope brief of earlier sessions' notes, each with its tier (T0/T1) and a staleness verdict (fresh / changed / orphaned / unanchored). Explicit "no relevant session memory" when nothing matches. These two tools are the only way a note is ever seen: every other read path — search, query/retrieve, ekos_ekl, the graph tools, ekos_state and ekos_graph_export — filters session claims out, so an unconfirmed note never reaches an answer dressed as a compiled fact, and never counts as a dependent in impact analysis. Commands: ekos session note|status|commit|recall|brief|review|capture|extract|purge|eval; only review (CLI) can promote a note.
ekos_clickhouse_query
Gated, off by default — live NL-to-SQL. See §14. The one non-review tool that writes: each live query is recorded as evidence + an event (the same query with the same answer is one event).

Every read tool (and ekos ekl run from the CLI) appends one line to .ekos/query-log.jsonl; a static heuristic classifies each call cheap/expensive from its own arguments and opportunistically caches an expensive one's result for an identical repeat while the workspace hasn't changed underneath it.RFC 0114

ekos ask "<question>" is the LLM-backed CLI counterpart. It now compiles the question — a rules planner routes it to fact lookups and graph traversals, executes them into a flat set of atomic, source-cited claims, and the model's job shrinks to "explain this evidence, cite each item." --explain prints the plan and the evidence set alongside the answer; --classic (implied by --stream) selects the older retrieve-and-dump-JSON path.RFC 0118 / 0123 / 0124

Claims now cite a place in a file, not just a file. Analyzers recorded a source_span for every Rust/Python/Elixir symbol, but never attached a file to it — those analyzers emitted no evidence at all, so a symbol knew it lived at lines 200-322 of an unnamed file, and measured over a full eval run 0 of 1,289 rendered claims carried a line number while only 26% carried any location. Each span-carrying symbol now emits real evidence naming its path, its line, and the actual source text (capped at 40 lines; the span still records the true range), and ekos ask --explain renders each claim as path:start-end. Asking what a function does now shows the model the function body rather than only its name. Persisting the fragment is safe precisely because it comes from the analyzer's own observed source, which has already passed RFC 0043 redaction — which is also why reading source at query time must come from the content-addressed artifact store, never the live filesystem: a disk read at query time would be a new raw-content entry point that redaction never saw.RFC 0140

Every Rust/Python/Elixir function or method symbol now also carries a real signature — real declaration text sliced from the actual source, not re-synthesized — because a discriminating term often lives in a return type the symbol's own name never mentions: asked "what builds the LlmProvider used by ekos ask," retrieval used to rank the type LlmProvider above the function build_llm_provider(...) -> Arc<dyn LlmProvider> that actually answers it, because nothing recorded the function's own interface. A Rust Calls edge (the only analyzer that builds one — Elixir/JavaScript recovery is deliberately not a call graph) now also carries call_count, the first (minimum, order-independent) call_site_line, and caller_is_test, so ekos_impact can eventually weigh a mostly-test dependent set differently from a production one instead of returning an undifferentiated list.RFC 0141

For the top few entities named in each answer's assembled evidence set, ekos ask reads the entity's real, on-demand source text straight from the content-addressed artifact store — never the live filesystem, so it can never bypass RFC 0043 redaction — via the audit trail's recorded source_artifact_id, and attaches it as one extra evidence item, capped at a generous 400 lines rather than the 40-line cap every symbol gets at compile time. Best-effort throughout: an entity with no real code span, no recorded source artifact, or an unreadable artifact is silently skipped, never a hard error. Separately, an opt-in [retrieval] rerank = "llm" (off by default) asks the model to name and order the evidence items that actually help answer the question before the real answer call — reordering only, never dropping an item — and lives entirely inside the REASON answer pipeline itself, never on the deterministic ranking path RFC 0126's CI gate checks, so turning it on can never affect that gate's reproducibility guarantee. Any failure to call the model or parse its response leaves the evidence set completely untouched.RFC 0140

Retrieval is lexical by default (BM25 + exact-name) — object names and content are indexed with English stemming (a singular mention like "the customer table" resolves the same real object a plural exact mention like "the Customers table" does). The opt-in [embeddings] table in ekos.toml (provider ollama/openai/mock) adds a vector arm: ekos commit embeds every compiled object into a derived, rebuildable <ledger-dir>/vectors/ index — a code symbol from its name, signature, and doc-comment description (RFC 0141), everything else from its name, kind, and ai_overview prose if present — and ekos query find --mode vector|hybrid (or ekos_search {mode}) fuses cosine-nearest hits with the lexical arms via the same Reciprocal Rank Fusion as RFC 0120. So "the thing that sends welcome emails" finds a function named dispatch_signup_notification. Off by default (no table → nothing embedded, byte-identical lexical path); embeddings are cheap and disk-cached so there is no spend prompt; single-node this phase, and a vector search with no index yet degrades to lexical with a visible note.RFC 0125

Retrieval quality is CI-gated. ekos_runtime::retrieval_eval holds a checked-in graded query set (~30 queries across all five query shapes) and a hand-built reference estate; a normal workspace test fails the build if Recall@10 / MRR / nDCG@10 or intent-classification accuracy falls more than 2% below the recorded baseline, and cargo bench --bench retrieval_eval prints the current scoreboard. Every fused retrieve also carries arm_timings — per-arm wall-clock and candidate counts — surfaced on the ekos_search / ekos_retrieve results, written into .ekos/query-log.jsonl, and shown by ekos query find --explain.RFC 0126

11

Enterprise Knowledge Language (EKL)

A small purpose-built query language over the compiled ledger — ekos ekl "FIND Object WHERE kind = 'Table' AND name CONTAINS 'order' ORDER BY name LIMIT 10" — with point-in-time AS OF <timestamp> queries, COUNT/GROUP BY aggregation, and SEMANTIC 'text' [LIMIT k] — start from a ranked retrieval candidate set instead of a full scan, then filter/project/order it like any other FIND Object (mutually exclusive with FROM / AS OF / COUNT). Every result row carries the evidence it was derived from, the same as any other read path. A field that is not a built-in column (id, name, kind, evidence, created_at) is read from the object’s or relationship’s properties, so WHERE status = 'confirmed', COUNT GROUP BY status and a nested stats.rows > 10 work. A field no candidate has is an unknown field error that names the built-in columns; it is never an empty result.RFC 0010 / 0096 / 0124

12

Eval harness

ekos eval run --dataset ekos-full grades whole ekos ask answers against a checked-in, 101-scenario suite across seven categories (evals/datasets/*.yaml — Architecture, Code, Dependencies, Lineage, History, Security, Adversarial), distinct from RFC 0126's narrower, CI-gated, LLM-free retrieval-ranking regression check. Every score is deterministic keyword/id matching, no LLM judge: did the answer state the expected facts, cite real (non-hallucinated) evidence, cover what was asked, surface the right objects in the top-10 (recall@10, reusing RFC 0126's rank-metric math), route to the right REASON planner query type, and — for the Adversarial category, the harness's strictest — actually say "Insufficient evidence" for a question with no grounded answer instead of inventing one. Prints a report (Answer correctness / Evidence groundedness / Completeness / Recall@10 / Hallucination rate, plus tokens spent and genuinely cache-attributed tokens saved, plus best-effort Linux CPU-time/peak-RSS readings, plus a PASS-FAIL gate), saves a timestamped JSON copy to evals/reports/, and supports --agent claude|ollama|openai to A/B a different provider than ekos.toml's configured one for one run. ekos eval history reads every saved report back as a trend table. Deliberately not wired into CI — real LLM calls against a real workspace, not yet fast/free/deterministic enough for every PR.RFC 0138

Recall@10 is graded against the exact keyword-only query reason::plan() actually searches with, not the raw natural-language question a reason/ask scenario asks — the two can rank differently, so grading the wrong one could pass or fail a scenario for a reason unrelated to what the model was shown. Lexical search itself also treats a bareword and/or as connector noise rather than a literal word a document must contain, matching how the same two words are already treated as English stopwords everywhere else in this codebase — so an "A or B" style query can match on either real term without silently requiring the word "or" itself to appear.RFC 0139

ekos eval run refuses outright, rather than silently grading a run, when the configured provider's API key isn't set and provider selection falls back to the stub MockLlmProvider — there is no --agent mock option, so reaching the mock can only mean the fallback fired, never a deliberate choice, and running anyway would produce a fully-formed, publishable report scoring canned answers no model actually generated. The error names the actual environment variable to set (an openai-configured workspace now defaults to checking OPENAI_API_KEY rather than ANTHROPIC_API_KEY when [llm] api-key-env isn't set explicitly — a real, separate bug this closes) and suggests --agent ollama as a local alternative. A local model is real and free to run but is not the reference baseline: it needs a genuinely powerful server to keep pace with a 101-scenario suite, so a published reference number should come from --agent claude/--agent openai against a cloud model.RFC 0138

A real recover/resolve/compile/commit rebuild plus a full ekos eval run against EKOS's own workspace measured a round of retrieval/signature/rerank fixes on 2026-09-14: the two headline scores landed bit-identical to the prior baseline — a real confirmation, not a null result, since [llm]'s mandatory temperature: 0 means an unchanged prompt can't produce a different local completion. The one metric that moved (recall@10) traced to a single scenario whose score corrected from a false perfect result to an honest zero — the recall-grading fix (above) working as designed. Independently verified outside the eval score entirely: a real function that previously ranked outside the top ten for a plain keyword search now ranks third, because it carries a real signature property it didn't have before.RFC 0139 / 0141

A follow-up Phase 1 pass (2026-09-15) found the suite's biggest remaining lever wasn't a retrieval-ranking or routing fix but three cheaper bugs, all in production code every ekos ask call runs, not just the harness: Search/Graph claims that rendered only an object's name and discarded its retrieved excerpt, a citation-block parser that couldn't resolve a bare claim position or an "evidence <id>"-wrapped uuid, and a refusal-wording mismatch where the model's own echoed prompt header wasn't recognized as a refusal — plus three grading-ruler defects that would otherwise have inflated the measured gain. Real result against the 20260914T154459Z baseline: 42/101 → 53/101, groundedness 51.6% → 65.9%, recall@10 47.1% → 64.7%, hallucination count 8 → 7. Two side effects are reported rather than hidden: richer evidence text made the model fabricate on 4 adversarial questions it had previously refused correctly, exactly offset by 4 different adversarial scenarios the wording fix corrected (net zero for that category, but a real trade-off); and the prompt-header restyling coincided with 3 scenarios where the model wrote a citation in prose instead of the required JSON block — history's only net regression.

Cloud models and context windows (2026-09-15). [llm] base-url points the openai provider at any OpenAI-compatible host (OpenCode Zen, OpenRouter, DeepSeek, Groq, vLLM), and [llm] model is now honoured there. For local Ollama, [llm] context-window (default 8192) is sent as num_ctx — before, Ollama silently truncated the front of any long prompt — with a warning when a prompt fills the window. Measured on the same model (deepseek-v4-flash via Zen) against the ledger before and after RFC 0144's document structure: 70/101 → 79/101 on a clean full run (84 in an earlier, mostly-cached composite), answer correctness 61% → 78%, groundedness 78% → 87%, for about $0.08 per full run. Reasoning models such as DeepSeek V4 Flash need [ai] max-tokens = 8192: their hidden reasoning counts against the limit and otherwise empties some answers. The same measurement found that every ekos commit since 2026-09-10 had left the full-text index stale for read-only readers (fixed).RFC 0144 / 0145

Planner routing and entity resolution (2026-09-15). A dependency phrase now counts only when it opens the question (“what depends on X”), not when it sits in a descriptive clause (“the function … used by …”), which had sent such questions to a graph walk instead of a search. Qualified paths like ekos_common::redaction resolve as mentions (matching the whole path:: family), generic nouns and question verbs never resolve as entities, fuzzy name matching needs comparable lengths, and a Lookup brings a small search along so a stub object is still citable. Same model: 79 → 87/101, groundedness 86.8% → 95.6%.

Measured results, cloud DeepSeek V4 Flash via OpenCode Zen (2026-09). The full ladder on the 101-scenario suite: local llama3 8B 42 (report 20260914T154459Z) → 53 (Phase 1 fixes) → 70 (DeepSeek V4 Flash, same ledger) → 79 (RFC 0144 document structure + max-tokens 8192) → 87 (planner and entity-resolution fixes). Per category, 53 → 87: Architecture 9/20 → 18/20, Code 8/15 → 13/15, Dependencies 10/12 → 10/12, Lineage 5/12 → 10/12, History 3/12 → 11/12, Security 7/12 → 10/12, Adversarial 11/18 → 15/18. Answer correctness 50.1% → 85.7%, groundedness 65.9% → 95.6%, hallucinated adversarial answers 7 → 3; recall@10 fell 64.7% → 55.9% because documents are now many heading-sections that outrank the crate or file a retrieval-only scenario expects (section-vs-document ranking is a tracked follow-up). A full run costs about $0.05–0.08 with a median 5 s per fresh call, against a local 8B model that hit 79 s p95 and was killed three times for lack of memory on a 15 GB workstation. Numbers were re-verified against the saved reports in evals/reports/zen-base (70), zen-final-8k (79) and zen-planner2 (87).

A model per task (2026-10-17). [llm.tasks.<task>] routes each kind of LLM call — ask, doc-semantics, description, semantics, sql-naming, session-capture, architecture, reconstruct, nl-sql, docs-prose, marketing — to its own model, with fallbacks on rate limits and server errors. provider = "zen" reaches every OpenCode Zen model on its own wire format (Messages for Claude and Qwen Flash, Responses for GPT and Grok, Chat Completions for the rest). Every call that reaches a host is priced and logged, metadata only, to .ekos/llm-usage.jsonl; ekos llm usage totals it, ekos llm routes shows each task's model and price, ekos llm probe shows which models a key can reach. [llm] budget-usd refuses a call whose worst case could cross the budget, and models whose provider may keep requests are refused unless allowed. Measured: claude-haiku-5-5 answered 98/101 eval scenarios for $0.10 a run; for document extraction gpt-6-luna and Haiku parsed 40/40 sections for about $0.03, where the previous default deepseek-v4.1-flash parsed 4/33 — its hidden reasoning used up the pass's fixed 2048-token cap, which is now the route's max-tokens. The same run found the document pass re-extracting every older copy of a document (fixed).RFC 0197

The 14 that still fail, by cause: a correct false-premise rejection not worded as a refusal the grader accepts (adv-004, adv-011, adv-015); the answer never reaching the top of the retrieved evidence (code-002, lin-008, dep-004); a fact not recorded yet — a workspace-inherited Rust edition, struct fields as claims, no caller reaching the evidence (code-004, lin-007, dep-005); retrieval-only scenarios graded on recall@10 (arch-009, hist-007); right in substance but wrong keyword (arch-020, sec-002, sec-010). Reproduce: export OPENCODE_API_KEY=…, then from ekos/ run cargo run -p ekos -- --config ../ekos.zen.toml eval run --dataset ekos-full --save-answers, with [llm] provider = "openai", base-url = "https://opencode.ai/zen/v1", model = "deepseek-v4-flash", api-key-env = "OPENCODE_API_KEY" and [ai] max-tokens = 8192. Any OpenAI-compatible endpoint works, including a self-hosted vLLM or llama.cpp server inside a closed network. Full account: EKOS from 53 to 87; devlog_183 / 185 / 186.

Measuring retrieval without a model. ekos eval run --evidence-only runs retrieval and evidence assembly for every scenario and calls no model. It prints Recall@10 and, for each scenario that states expected facts, whether every fact is in the evidence a model would be shown. It costs nothing, takes seconds, and is the number a retrieval change is measured against. A run that does call a model now counts refusals apart from uncited answers, reports “facts in evidence”, and fails when any scenario errored (a run where every call failed used to print PASS). Documents that quote a scenario’s question word for word are left out of search for the run and counted in the report; --keep-question-echoes searches the corpus as it is.RFC 0177

What the 2026-10-11 follow-up found and fixed. Nine causes, each read out of a saved transcript: a symbol’s doc comment and signature were not searchable; partial matches were ranked by BM25 alone, so a short name holding one word of a question outranked the passage holding most of them; a hit showed the opening of its text, not the passage that matched; the “full source text” lookup attached nothing on a real ledger (0 of 101 scenarios) and now falls back to the source recorded with the symbol; “what depends on module M” answered for one import path only; a file was not found by its bare name; a false-premise rejection had no fixed wording the grader could know (now Premise not supported.); 11 correct refusals were reported as uncited answers; and Recall@10 resolved a name borne by two objects to one of them. Measured without a model on the same ledger: Recall@10 55.9% → 70.6%, scenarios whose expected facts all reach the evidence 52 → 60 of 63. With deepseek-v4.1-flash answering: 88 → 95 of 101 scenarios, answer correctness 86.2% → 95.8%, and no remaining failure attributed to retrieval. Six still fail: two correct denials worded outside the two fixed openings, one real question answered as a false premise, one answer with nothing valid to cite, one whose citations a parser bug dropped (fixed after the run), one keyword mismatch. A 2026-10-12 follow-up tied the two refusal openings to the outcome (a denial in the model’s own words is a refusal and may carry cited context; the premise opening only when the question depends on the unsupported part) and stopped the grader from reading a refusal out of a phrase deep in a full answer: all 20 refusal scenarios pass and the suite is at 98 of 101, against 96 for the earlier prompt under the same grader. An index built by an older version is rebuilt on the next command that writes to the ledger.RFC 0177

13

Marketing agent

ekos marketing publish [devlog] turns a devlog_N.md into a human-approved X (Twitter) release announcement: classifies the devlog's importance, drafts a tweet through the same LlmProvider used elsewhere, validates it, asks for Y/N/E approval, and publishes via a real OAuth 1.0a-signed API call — with a posted-history file preventing the same devlog from ever being posted twice. Off by default; requires explicit Twitter credentials and an explicit opt-in flag. ekos marketing post <file> --confirm sends a hand-written text exactly as written instead — no LLM, up to X Premium's 25,000 characters, a preview without --confirm.RFC 0030

14

World Engine simulation (experimental, opt-in)

Auxiliary tooling built on the same ledger, kept deliberately separate from the compiler pipeline above: multi-agent scenarios with beliefs and goals, deterministic round-based decision-making with seed-reproducible conflict resolution, a VirtualForum (channels, replies, likes, follows, shares), a durable replayable event log, and real document ingestion into a scenario's starting world. ekos simulate scenario.yaml runs it; ekos replay scenario.yaml reads it back, read-only. Writes to a dedicated per-scenario ledger by default — never the real workspace ledger, since the ledger has no delete/tombstone mechanism and simulated agents/events are fictional.RFC 0047–0055

15

ClickHouse connector

Two independent pieces. Compiled metadata: ekos build/recover observe a configured ClickHouse database's live schema over its stock HTTP interface and compile every table into a real, searchable, cross-system identity-resolvable Table object. Live NL-to-SQL (ekos clickhouse ask "…"): the one path in EKOS that intentionally crosses the "AI never touches raw systems directly" invariant — an LLM builds a ClickHouse SELECT, it's parsed and hard-rejected unless it's exactly one read-only statement, then run live, redacted, and returned, with every call recorded as an Evidence/Event pair for audit. The matching ekos_clickhouse_query MCP tool is off by default, gated behind an explicit ekos.toml opt-in.RFC 0056

16

EKOS Migrate (PostgreSQL → ClickHouse, opt-in)

Evidence-backed migration of a PostgreSQL database to an analytical target — positioned as a migration proof system rather than a SQL converter, since DDL conversion is commoditized and evidence is not. Every scope decision, finding, mapping, execution, validation result and human approval is a ledger fact with provenance, and the migration report cites those facts or does not ship. It covers the whole PostgreSQL surface, including views, triggers and PL/pgSQL: every source object is recovered, classified and dispositioned — translated with evidence, or a finding a named human decided on. Nothing is silently dropped or approximated.

Shipped and verified live: a real table migrated PostgreSQL → ClickHouse and validated at V1, V2 and V3. A planted one-row change was caught by V3, with V1 and V2 correctly blind to it. The pipeline is init → discover → profile → assess → map → review/approve → load → validate → report, and every stage writes facts, never free text. Off by default; needs [migrate] enabled = true. Sandboxes are in docker-compose.migrate.yml.

ekos migrate init --name app --source postgres://pg/app --target clickhouse://ch/app --source-secret-env PG_PASSWORD
ekos migrate discover && ekos migrate profile --tier p1 && ekos migrate assess
ekos migrate map --emit out.sql
ekos migrate review --unit public.orders --env staging
ekos migrate approve <REQUEST> --as alice --show-evidence
ekos migrate load --unit public.orders --dry-run
ekos migrate load --unit public.orders && ekos migrate validate --unit public.orders --tier v3
ekos migrate report --out report.md
ekos migrate lower --database lsmb --sql
ekos migrate check-candidate --source open_items --candidate view.sql
ekos migrate v5 --source-sandbox postgres://sbpg/scratch --target-sandbox clickhouse://sbch/scratch
ekos migrate reconstruct --source close_period

No credentials, no row values

A DSN names an alias. [migrate.connections.<alias>] resolves it to host/port/user, and secret-env names the password variable. A connection string carrying a password is refused, not stripped. Profiles hold statistics only, and a column classified as PII gets no bounds and no top-k at any tier. Source sessions are read-only, with timeouts and a replica-lag guard.RFC 0154 / 0157

Discover, profile, drift

Live catalog introspection across 23 object kinds, reconciled against the server's own counts. P0/P1/P2 profiling tiers with EXPLAIN cost estimates and a budget that refuses. Live-vs-repository drift down to the column, compared against the schema EKOS compiled from the repo.RFC 0157

Assess by measurement

13 data-quality and target-compatibility rules, each with a positive and a negative fixture. Findings count affected rows and carry the SQL that measured them. Losing FK or uniqueness enforcement blocks even at warn severity. A key-shaped column that a declared foreign key references from is not treated as a duplicate-key finding. Undeclared foreign keys are inferred from real joins in the compiled code and in pg_stat_statements, then checked by inclusion before anything is claimed.RFC 0158

Map and design

Type mapping with lossiness classes: a narrowing cites the profile that proves it is safe. ClickHouse engine, ORDER BY weighted by real query shapes from pg_stat_statements, a partition guard, and nullability dropped only where the profile proves it. The emitted DDL carries its reasoning as comments.RFC 0159

Logic lowered by rule

ekos migrate lower turns each recovered PostgreSQL view into a ClickHouse view, rewriting where the same text would return something else: sumOrNull for sum, join_use_nulls for outer joins, explicit NULLS FIRST, Nullable(T) casts, char_length. Every unaliased select-list item is given the column name PostgreSQL gives it (ClickHouse would call u.id just that). Every rewrite is listed. A construct with no rule blocks the view by name, and a materialized view always blocks. The result must parse on the target, read back identically, and state nothing the source does not. Triggers get a proposed replacement per class with the behaviour change it implies; routines get an eligibility verdict. ekos migrate check-candidate rejects a statement that names a table, column, literal or comparison its source lacks. --record saves each lowering and trigger proposal as a MigrationProposal, status proposed.RFC 0164

Run against the source (V5)

ekos migrate v5 builds a small fixture from the view’s own comparisons — the literal and its neighbours for every column <op> literal, NULL in every column, compared columns crossed — loads it into a PostgreSQL sandbox and a ClickHouse sandbox, and compares what the source view and the lowered view return. The verdict always carries branch coverage: how many of the view’s comparisons the fixture took both ways. A view that cannot be run is not run, with the reason, and that is never a pass. Both connections must be marked sandbox = true; a migration source is refused even with the mark. PostgreSQL work is one rolled-back transaction, ClickHouse work a database that is dropped. A live suite damages a passing lowering nine ways and requires V5 to fail on each. Row order is not checked.RFC 0164

Model rewrites, fenced

ekos migrate reconstruct sends one PL/pgSQL routine to the configured model. A routine with unrecovered statements or dynamic SQL is never sent. A model that is not local needs --allow-metered on that run. The answer is rejected if it names a table, column, literal, function or comparison the routine does not. A survivor is an unexecuted proposal: the check finds what was added, not what was left out, and V5 for routines is not built.RFC 0164

Guarded execution

Every statement is parsed and classified, never string-matched. Unparseable, unknown or credential-carrying statements are refused. Outside the sandbox, a statement runs only if an approval matches its artifact id, content hash and environment, with no override. Loads are chunked by half-open key ranges, and --dry-run gates and prints everything while executing nothing.RFC 0160

Computed risk, human-only approval

Risk is statement class × environment × lossiness × blast radius (compiled dependents of the table) × affected rows. An approval is pinned to an evidence snapshot and dies if the evidence changes. Self-approval is refused, and R4 needs two distinct approvers plus a typed confirmation. The hash of migrate.policy.toml is recorded on every approval. No MCP tool can approve, which a source-scan test enforces.RFC 0161

Validation that cannot pass silently

One canonical value serialization, agreed three ways by PostgreSQL, ClickHouse and Rust. Tiers V1 (counts), V2 (aggregates) and V3 (bucketed row hashes, bisected to exact keys). A tier passes only with zero blocking divergences and every planted control fired.RFC 0155 / 0156

Compiled report

Built from ledger queries, not generated. Every factual sentence carries fact ids that are resolved and structurally checked (a row count cannot cite a design). Prose containing a number is refused. Five sign-off preconditions are reported together. The same snapshot recompiles byte-identically.RFC 0162

Not yet: logic translation. The source side is recovered — PL/pgSQL routines statement by statement (RFC 0163; 212/212 LedgerSMB routines), views (RFC 0169) and triggers classified by what they do, all linked to the tables they touch — but no lowering, constrained reconstruction or V5 differential execution (RFC 0164) consumes it yet. Also missing: V0 structural and V4/V5 validation, ekos migrate signoff, the Delta/Databricks target, CDC and cutover, and the agent pack.RFC 0154–0167

17

Live, generated evidence: this repo, compiled by itself

Everything below this line is rendered straight from ekos docs generate --layout curated and --layout solution-architect's own output, run against EKOS's real, currently-committed ledger — the same commands and output shape any other project gets. Nothing here is hand-typed.

Architecture summary

1,829
RustSymbol
691
File
625
RustModule
472
Claim
1,347
Section (doc)
267
Document
133
Risk
59
Rollup
50
Technology
46
Crate
2
Issue
1
PullRequest
40% architecture confidence — completeness: 0% of 46 crates classified, evidence coverage: 100% of 472 claim/gap objects (RFC 0065 Phase 3)

"Purpose: not yet computed — no real EKOS source for a project's stated purpose today."
"Architecture style: not yet computed — requires reasoning EKOS doesn't perform yet."
— verbatim from the generated page. An honest placeholder, not a fabricated 100%.

Concentration risk

Real compiled dependency fan-in, not a guess — the top of a 133-entry Risk list.

Top technologies, the same way:

Solution-architect layout · real findings

44 crates with no declared version
Every internal crate uses version.workspace = true rather than a literal per-crate version string — the analyzer reads the literal field, so it correctly reports "not declared" rather than resolving the workspace-inherited value. A real analyzer limitation, not a real versioning problem.
625 / 625 RustModule objects have no captured doc comment
Every module-level item in the workspace, undocumented at the module level (function/struct-level docs are separately, better covered below).
982 / 1,829 RustSymbol objects have no captured doc comment
54% of real compiled Rust symbols carry no doc comment — harder to safely change, and no real text for an AI-Assisted Overview to ground on if one were requested.

Onboarding guide · excerpt

A real repository-layout table from the actual Cargo.toml graph, not a hand-maintained architecture doc that drifts:

| Path | Crate | | benchmark | ekos-benchmark | | ekos/crates/cli | ekos | | ekos/crates/ledger | ekos-ledger | | ekos/crates/recovery | ekos-recovery | | ekos/crates/runtime | ekos-runtime | … 41 more, every one a real compiled Cargo.toml → path edge …
This page is a living document — updated after every devlog that ships a new or changed capability (see CLAUDE.md's Devlog Rule). The §15 numbers reflect this repository's state as of the last regeneration; everything above §15 describes the platform's stable, shipped capability surface and changes far less often.