A Pentaho step, a SQL SELECT, a VIEW,
and a stored procedure are all the same underlying idea — a transformation of data from sources
to a sink. EKOS compiles all of them into one shared, evidence-backed representation, so an AI
agent can explain what a legacy pipeline does and prove a migration didn't change its meaning.
A developer needs to reproduce an existing Pentaho job's business logic in a new pipeline —
with one rule changed — across an estate spanning GitHub, PostgreSQL, Databricks, Synapse,
an Informix database with no source repo, an outdated Confluence, and legacy Kettle ETL.
Today that means manually reading .ktr/.kjb
XML and hunting for tribal knowledge in Confluence.
Building a separate extraction path per format would produce N incompatible semantic models that can't be diffed against each other — which defeats the point. So everything maps into one intermediate representation first.
TableInput, a SQL FROM clause.FilterRows, a SQL WHERE clause.MergeJoin/DatabaseJoin, a SQL JOIN.GroupBy, a SQL GROUP BY with aggregate functions.Calculator, a SQL scalar expression.TableOutput, a CREATE VIEW target.
Two independent parsers compile into the same TransformNode
enum — a Pentaho XML DOM walker (roxmltree) and a SQL AST walker
(sqlparser-rs, Postgres/T-SQL/Databricks dialects). The same
bytes always parse to the same graph, with zero judgment calls — that's what keeps this a
deterministic compiler pass, not an LLM guess.
Unmapped, honestly.pub enum TransformNode { Source { object_name: String, columns: Vec<String> }, Filter { condition: String }, Join { left: NodeId, right: NodeId, keys: Vec<(String, String)>, kind: JoinKind }, Aggregate { group_by: Vec<String>, aggs: Vec<AggExpr> }, Calculate { output: String, expr: String }, Sink { object_name: String, columns: Vec<String> }, // deliberate — never a silent drop Unmapped { raw: String, reason: String }, }
ekos_transformation_explain — every claim cites its source.
Point the tool at a pipeline's Sink and it walks the chain of
real IR nodes feeding into it, root-first, rendering each into a plain-English step with the
exact source fragment behind it. Unresolved steps are surfaced by name — ⚠
not understood — not silently omitted.
.ktr file — every fact comes through this one tool.{ "node_type": "Filter",
"summary": "filters rows where status = 'active'",
"evidence": [{ "source": "load_customers.ktr",
"fragment": "status = 'active'",
"confidence": 1.0 }] }
ekos_transformation_diff — exactly what changed, nothing more.Draft the new pipeline, then diff its Sink against the
original's. Sources, filters, joins, aggregates, and calculates are bucketed and compared as
sets — not by position, since a pipeline can reorder independent steps without changing meaning.
"sources": { added: [], removed: [] } // unchanged "sinks": { added: [], removed: [] } // unchanged "filters": { removed: ["status = 'active'"] added: ["status = 'active' AND region = 'EU'"] }
Verified end to end against a real
benchmark: a two-source, filtered, joined, calculated Pentaho job redrafted as SQL with one
changed rule — 100% coverage, zero Unmapped nodes, and the diff
isolated exactly the one rule that changed.
Informix cust_mstr, Postgres customers,
Databricks gold.dim_customer — the same real-world entity, three
different names, three different systems. ekos identity scan
scores candidate matches on column overlap, naming-pattern similarity, and type
compatibility, and writes each one as an unconfirmed relationship.
ekos_identity_review — the one write-capable MCP tool, scoped to exactly this relationship kind.A candidate cross-system match carries an explicit unconfirmed status — structurally distinguishable from an observed fact, never indistinguishable.
Every claim in an explanation cites the exact source file and fragment it came from — nothing is asserted without a citation.
What can't be parsed is recorded as "something is here, not yet understood" — never dropped, never presented as a complete answer.
The same source re-parsed twice gets the same id; a re-parse that changed is a new version, never an in-place rewrite.
# compile every .ktr/.kjb and SQL file once $ ekos build && ekos recover && ekos resolve && ekos compile && ekos commit # link the same entity across systems (optional) $ ekos identity scan # serve it to any MCP client over stdio $ ekos mcp serve --workspace .
| MCP tool | Answers |
|---|---|
| ekos_transformation_explain | What does this legacy pipeline actually do, with evidence |
| ekos_transformation_diff | Did my redraft preserve the original logic |
| ekos_identity_review | Confirm/reject a cross-system identity match |