EKOS recovers Pentaho ETL logic into the Transformation IR (RFC 0027) — the same
compiled graph ekos docs generate already renders as documentation.
ekos dbt generate renders that same graph a second way: real dbt SQL
models, ref()/source()-chained via the
compiled data-flow edges. Every example on this page is real output from a real cloned Pentaho
repo, unedited — including the five real bugs it found, and fixed, along the way.
ekos dbt generate$ ekos dbt generate dbt models generated. Models rendered: 98 Source tables: 23 Output: dbt-generated
98 real dbt .sql models from 10 real Pentaho jobs — one file per
compiled TransformNode, ref()-chained via
the same real FeedsInto edges the documentation deck's diagrams
already draw from. Zero LLM calls, zero new analyzer: this reads the exact graph
ekos docs generate reads.
<step>
<name>Sales Person</name>
<type>TableInput</type>
<connection>AdventureWorks</connection>
<sql>SELECT
BusinessEntityID
, TerritoryID
FROM Sales.SalesPerson
</sql>
select
BusinessEntityID,
TerritoryID
from {{ source('pentaho', 'Sales.SalesPerson') }}
The FROM clause resolves into a real dbt
source() reference, and the real declared columns — read from the
step's structured <row-meta>/<value-meta>/<name>
metadata, not parsed out of the free-text SQL — become a real explicit column list instead of
select *.
<step>
<name>Sales Territory Lookup</name>
<type>StreamLookup</type>
<lookup>
<key>
<name>TerritoryID</name>
<field>territory_id</field>
</key>
</lookup>
-- NOTE: l/r assignment is positional... select * from {{ ref('fact_sales_ktr_0') }} as l left join {{ ref('fact_sales_ktr_14') }} as r on territory_id = TerritoryID -- TODO: verify column qualification, source dialect: Pentaho
Real finding from this session's Phase 2 testing, now fixed: every real
MergeJoin/StreamLookup step in this repo
first compiled with empty keys — each uses its own real key XML shape
(<lookup><key><name>/<field> for
StreamLookup; <keys_1>/<keys_2>
for MergeJoin), neither matching DatabaseJoin's
<keys><key><value1>/<value2> shape the analyzer
originally only read. Both shapes are parsed correctly now — real keys, real join kind, real
topology, for every real join in this repo.
-- Unmapped: unrecognized step type: Sequence -- Raw source: -- <step><name>Create surrogate key</name> -- <type>Sequence</type>... </step> select * from {{ ref('dim_customer_ktr_5') }} -- passthrough stub, not translated
A real Kettle Sequence (surrogate-key generator) step —
no structured semantics to render, so it becomes an honest passthrough: the raw XML and reason
preserved as a comment, still ref()-chained to its real upstream, so
the pipeline's shape stays connected end to end even where content isn't fully translated.
"Unmapped is a citizen, not a failure" — the same rule the documentation generator runs on,
applied to a second output format.
select * from {{ ref('dim_date_ktr_3') }}
version: 2
sources:
- name: pentaho
tables:
- name: "Sales.SalesPerson"
- name: "Sales.Customer"
- name: "fact_sales"
... 20 more, real
models:
- name: dim_customer_ktr_0
... 97 more, real
23 real source tables, 98 real models — every sources:
entry traces to a real compiled Source node's
object_name, never invented.
-- Filter: select * from {{ ref(upstream) }} where {condition} -- TODO: verify, source dialect: Pentaho -- Calculate: select *, {expr} as {output} -- TODO: verify, source dialect: Pentaho from {{ ref(upstream) }}
Kettle's Filter Rows/Calculator
conditions and expressions are raw, un-parsed source-dialect text — RFC 0027 already rejected
building a cross-format expression AST as out of scope, since a wrong guess here would be worse
than an honest TODO. Every one of these gets inlined exactly as compiled, flagged for manual
verification, never silently rewritten.
TableInput steps compiled with columns: [] — the real declared columns were in <row-meta>, never read. Fixed: read from the real structured metadata.<lookup><key><name>/<field>) doesn't match DatabaseJoin's. Fixed: a dedicated extractor for the real shape.<keys_1>/<keys_2>, two separate lists) doesn't match DatabaseJoin's either. Fixed: positional pairing of the two real lists.l./r. prefix produced l.o.customer_id — invalid. Fixed: keys pass through unmodified.on true fallback had a "verify column qualification" comment appended with nothing to verify. Fixed: the comment only attaches to real key text.select * from {{ source('pentaho', 'Sales.SalesPerson') }} -- fixed: select BusinessEntityID, TerritoryID from {{ source('pentaho', 'Sales.SalesPerson') }}
All five found by running this against a real 198-object Pentaho repo, not by
unit tests — the same pattern that found real bugs in the documentation generator (RFC 0035).
Three live in the recovery layer (pentaho_analyzer.rs, shared with
ekos docs generate), two in this feature's own rendering. Real, messy
data keeps finding what hand-built fixtures miss.
# compile once $ ekos build && ekos recover && ekos resolve && ekos compile && ekos commit # real dbt models, ref()-chained, zero LLM cost $ ekos dbt generate