The Pentaho + SQL recovery deck documented an honest gap — AdventureWorks' SQL files all failed to
parse — as a limit, not hidden. It became GitHub issue #3. A second gap, `StreamLookup` steps
falling through to Unmapped, became issue #2. Both are closed, and this
deck reruns the exact repro shapes from each issue against the fixed pipeline, live.
Both bugs were found the same way the recovery deck found them originally: run EKOS cold against real files, don't fabricate anything to paper over a gap, and report exactly what didn't map.
StreamLookup — one of the most common real-world Kettle step types — had no match arm in map_step, so it fell through to Unmapped.# line comments broke the tokenizer outright; hand-written scripts with no ; between statements failed the whole file, not just one statement.# issue #3, root cause 1 sqlparser error: Expected: an SQL statement, found: # # issue #3, root cause 2 sqlparser error: Expected: end of statement, found: update # issue #2 node_type: "Unmapped" reason: "unrecognized step type: StreamLookup"
RFC 0031 made SQL dialect selection pluggable and config-driven — a per-file
[[recover.sql.dialect-rules]] glob resolves which
SqlDialectParser to use, instead of every recovery pass
hardcoding GenericDialect. MySqlDialect
already tokenizes #-style comments correctly — the fix is picking
it, not writing a new tokenizer.
generic, mysql, postgres — a new dialect crate plus one registry line, same pattern as every Observer plugin.ekos.toml: path-glob = "*MySQL*/*.sql" → dialect = "mysql".let stmts = match Parser::parse_sql(dialect, sql) { Ok(s) => s, Err(first_err) => { // only after the unmodified text // has already failed to parse let repaired = ensure_statement_separators(sql); Parser::parse_sql(dialect, &repaired) .unwrap_or_else(|_| { /* ... */ vec![] }) } };
Hand-written scripts often separate statements with a blank line instead of
; — sqlparser requires the explicit
separator. Inserting one unconditionally would risk corrupting a legitimate
UNION ALL SELECT ... chain, which also starts a line with
SELECT. Restricting the repair to a retry-after-failure means it
can never touch input that already parses correctly.
UNION/INTERSECT/EXCEPT) so multi-line constructs are never split.SqlAnalyzerPass (DDL) and SqlTransformAnalyzerPass (DML) — one fix, two consumers.StreamLookup is a left join. Now it's modeled as one.
Kettle's StreamLookup XML has no join_type
field, unlike DatabaseJoin/MergeJoin —
because a stream lookup is semantically always a left join against the lookup stream on the
configured key(s). The fix reuses the existing join-extraction shape and forces
JoinKind::Left, rather than adding a new IR variant.
TransformNode::Unmapped { reason: "unrecognized step type: StreamLookup" }TransformNode::Join { kind: Left, keys: [...] } — same evidence-backed shape as every other join.id name kind 45590d6b-... eae_data_management_mmjja Table b4206f56-... testing_scenarios Table 2 row(s).
{ "node_type": "Join",
"summary": "Left joins on
[[\"SalesTerritoryKey\",\"SalesTerritoryKey\"]]",
"evidence": [{ "fragment":
"Left JOIN ON [(\"SalesTerritoryKey\",
\"SalesTerritoryKey\")]" }] }
The first: a scratch DB Scripts/Destination MySQL/ +
Source MSSQL/ fixture reproducing both of issue #3's root causes —
# comments in one file, zero semicolons across three statements in the
other — recovers both tables. The second: a real 3-step .ktr
(TableInput → StreamLookup → TableOutput) explained via a live MCP
JSON-RPC call, with the join key cited as evidence straight from the file.
| Fixture | Before | After |
|---|---|---|
| Dialect-mixed DB Scripts (issue #3 repro) | 0/2 tables recovered | 2/2 tables recovered |
| fact_sales.ktr, 3 steps (issue #2 repro) | 66% mapped (StreamLookup → Unmapped) | 100% mapped |
# recover a dialect-mixed estate — config-driven, per file $ cat ekos.toml [[recover.sql.dialect-rules]] path-glob = "*MySQL*/*.sql" dialect = "mysql" # then the usual pipeline $ ekos build && ekos recover && ekos resolve && ekos compile && ekos commit
| Doc | Where |
|---|---|
| Demo Acts 11 & 12 | demo/DEMO.md — full repro + verification steps |
| Issue #2 / #3 | github.com/alexeyban/EKOS/issues (closed) |