Plan: Dramatically expand end-to-end test coverage of blr

Status: Complete (all 7 phases; committed per phase on quote)
Date: 2026-07
Ticket numbers (§3.3, filed on the blr tracker): #3 closures/abstractions across boundary, #4 non-primitive list elements, #5 lists in remaining boundary contexts, #6 strings in remaining boundary contexts, #7 scalar heap-result path, #8 Frame/DataFrame boundary positions, #9 bool flat slots, #10 row monomorphization, #11 unification error reporting, #12 Expr::Tag simplification, #13 uncategorized codegen gaps, #14 val_to_string non-list shapes, #15 designed v1-limit panics, #16 recursive types. Filed during Phase 1 (M1 parity probing): #17 sum extern-result as main return, #18 list element corruption, #19 list literal codegen panic, #20 list row type error, #21 record list field, #22 sum list payload inference, #23 main dropped from exports (record containing sum). Filed during Phase 2–7: #24 single-field record flattening, #25 sum-in-sum boundary, #26 quote capture of free variable, #27 float literal in predicate, #28 list literal with expression elements, #29 bool in export contexts, #30 string concat invalid component, #31 sum record payload as main return, #32 undefined item panic, #33 df missing-column host error, #34 wide-record 2MB stack overflow, #35 params_on_heap todo (>16 flat slots).

Goal: systematically expand E2E coverage so that every language feature and
type-system feature is exercised in combination, with emphasis on how
features compose across the four seams where blr is most likely to break:

1. Local codegen (air → crust → mantle → core → nucleus WASM)
2. The Wasm component boundary (lift/lower of values in both directions)
3. Unification-based inference (rows, type vars, monomorphization)
4. Data (csv → DataFusion → quoted predicates → frames)

Hard constraint: this plan adds tests only. It does not expand the scope
of the implementation. Where a matrix cell hits an unimplemented path
(todo!(), designed v1-limitation panic), the test is ignored and the
gap is recorded as a ticket on the blr todo tracker via the hut CLI
(§3). No phase of this plan implements new language features, changes
semantics, or converts panics to typed errors.

Approved scope exceptions (user-directed, both land in Phase 0):

1. val_to_string for Val::List (lang/src/lib.rs:84, currently
   todo!()) — needed so tests can return a list of results and so
   list-typed cells can print. Format mirrors the literal syntax:
   [], [1, 2.5], [Bar(1.0), 2]. No changes to any *existing* arm's output. (Val::Tupleand friends staytodo!()` → ticket.)
2. Test auto-discovery via lang/build.rs codegen (§1.2) — build
   infrastructure, not language scope.

Companion reading: list-types,
sum-types,
dataframe-row-types,
declared-layout-order,
recursion.

1. Current state

Existing E2E suites

Harness limitations (answer: the system needs expansion)

The core idea (snapshot the printed result of blr_lang::run) is right and
should stay. But:

1. Manual registration. Every test must be added to the test_case! list
   in integration.rs. A new .blr that is never registered is silently
   untested. → Build-script auto-discovery (§1.2): the test_case!
   invocations are generated from the directory contents; registration
   disappears as a concept.
2. Single observation point. Only main's final value (or the last
   expression) is visible; there is no stdout. print.blr is effectively dead
   ("ignore till we get more complete cm+gc support"). Convention to adopt:
   tests return a list of results ([a, b, c]) or a record of named
   results ({a: ..., b: ...}) so one test can observe many values.
   Enabler (approved exception, Phase 0): implement val_to_string for
   Val::List (lang/src/lib.rs:84), format mirroring the literal syntax.
3. Error and panic outputs. Error outputs are already
   full-string expect_file snapshots (Report::from_error(e).to_string()),
   which is exactly what we want: error messages will be significantly
   improved later, and full snapshots make every wording change visible as a
   small, reviewable diff that UPDATE_EXPECT=1 updates in one pass. → Add
   integration/errors/ running through the same full-snapshot mechanism
   (no substring/marker asserts). Error tests are expected to be the churning
   suite; that churn is a feature, not a bug, during the message-improvement
   work. For designed guard panics (e.g. "bool cannot cross the component
   boundary in v1", crust/mod.rs:468), the runner executes the program in a
   spawned task so the panic becomes a JoinError, and the
   full panic message is snapshotted the same way (§1.2) — the single
   test_case! runner handles success, error, and panic uniformly, so
   errors/ is an organizational subdirectory, not a different mechanism.
4. One data file, one row. test.csv is a,b,c / 1,2,3.4. Every df test
   runs on identical, degenerate data. → lang/tests/data/*.csv fixtures
   (§6).
5. type_check snapshots the entire module Debug. Any compiler-internal
   refactor rewrites all 7 snapshots. → Split into
   type_check/ok/ (assert success + the solved type of main, not the
   whole module) and type_check/err/ (full-string error snapshots, see #5).
6. No ignore concept. Cells the matrix (§4) knows to be unimplemented
   would panic!/todo!() if written. → .skip.blr convention + hut
   tickets (§3). This turns the matrix's unknowns into tracked, bounded debt
   instead of landmines.
7. No scale or determinism dimension. → Phase 7 (bounded: scale tests are
   ignored if they hit a todo!()).

Everything else about the harness (expect_file, UPDATE_EXPECT workflow,
BLR_SNAPSHOT debug dumps, blr-compare-snapshots.sh) is good and stays.

1.2 How auto-discovery works

lang/build.rs already exists (it runs lalrpop::process_root()). Extend it
to also generate the test-registration list:

lang/build.rs
  ├─ scans (sorted):
  │    tests/integration/*.blr            (excl. *.skip.blr)
  │    tests/integration/errors/*.blr
  │    tests/type_check/ok/*.blr
  │    tests/type_check/err/*.blr
  ├─ emits $OUT_DIR/generated_tests.rs:
  │    test_case!(one_plus_one);
  │    test_case!(df_chain);
  │    ...
  │    test_case!(bool_boundary);        // from errors/
  │    test_type_ok!(select_rows);
  │    test_type_err!(bool_mismatch);
  └─ emits cargo:rerun-if-changed=tests/integration, ...
     so adding/removing/renaming a .blr re-runs the build script

lang/tests/integration.rs
  ├─ keeps the single test_case! macro and blr_run()/blr_type_check()
  └─ the 36-line manual list becomes:
       include!(concat!(env!("OUT_DIR"), "/generated_tests.rs"));

One runner for everything — blr_run spawns the program in a task so all
three outcomes are just strings to snapshot:

async fn blr_run(prefix: &str) {
    // ... read tests/integration/[errors/]<prefix>.blr into src ...
    let res = tokio::task::spawn(async { blr_lang::run(&src, "..").await }).await;
    let out = match res {
        Ok(Ok(value)) => blr_lang::val_to_string(value),
        Ok(Err(e))    => Report::from_error(e).to_string(),
        Err(join)     => format!("panic: {join}"),  // full panic message
    };
    expect_file![filepath_out].assert_eq(&out);
}

No separate "error case" macro: success, Err, and panic all flow through
the same full-string snapshot. The old safety property ("a panic in the main
suite fails the build") is preserved by the review workflow instead: every
.blr must have a committed, reviewed .out (build guard), so a new bug's
panic shows up as a snapshot diff in the PR, never silently.

Properties this buys:

• No registration step. Drop a .blr in the directory → it is a test on
  the next build. Delete it → it vanishes (stale .out caught by the guard
  below).
• Per-test isolation preserved. Each generated line is still its own
  #[test(tokio::test)], so parallelism, cargo test <name> filtering, and
  failure isolation are unchanged from today.
• Path resolution is unaffected. expect_file! is invoked inside
  blr_run/blr_type_check in integration.rs, so relative .out/.typ
  paths still resolve against tests/ regardless of where the test fn is
  defined; the generated file contains only macro invocations, no paths.

Guards (in build.rs, hard build failures with a named file + reason):

• every .out/.typ has a corresponding .blr (stale snapshot)
• every non-skip .blr has a corresponding .out (missing expectation;
  developer runs UPDATE_EXPECT=1 to create it)
• every *.skip.blr carries a well-formed // @skip blr#N — … header (§3.2)
• file stems are valid Rust identifiers

The richer cross-checks (manifest ↔ files ↔ tickets, §4.7) need the
manifest's content, so they stay a #[test] in the style of
stdlib_guard.rs; the build script handles only the cheap filesystem
invariants.

Multi-file programs — out of scope

Only import std::… is exercised today. Local multi-module programs have no
fixture mechanism; supporting them is an implementation question, not a test
one. Parked under open question Q1 (§9); no harness work in this plan.

2. Feature inventory

2.1 Expression forms (from air::Expr)

2.2 Type-system features

2.3 Runtime / environment features

3. Scope bounding: ignores and hut tickets

3.1 Principle

The matrix (§4) describes the space of compositions; the implementation
covers a subset of it. The plan's job is to make the difference visible,
enumerated, and tracked — not to close it. Rules:

• A cell that passes today gets a real test.
• A cell that hits todo!()/panic/design-limit is not worked around, not
  implemented, and not flaked — it becomes an ignored cell: a
  .skip.blr file that documents the intended test, plus a ticket on the
  blr todo tracker naming the exact todo!()/guard site.
• When (in some future plan) an item is implemented, the same PR deletes the
  .skip.blr (promoting it to a live test) and closes the ticket. Tickets
  are the single source of truth for "why is this cell empty".

3.2 Ignore mechanism (harness)

• Naming convention: foo.skip.blr = "intended test, not run". The
  auto-discovery harness (§1.1) excludes .skip.blr from execution and
  from the orphan guard.
• A skip file must carry a header line, enforced by a new guard test
  (alongside stdlib_guard.rs):

  // @skip blr#44 — todo!("unsupported list element type") at lang/src/compiler/nucleus/mod.rs:456
  pub fn main() -> list<{x: i64}> { [...] }
  

  i.e. // @skip <tracker>#<n> — <reason, incl. code site>. A .skip.blr
  without a well-formed directive fails the build. (This keeps skip files
  honest: they are documentation of a known gap, not a dumping ground.)
• The matrix manifest (§4.7) cross-references: every ⛔ cell cites a
  .skip.blr file + ticket; every ✓ cell cites a live test. A guard test
  (or a just target, if a test is overkill) fails on mismatch.

3.3 hut ticket workflow

Tracker: blr on sr.ht under ~nathanielc (hut -t "~nathanielc/blr"; first
migrated ticket #3
already filed). Tickets record unimplemented behavior, one per distinct
behavior, listing every todo!()/guard site that enacts it — not one
ticket per code line.

Justfile additions:

# List open blr todo tickets
todos:
  hut -t "~nathanielc/blr" todo ticket list

# Create a ticket: just todo-add "Title" "body line 1..."
todo-add title:
  #!/usr/bin/env bash
  printf '# %s\n\n%s\n' "$title" "$(cat)" | hut -t "~nathanielc/blr" todo ticket create

Ticket body format (proposed):

# <behavior>, e.g. "list<record> cannot cross the component boundary"
Sites:
- lang/src/compiler/nucleus/mod.rs:456  todo!("unsupported list element type {elem_ty:?}")
- …
Tests blocked (lang/tests/integration):
- list_record_roundtrip.skip.blr
Design ref: knowledge/design/list-types.md (scope: no record elements yet)

Initial inventory (Phase 0): grep found 51 todo!()/unimplemented!()
sites plus the designed v1-limitation panics in crust/mod.rs:468-476.
Provisional grouping (~12–15 tickets; exact split finalized during Phase 0):

(val_to_string for Val::List, lang/src/lib.rs:84, was in this inventory
but is implemented in Phase 0 as an approved scope exception — no ticket.)

Each ticket gets its number written into the header of every .skip.blr it
blocks and into the matrix manifest.

4. The composition matrix

The interesting bugs in blr live at intersections. Six axes:

• A. Type shape — i64, f64, bool, string, unit, record, sum,
  list<T>, DataFrame<'a>, alias-of-any-of-these
• B. Boundary position — local-only / extern param / extern result
  / main return (component export) / record field crossing / sum
  payload crossing / list element crossing / df row (via csv)
• C. Construction — literal / .. concat / match / closure / quote
• D. Typing mode — inferred / aliased / qualified-extern / polymorphic 'a' /
  open-row (row superset)
• E. Consumption — field select / >> chain / binary op / local call /
  extern call / match pattern / printed via val_to_string
• F. Size — empty, minimal (1), large (n), edge values (i64 min/max,
  0.0, negatives, empty string, deep nesting)

The full product is ~10⁴; we don't write it all. Instead, five generated /
systematic suites cover the high-yield slices, plus hand-written suites for
the rest. Every cell resolves to exactly one of: live test ✓, skip+ticket ⛔,
or explicitly out-of-matrix ○ (recorded in the manifest, §4.7).

M1. Boundary parity (A × {param, result, main-ret})

The single highest-value suite. For a value of type T, local identity
must equal the round-trip through an extern identity:

// expect .out identical to the local twin
pub fn main() -> T { ext::identity(VALUE) }   // vs local: VALUE

Cells: every A × {record shapes, sum shapes, list-of-X} from the nesting
lattice below, in both directions (param-in, result-out, main-return). This
mechanically exercises lift+lower+resv+GC for every supported shape and
would have caught every class of bug in the sums/list ABI work. Implement as
one test per cell; generate the fixtures with a small script
(lang/tests/gen_boundary.rs or just gen-tests) since they're formulaic.
The existing test worlds (std::sums, std::math) get an identity
extern per supported shape (or a dedicated parity test world — cleaner,
follows the sums precedent). Shapes whose lift/lower is a todo!() become
skip+ticket cells (e.g. list<record> round-trip vs
nucleus/mod.rs:456); the parity suite thereby also enumerates the
boundary's actual capability surface.

M2. Nesting lattice (depth ≤ 2–3)

Container pairs {record, sum, list} × element types {i64, f64, bool, string,
record, sum, list}:

• list<record>, list<sum>, record{list<…>}, sum with list payload,
  list<list<T>>, sum-in-sum-in-record, etc.
• Each cell: local construct+consume and M1 boundary parity where the
  boundary supports it; skip+ticket where it currently todo!()s.
• After the Phase 0 val_to_string Val::List implementation, list values
  are directly printable as main results, so no cell needs the consume-via-
  extern workaround; nested list elements print recursively.

M3. Row algebra (T2 × T3 × E3/E4)

Records are structural and order-canonical; that's a whole combinatorial
surface:

• literal {x, y} vs {y, x} unify; alias {x, y} vs extern {y, x}
• r .. {z} .. {w} chains; concat then select; concat then cross boundary
  (field order in WIT vs concat order)
• select in different orders than declaration
• row superset E2E: df::filter(df::filter(fromcsv, :(r.a…r.b)), :(r.c…r.d))
  actually runs on a csv with columns a,b,c,d (today only type-checked in
  select_rows); plus a negative case where a predicate references a
  non-existent column

M4. Inference composition (T9)

• poly item over record / sum / list<T> / DataFrame<'a> / nested row
  (skip+ticket where row monomorph todo!() bites, mantle/monomorph.rs:119)
• one poly item instantiated at two different row types in one program
  (forces two monomorphizations — the instances map path)
• poly item that is never called (regression for the silent main-drop
  hazard documented in quote-row-superset —
  tests current behavior, whatever it is, via snapshot; if it's the drop,
  that's recorded in a ticket, not fixed here)
• partial application of poly items; closures capturing sums/lists then used
  through >> chains
• quote as param: user fn fn apply(f: DataFrame<'a>, q: Quote<'a -> bool>)
  delegating to df::filter

M5. Operator & boolean matrix (E1/E15 × A)

• + - * / × {int, float} incl. negatives, i64::MAX/MIN, 0.0,
  0.1 + 0.2 (determinism: local vs boundary must print identically)
• > < = × {int, float} → bool executed locally (zero E2E coverage
  today); if local bool codegen turns out to be a todo!(), the whole bool
  row of this suite becomes skip+ticket in one batch — that's the scope-bounded
  outcome, not a fix
• bool as: main return, record field, sum payload, list element, boundary
  param/result — boundary positions are a designed v1 limit
  (crust/mod.rs:468) → error tests (full panic snapshots) + one ticket covering
  the design limit
• string concat (the Concat binary op exists in the AST but has no E2E) —
  pin which syntax works today; missing behavior → skip+ticket

M6. Quote/data composition (E12 × R3/R4)

• predicate operators: > < =, arithmetic in the body, chained selects,
  int + float mixes (operators the DataFusion node builder doesn't support →
  error tests or skip+ticket as discovered)
• quote values: let q = :(…); reused twice; quote built in a helper and
  passed into df::filter (exercises to_nodes on a parameter quote)
• csv shape variations (§6) through fromcsv → filter → show and
  fromcsv → show
• DataFrame<'a> in user items (T7): user fn taking/returning
  DataFrame<'a>

4.7 Matrix manifest

lang/tests/MATRIX.md: one table per suite (M1–M6 + hand-written), each row a
cell with status:

| cell | status | test / ticket |
|---|---|---|
| list<record> × extern param | ⛔ | list_record_param.skip.blr · blr#N |
| record{x: i64, y: f64} × extern param | ✓ | parity_record_xy.rs |
| bool × boundary | ⛔ | (designed limit) blr#M |
| quote-in-record field | ○ | not representable in v1 grammar |

The guard test (§3.2) fails if: a ✓ cell has no live test file, a ⛔ cell
has no .skip.blr with a matching ticket header, or a .skip.blr's ticket
number doesn't appear in the manifest.

5. Hand-written suites

5.1 match deepening

Pattern vars from payloads, _ on record payloads, match inside >> chains,
match on sums returned from externs with record payloads (sum-in-record E2E),
nested match, match exhaustivity errors (err tests).

5.2 df/data (see §6) — real data, not one row.

5.3 Ordering & ABI traps (T8)

• WIT case order ≠ blr decl order → error test snapshotting the full
  current panic message (documents the load-bearing
  order rule; the panic stays a panic — converting it to a typed error is
  out of scope)
• record field order WIT vs blr — same treatment
• rename a case/field on one side → covered already by stdlib_guard, keep

5.4 val_to_string printer coverage (E × A)

One test per supported type shape asserting the printed form: nested record,
sum (which case prints how?), string escapes, bool, unit. List printing is
todo!() (lib.rs:84) → skip+ticket. The snapshot suites are meaningless if
the printer silently mangles; this suite pins the printer as-is.

5.5 Error matrix (integration/errors/)

Full-string expect_file snapshots (same mechanism as success tests, plus
the spawn/JoinError path for panics, §1.2) for: type mismatch (each
operator pair), missing field, unknown label in match, undefined item/module,
arity mismatch, quote type error, row mismatch (closed vs closed rows),
designed-limit panics (bool/DataFrame/quote/polymorphic-extern across
boundary, WIT order mismatch — full panic message snapshotted), recursion
(T11), closure across boundary (R7). Rationale: error messages will be
significantly improved in a later plan; full snapshots make every wording
change a small visible diff, cheap to update with UPDATE_EXPECT=1. These
snapshots document today's messages; updating them en masse belongs to the
message-improvement plan, not to test additions here.

5.6 Entry-point variants

Explicit main with non-i64/non-'a returns (record, list, sum — list is
skip+ticket per printer todo), implicit main for every printably-supported
value shape.

5.7 Module structure

import of a module used for types only (import std::quote for
quote::Node), unused import, two imports with shared types
(std::df + std::quote).

6. Data & size fixtures

lang/tests/data/:

Value-scale tests: list of 10k f64 (GC/realloc path, lsum/lreverse),
record with 16+ fields, sum with 8+ cases, nesting depth ~10,
i64::MAX/MIN and 0.1+0.2 (M5). Scale tests that hit a todo!() (e.g.
GC under heavy allocation) become skip+ticket rather than being dropped.

Determinism: a harness option (or a separate just target) running every
integration test twice and asserting identical .out — cheap, catches
GC/iteration-order nondeterminism early.

7. Phased rollout

Each phase is a self-landing PR. End state: roughly 200–250 matrix cells
accounted for (~120–150 live tests + ~50–80 .skip.blr + ○), vs 36 today —
with the ⛔ set decreasing only when a separate implementation plan closes
its ticket and promotes the skip file in the same PR.

8. Explicit non-goals (scope bounds)

• No new language features, operators, or stdlib functions.
• No implementation of any todo!() site listed in §3.3 — this plan only
  files and links them.
  • Exception: val_to_string for Val::List (approved, Phase 0).
• No panic → typed-error conversions (designed-limit panics are pinned as
  full-snapshot error tests, message unchanged).
• No changes to error message wording in this plan; full snapshots make the
  later message-improvement plan's diffs reviewable.
• No multi-file local-module fixture support (Q1).
• No changes to existing val_to_string output formats; the printer is
  pinned as-is (only the Val::List arm is added, approved exception).

9. Open questions (do not block Phase 0)

1. Q1 — local imports: do local (non-std::) multi-module imports exist?
   If yes, they enter the matrix as a future suite; if no, ticket it and
   move on.
2. Q2 — local bool codegen: is bool runnable end-to-end locally? First
   M5 test reveals it; a todo!() means a batch of skip cells + ticket,
   not a fix in this plan.
3. Q3 — div-by-zero / NaN: pin current WASM behavior via snapshot tests;
   if it traps, that's the documented behavior. Semantics design is a
   separate plan.
4. Q4 — ticket granularity: confirm ~12–15 behavior-level tickets
   (vs one per site) with the tracker's intended use before Phase 0 files
   them.