Vetting

PreFlight — Deterministic Upstream Data Ingestion Gate

pf_v1.01 is tested against a corpus of real-world structural failures; this page documents the suites and results.

The test suite

Engine structural validation lives in the step10 pytest package (tests/step10/). The suite currently comprises TODO-CLAY: confirm tests.

Each evaluation runs seven tests in fixed order:

1. Schema Structural Integrity
2. Row Iteration Viability
3. Numeric Coercibility
4. Multiple Valid Representations
5. High Null Density Columns
6. Malformed or Truncated Rows
7. Text Encoding Integrity

Final verdict reduction is mechanical:

1. Any FAIL → FAIL
2. Else any WARN → WARN
3. Else any ANALYSIS_INCOMPLETE → ANALYSIS_INCOMPLETE
4. Else all PASS → PASS
5. Else → ANALYSIS_INCOMPLETE

Fail-closed behavior:

• Content plausibility, delimiter detection, and evidence sufficiency gates run before the seven tests; failure short-circuits to ANALYSIS_INCOMPLETE without issuing PASS.
• Evidence gate minimums: 2 columns, 30 data rows, 20 non-empty rows (logical csv.reader rows).
• Reduction never promotes a dataset to PASS when any test returns FAIL, WARN, or ANALYSIS_INCOMPLETE.

The step10 suite gates every engine change to preflight_core.

Mutation testing

Enforcement logic is deliberately mutated; the step10 suite must kill each competent mutant or the gap is closed. Equivalent mutants (no behavioral effect on current fixtures) are documented separately.

Latest cosmic-ray run (dataset_testing/mutation_results.json):

Mutants generated

1,306

Completed

1,306

Killed

1,306

Survived

0

Mutation score

100.0%

Runtime

1,286.1 s

Survivors on disk

none (mutation_survivors.json is empty)

The corpus

Validated against TODO-CLAY: confirm structural failure patterns drawn from real-world failures. The corpus grows; the published N updates.

Documented corpus sweeps in verification artifacts:

• dataset_testing/s2_results.json — 28 degenerate-input fixtures with reconciled verdicts
• dataset_testing/s2b_num_results.json — 13 numeric-shape fixtures with reconciled verdicts
• dataset_testing/degenerate_inputs/ — 61 committed CSV fixtures

No engine change ships with a corpus regression against these sweep results.

Check it from outside

• Evidence — download inputs and reports; verify SHA-256 hashes
• Examples — reproducible verdicts from precomputed reports
• Assurance — attestations and verification summaries