Evidence Bundles

Before using a rule for real decisions, you need examples of what it should return and a way to check those examples against the version you plan to release. An evidence bundle is Devlish’s machine-readable report of those tests. It connects the test results to one compiled artifact rather than just a filename that could later change.

This page builds on Governed Rules. You will define inputs and expected results, generate a report, and check that report’s contents have not been altered. Passing cases demonstrate those cases; they do not prove the rule is correct for every possible input.

Define test cases

A golden case is a named example whose expected result has been reviewed. Each case supplies the program’s inputs and the value expected from Respond with. Choose ordinary cases and important boundaries, including situations that should be rejected.

The following example assumes credit_dti.dvl accepts income and debt and returns a boolean indicating whether the debt-to-income ratio is acceptable. Save these cases as credit_dti.cases.json beside the rule, or select a different case file with --cases:

[
  {
    "name": "acceptable debt-to-income ratio",
    "input": { "income": 6000, "debt": 1800 },
    "expected": true
  },
  {
    "name": "ratio above threshold",
    "input": { "income": 3000, "debt": 1500 },
    "expected": false
  }
]

Generate the report

Generation compiles the rule once and runs every case against that artifact. It compares each returned value with the case’s expected value and exits with a nonzero status if any case fails.

With the example rule and cases in place, write the report to evidence.json:

devlish evidence credit_dti.dvl --output evidence.json

The report includes the rule identifier and version, compiler version, artifact hash, pass/fail totals, and per-case hashes of inputs, outputs, and expectations. Hashes are content fingerprints that bind the report to specific data.

Verify a saved report

Verification recomputes the report’s digest and rejects a report whose stored digest no longer matches. This checks report integrity; it does not rerun the cases or validate the quality of the expectations.

devlish evidence --verify evidence.json

Controlled Releases records the evidence report hash when proposing a version and rejects a proposal whose cases fail. Audit and Replay covers a different question: what happened when a rule actually ran after testing.