Audit and Replay

A test report tells you how a program behaved on selected cases. After the program is put into use, a different question arises: which version produced a particular result, with which inputs? An audit log records that execution so someone can inspect it later.

Devlish also supports replaying a recorded run. If the program read a file or called a service, reproducing its calculation requires the responses it received at the time. An effect journal saves those interactions alongside the run. This page explains what each record contains and then shows how to create and verify it.

Start with Governed Rules for rule identity. The examples below assume pricing.dvl is a program with a Rule: header and that amount is one of its inputs.

Record an execution

The audit log stores one JSON record per completed governed run, including failed runs. A record identifies the rule and runtime, records whether execution succeeded, and includes hashes of the compiled artifact, input, and output. A hash is a content fingerprint used to compare data; it is not the data itself.

The following command runs the example rule and appends its execution record to audit.jsonl:

devlish run pricing.dvl \
  --input '{"amount": 100}' \
  --audit-log audit.jsonl

Each record links to the previous record’s hash. If the required audit record cannot be written, a governed run does not report success. Ungoverned programs do not acquire rule provenance merely by passing this flag.

Check the log’s chain

Verification checks the links between records and reports an error when the chain no longer matches. Use it to check the log before relying on its recorded history:

devlish audit-verify audit.jsonl

A hash chain alone cannot detect every change to the end of a log. Preserve the latest hash independently when you need to detect replacement or removal of its tail. Successful chain verification does not establish that the host recorded truthful inputs or enforced a protected execution environment.

Capture external responses

An audit record contains fingerprints of the run, but not all the data needed to repeat it. For example, an HTTP response may have changed by the time someone investigates the result. An effect journal archives the compiled bytecode, full input, and ordered requests and responses for supported host operations.

To keep that replay data when running the example, supply an attachment location with --journal:

devlish run pricing.dvl \
  --input '{"amount": 100}' \
  --audit-log audit.jsonl \
  --journal audit.jsonl.attachments

Attachments are addressed by their content hashes so replay can check that it is using the recorded data. Credentials are resolved below the journal boundary, but request bodies, response bodies, and inputs can still contain sensitive information. Store attachments with access appropriate to their contents.

Replay a recorded run

Replay runs the archived bytecode with the captured responses instead of calling the current external systems. It verifies the log chain and attachment hashes, checks that effect requests match in type, shape, and order, and compares the final output hash and instruction count.

Replay the log, or select a particular record with --line:

devlish replay audit.jsonl
devlish replay audit.jsonl --line 3

A matching replay shows that the archived program reproduces the recorded result from the captured data. It does not establish that the rule is correct for every input or that an external service’s response was accurate. Use Evidence Bundles to assess named test cases and Controlled Releases to track which artifact was approved.