Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Testing

Zingod has two kinds of test. The unit and model tests in packages/* run on the host with cargo nextest run --workspace and need no validator, although they are not network-free (zingod-grpc, for example, binds a loopback socket). Their blocks, headers and validators come from one builder, MockChain. The live tests in live-tests/ run a deployed zingod against a real zebrad on Kubernetes, driven by ztest.

SetWhat it covers
live-tests/clientlesszingod’s gRPC answers checked against the validator as oracle, with no wallet
live-tests/e2ea real wallet syncing and sending through zingod’s gRPC to a live validator
live-tests/non-finalized-statereorgs forced on a regtest zebrad, checked across every index (README)
live-tests/syncthe long-running mainnet profiles zingod_index_construction and zingod_index_crash_consistency

live-tests/ is a standalone Cargo workspace, and live-tests/sync/ a second one nested inside it, each with its own lock. The rules for writing live tests: live-tests/CLAUDE.md.

Model tests in bulk

Every model suite runs at its in-code case count under cargo nextest run --workspace. To run them harder, use the models nextest profile (.config/nextest.toml), which selects only the model suites and lifts the default profile’s 60 s per-test kill, and set PROPTEST_CASES:

PROPTEST_CASES=5000 cargo nextest run --profile models

PROPTEST_MAX_SHRINK_ITERS shrinks a failure harder and PROPTEST_VERBOSE=1 prints each case. Raising PROPTEST_CASES across the whole workspace also scales transactions_decode_to_their_source (zingod-source), which is slow.

A new model suite joins the profile by its name: its test path contains model or random_histories. Its oracle is written independently of the code under test, and moves the model cannot apply are clamped rather than filtered out, so shrinking stays effective. A failing case is saved and replays on every run: beside an integration test as <test>.proptest-regressions, or under the crate’s proptest-regressions/. Commit it with the fix.

Heavy run

A change to zingod-persistence, zingod-header-chain, zingod-nfs, zingod-grpc-snapshot or zingod-traffic is done only once this passes for that crate. Each round runs the crate’s whole suite on fresh seeds, so the timing-sensitive tests (background merges, failed I/O, crash states) repeat alongside the models. Then the crates built on it run once:

crate=zingod-persistence
end=$((SECONDS + 180)); round=0
while [ $SECONDS -lt $end ]; do
  round=$((round + 1))
  PROPTEST_CASES=1000 cargo nextest run -p "$crate" --all-features --profile heavy \
    || { echo "$crate: FAILED in round $round"; break; }
  echo "$crate: round $round passed"
done

cargo nextest run --workspace -E "rdeps($crate)"   # after the loop passes
  • --profile heavy lifts the default profile’s 60 s kill (at 1000 cases a model test outlives it): a test still running at 300 s is reported slow, never killed.
  • --all-features builds the crate with every optional feature on.
  • Lengthen the run by raising 180, never shorten it. Keep the test profile (no --release): overflow checks stay on.
  • A failure saves its shrunk case (above). Fix the bug and rerun the whole block.

Commands

cargo nextest run --workspace              # from the repo root

cd live-tests
ztest run                                  # clientless, e2e and non-finalized-state
ztest run -p clientless                    # one partition
ztest run --rerun latest -p clientless     # the last run's failures
ztest run --no-cleanup -p clientless       # keep each test's namespace (pods, logs) to inspect

cd sync                                    # now in live-tests/sync
ztest sync start zingod_index_construction --watch
ztest sync start zingod_index_crash_consistency

ztest run MUST be launched from live-tests/ and ztest sync from live-tests/sync/: neither finds its tests from any other directory. ztest run takes cargo nextest run arguments, but its own flags such as --rerun and --cluster MUST come before them.

The ztest CLI

Both live workspaces depend on the published ztest crate (ztest = "0.2.0" in each Cargo.toml). Install the matching CLI from crates.io:

cargo install --locked ztest_cli

The sync profiles build zebrad’s image from a zebra checkout beside this repository.

Cluster setup

Any reachable Kubernetes cluster works. For a local kind cluster:

kind create cluster --name ztest
ztest cluster add kind --kind ztest --set-default   # profile "kind" on kind cluster "ztest"
ztest cluster setup                                 # provisions ztest's resources; takes a few minutes
ztest cluster check                                 # read-only readiness report

The sync profiles restore validator chain-state snapshots of 40 to 200 GB. A CSI driver with copy-on-write VolumeSnapshot support makes that restore fast (without one the snapshot is copied); the other live tests need neither. Pick the driver when you add the cluster profile:

ztest cluster add kind --kind ztest --storage-driver topolvm.io --set-default                    # LVM thin pools
ztest cluster add kind --kind ztest --storage-driver rook-ceph.cephfs.csi.ceph.com --set-default # shared cluster