Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Zingod

Zingod is a Zcash indexer for light wallets. It follows the chain from one or more trusted validators (Zebra or Zakura), optionally also from Zcash p2p peers, builds its own indexes, and serves the lightwalletd-compatible CompactTxStreamer gRPC service.

The validator validates. Zingod indexes and serves.

  • Wallet developers: connecting, how each method behaves, and the status codes to handle.
  • Operators: installing, configuring, monitoring and exposing a Zingod server.
  • Contributors: how Zingod is built, and how it is tested.

Source and releases: github.com/zingolabs/zingod.

Connecting

Zingod serves the CompactTxStreamer gRPC service from the LightWallet Protocol, the same service as lightwalletd. A wallet that works with lightwalletd works with Zingod without changes. Zingod does not serve JSON-RPC.

Transport

gRPC over HTTP/2 on a single port, 8137 by default.

Public servers use TLS with a CA-issued certificate. Plaintext is for a wallet on the same host, or for a server behind the operator’s own TLS-terminating proxy. An operator can also run a self-signed certificate; only clients that pin its fingerprint can connect to it.

Browser clients

If the operator enables gRPC-web, browsers can connect on the same port. CORS allows any origin. With gRPC-web off, browser requests fail CORS. Native gRPC clients work either way.

  • Over HTTP/1.1, each in-flight stream uses its own connection, and connections are limited per client address. Connect over TLS so the browser can use HTTP/2.
  • Operators can block an origin. Requests from a blocked origin fail with PERMISSION_DENIED.

Server info

GetLightdInfo is answered from Zingod’s own state, so it works even when the server’s validators are unreachable.

FieldValue
vendorzingod
chainNamemain, test or regtest
blockHeightthe served tip (see heights)
estimatedHeightthe best block Zingod has verified; higher than blockHeight while syncing
consensusBranchIdthe consensus branch at blockHeight
upgradeName, upgradeHeightthe next network upgrade after blockHeight, or empty and 0
lightwalletProtocolVersionthe protocol release Zingod implements

Connection limits

Defaults are shown; operators can change them.

LimitDefault
concurrent streams per connection100
connections per client address128
connection lifetime30 minutes (±10%), then HTTP/2 GOAWAY
idle connectionclosed after 30 minutes
in-flight streams after a GOAWAY10 minutes to finish
client not reading a streamconnection closed after 5 minutes
stream with nothing to sendends UNAVAILABLE after 5 minutes (except GetMempoolStream)

Streams have no total time limit, so a full-chain GetBlockRange runs for as long as the client keeps reading. gRPC libraries reconnect after a GOAWAY on their own. A server being taken out of service also sends GOAWAY, so reconnecting is how a wallet moves to another server.

Health check

The gRPC port also answers two HTTP paths, useful for picking a server:

  • GET /readyz: 200 when the server is ready for wallets, otherwise 503.
  • GET /status: JSON with the server’s state, tip and served methods (format).

Methods

The protocol files define every method. This page covers only behaviour the protocol leaves open, and the places where Zingod differs from lightwalletd.

Heights

The served tip is the block Zingod currently answers at. Every index holds it.

  • Methods that take no height answer at the served tip: GetLatestBlock, GetLatestTreeState, GetSubtreeRoots, the address methods, and GetLightdInfo.blockHeight. A wallet that only asks for heights up to GetLatestBlock always gets an answer.
  • Methods that take a height (GetBlock, GetBlockRange, GetTreeState) answer up to the tip of the index they read. While the server is syncing, that can be above the served tip.
  • If a BlockID has both a height and a hash, the hash is used.
  • Unmined transactions are reported at height 0.

Blocks

GetBlockRange has no limit on range size, and it never returns fewer blocks than asked for without an error. If the range extends past the tip, the stream sends every block up to the tip and then ends with OUT_OF_RANGE. If the range starts past the tip, it fails with OUT_OF_RANGE before sending anything.

GetBlock returns every pool, including transparent inputs and outputs, as lightwalletd does. It takes no poolTypes.

CompactTx.fee is filled in for mined and mempool transactions. lightwalletd leaves it 0. Zingod reports 0 for coinbase transactions, for a wallet’s own transaction before another node lists it, and for fees of 2^32 zatoshis or more, which do not fit the uint32 field.

Tree state

GetTreeState answers for exactly the requested height. The block hash is in display order. Each tree field holds a hex-encoded CommitmentTree:

  • "" if the pool is not active at that height.
  • "000000" if the pool is active but empty.

GetSubtreeRoots either streams the whole answer or fails before the first message. A startIndex past the last root returns an empty stream. maxEntries 0 returns every root. An unknown shieldedProtocol fails with UNIMPLEMENTED.

Transactions

GetTransaction accepts only a txid (TxFilter.hash). A (block, index) filter fails with INVALID_ARGUMENT. Zingod asks its validators for the transaction and returns NOT_FOUND only when every validator asked reports it missing. If no validator responds, Zingod serves the transaction from its mempool or fetches its block from p2p peers; if neither works, the call fails with UNAVAILABLE.

SendTransaction reports a rejected transaction in the response, as lightwalletd does. The call returns OK with:

errorCodeMeaning
0accepted; errorMessage is the txid, quoted, in display order
-22the transaction could not be parsed
-25the transaction has expired, or targets the wrong consensus branch
otherthe validator rejected it; errorMessage is the validator’s message verbatim

The call itself fails with UNAVAILABLE only when no validator accepted the transaction and at least one could not be reached. Retry in that case.

Mempool

The mempool Zingod serves contains transactions listed by a trusted validator, transactions a quorum of its p2p peers agree on, and transactions submitted through this server. Both mempool methods fail with UNAVAILABLE when no trusted validator at the best block is reachable and Zingod has too few p2p peers for a quorum.

GetMempoolStream starts with the current mempool, then sends each new transaction once, each with height 0. The stream ends with OK when a new block arrives. By then the server already serves that block, so a GetLatestBlock sent next returns it.

GetMempoolTx accepts poolTypes like GetBlockRange.

Transparent addresses

The address methods (GetAddressUtxos, GetAddressUtxosStream, GetTaddressBalance, GetTaddressBalanceStream, GetTaddressTransactions) answer at the served tip.

  • A request covering more than 100,000 received outputs (the default limit) fails with RESOURCE_EXHAUSTED. Send fewer addresses per request.
  • An address for another network fails with INVALID_ARGUMENT.
  • Do not page GetAddressUtxos by startHeight. It is not a unique resume key and paging can stall (lightwallet-protocol#33).

Address queries tell the server which addresses belong to one wallet. To find transparent activity privately, request the TRANSPARENT pool in GetBlockRange and match scripts locally. Coinbase inputs are omitted (a coinbase is the transaction at CompactTx.index 0), and CompactTxIn carries no value, so balances still require the address methods.

Fees

GetStandardFee returns the ZIP 317 marginal fee per logical action, in zatoshis, for the next block. The estimator version is 0.

Pool filtering

poolTypes on GetBlockRange and GetMempoolTx selects the pools returned:

  • Empty: Sapling, Orchard and Ironwood. Transparent data is excluded.
  • Otherwise: exactly the listed pools.
  • POOL_TYPE_INVALID or an unknown value fails with INVALID_ARGUMENT.

GetBlockRangeNullifiers ignores TRANSPARENT, as the protocol specifies.

Deprecated and unsupported

  • GetBlockRangeNullifiers and GetTaddressTxids are deprecated and will be removed. Use GetBlockRange with shielded poolTypes, and GetTaddressTransactions.
  • GetBlockNullifiers and Ping are not supported and return UNIMPLEMENTED.
  • Operators can disable individual indexes. Methods that depend on a disabled index return UNIMPLEMENTED. /status lists the methods each enabled index serves.

Status codes

How to handle each gRPC status Zingod returns. A rejected SendTransaction is not an error status; it is reported in the response (details).

CodeWhenClient action
UNAVAILABLEthe server is starting, at capacity, or cannot reach its data sources; a stream had nothing to send for 5 minutesretry with backoff
NOT_FOUNDthe height, hash or txid is not in the chain, or is above the index’s tipdo not retry the same request
OUT_OF_RANGEGetBlockRange reached the tip, after sending every block up to itretry once the tip advances
INVALID_ARGUMENTmalformed request, height above the protocol maximum, invalid or wrong-network address, invalid poolTypesfix the request
RESOURCE_EXHAUSTEDrequest body too large (64 KiB; 2 MB + 1 KiB for SendTransaction), or too many address outputs in one requestsend less per request
DEADLINE_EXCEEDEDthe request’s grpc-timeout passed, or the request body took more than 30 seconds to arriveretry with a longer deadline
UNIMPLEMENTEDthe method or shieldedProtocol is unsupported, or its index is disableddo not retry
PERMISSION_DENIEDthe browser origin is blocked by the operatordo not retry
INTERNALserver faultreport to the operator

When the server is at capacity, the UNAVAILABLE response carries grpc-retry-pushback-ms (250). Wait at least that long before retrying.

Protocol files

The wallet API is defined by the LightWallet Protocol .proto files and their CHANGELOG. There is no ZIP for it. Zingod implements one protocol release and reports it in GetLightdInfo.lightwalletProtocolVersion.

The files below are the ones Zingod builds from. They differ from upstream in service.proto:

  • Ping and GetBlockNullifiers are removed, and return UNIMPLEMENTED.
  • GetBlockRangeNullifiers and GetTaddressTxids are marked deprecated.

service.proto

// Copyright (c) 2019-2020 The Zcash developers
// Distributed under the MIT software license, see the accompanying
// file COPYING or https://www.opensource.org/licenses/mit-license.php .

syntax = "proto3";
package cash.z.wallet.sdk.rpc;
option go_package = "lightwalletd/walletrpc";
option swift_prefix = "";
import "compact_formats.proto";

// An identifier for a Zcash value pool.
enum PoolType {
    POOL_TYPE_INVALID = 0;
    TRANSPARENT = 1;
    SAPLING = 2;
    ORCHARD = 3;
    IRONWOOD = 4;
}

// A BlockID message contains identifiers to select a block: a height or a
// hash. Support for specification by hash is not mandatory. (If `hash` is
// non-empty, the rpc may return an error.) This field is present to support
// a possible future upgrade.
message BlockID {
    uint64 height = 1;
    bytes hash = 2;
}

// BlockRange specifies a series of blocks from start to end inclusive.
// Both BlockIDs must be heights; specification by hash is not yet supported.
//
// If no pool types are specified, the server should default to the legacy
// behavior of returning only data relevant to the shielded (Sapling, Orchard,
// and Ironwood) pools; otherwise, the server should prune `CompactBlock`s
// returned to include only data relevant to the requested pool types. Clients MUST
// verify that the version of the server they are connected to are capable
// of returning pruned and/or transparent data before setting `poolTypes`
// to a non-empty value.
message BlockRange {
    BlockID start = 1;
    BlockID end = 2;
    repeated PoolType poolTypes = 3;
}

// A TxFilter contains the information needed to identify a particular
// transaction: either a block and an index, or a direct transaction hash.
// Currently, only specification by hash is supported.
message TxFilter {
     BlockID block = 1;     // block identifier, height or hash
     uint64 index = 2;      // index within the block
     bytes hash = 3;        // transaction ID (hash, txid)
}

// RawTransaction contains the complete transaction data. It also optionally includes
// the block height in which the transaction was included, or, when returned
// by GetMempoolStream(), the latest block height.
//
// FIXME: the documentation here about mempool status contradicts the documentation
// for the `height` field. See https://github.com/zcash/librustzcash/issues/1484
message RawTransaction {
    // The serialized representation of the Zcash transaction.
    bytes data = 1;
    // The height at which the transaction is mined, or a sentinel value.
    //
    // Due to an error in the original protobuf definition, it is necessary to
    // reinterpret the result of the `getrawtransaction` RPC call. Zcashd will
    // return the int64 value `-1` for the height of transactions that appear
    // in the block index, but which are not mined in the main chain. Here, the
    // height field of `RawTransaction` was erroneously created as a `uint64`,
    // and as such we must map the response from the zcashd RPC API to be
    // representable within this space. Additionally, the `height` field will
    // be absent for transactions in the mempool, resulting in the default
    // value of `0` being set. Therefore, the meanings of the `height` field of
    // the `RawTransaction` type are as follows:
    //
    // * height 0: the transaction is in the mempool
    // * height 0xffffffffffffffff: the transaction has been mined on a fork that
    //   is not currently the main chain
    // * any other height: the transaction has been mined in the main chain at the
    //   given height
    uint64 height = 2;
}

// A SendResponse encodes an error code and a string. It is currently used
// only by SendTransaction(). If error code is zero, the operation was
// successful; if non-zero, it and the message specify the failure.
message SendResponse {
    int32 errorCode = 1;
    string errorMessage = 2;
}

// Chainspec is a placeholder to allow specification of a particular chain fork.
message ChainSpec {}

// Empty is for gRPCs that take no arguments, currently only GetLightdInfo.
message Empty {}

// LightdInfo returns various information about this lightwalletd instance
// and the state of the blockchain.
message LightdInfo {
    string version = 1;
    string vendor = 2;
    bool   taddrSupport = 3;            // true
    string chainName = 4;               // either "main" or "test"
    uint64 saplingActivationHeight = 5; // depends on mainnet or testnet
    string consensusBranchId = 6;       // protocol identifier, see consensus/upgrades.cpp
    uint64 blockHeight = 7;             // latest block on the best chain
    string gitCommit = 8;
    string branch = 9;
    string buildDate = 10;
    string buildUser = 11;
    uint64 estimatedHeight = 12;        // less than tip height if zcashd is syncing
    string zcashdBuild = 13;            // example: "v4.1.1-877212414"
    string zcashdSubversion = 14;       // example: "/MagicBean:4.1.1/"
    string donationAddress = 15;        // Zcash donation UA address
    string upgradeName = 16;            // name of next pending network upgrade, empty if none scheduled
    uint64 upgradeHeight = 17;          // height of next pending upgrade, zero if none is scheduled
    string lightwalletProtocolVersion = 18; // version of https://github.com/zcash/lightwallet-protocol served by this server
}

// TransparentAddressBlockFilter restricts the results of the GRPC methods that
// use it to the transactions that involve the given address and were mined in
// the specified block range. Non-default values for both the address and the
// block range must be specified. Mempool transactions are not included.
//
// The `poolTypes` field of the `range` argument should be ignored.
// Implementations MAY consider it an error if any pool types are specified.
message TransparentAddressBlockFilter {
    string address = 1;     // t-address
    BlockRange range = 2;   // start, end heights only
}

message Address {
    string address = 1;
}
message AddressList {
    repeated string addresses = 1;
}
message Balance {
    int64 valueZat = 1;
}

// Request parameters for the `GetMempoolTx` RPC.
message GetMempoolTxRequest {
    // A list of transaction ID byte string suffixes that should be excluded
    // from the response. These suffixes may be produced either directly from
    // the underlying txid bytes, or, if the source values are encoded txid
    // strings, by truncating the hexadecimal representation of each
    // transaction ID to an even number of characters, and then hex-decoding
    // and then byte-reversing this value to obtain the byte representation.
    repeated bytes exclude_txid_suffixes = 1;
    // We reserve field number 2 for a potential future `exclude_txid_prefixes`
    // field.
    reserved 2;
    // The server must prune `CompactTx`s returned to include only data
    // relevant to the requested pool types. If no pool types are specified,
    // the server should default to the legacy behavior of returning only data
    // relevant to the shielded (Sapling, Orchard, and Ironwood) pools.
    repeated PoolType poolTypes = 3;
}

// The TreeState is derived from the Zcash z_gettreestate rpc.
message TreeState {
    string network = 1;     // "main" or "test"
    uint64 height = 2;      // block height
    string hash = 3;        // block id
    uint32 time = 4;        // Unix epoch time when the block was mined
    string saplingTree = 5; // sapling commitment tree state
    string orchardTree = 6; // orchard commitment tree state
    string ironwoodTree = 7; // ironwood commitment tree state
}

enum ShieldedProtocol {
    sapling = 0;
    orchard = 1;
    ironwood = 2;
}

message GetSubtreeRootsArg {
    uint32 startIndex = 1;                  // Index identifying where to start returning subtree roots
    ShieldedProtocol shieldedProtocol = 2;  // Shielded protocol to return subtree roots for
    uint32 maxEntries = 3;                  // Maximum number of entries to return, or 0 for all entries.
}
message SubtreeRoot {
    bytes rootHash = 2;                     // The 32-byte Merkle root of the subtree.
    bytes completingBlockHash = 3;          // The hash of the block that completed this subtree.
    uint64 completingBlockHeight = 4;       // The height of the block that completed this subtree in the main chain.
}

// Results are sorted by height, which makes it easy to issue another
// request that picks up from where the previous left off.
message GetAddressUtxosArg {
    repeated string addresses = 1;
    uint64 startHeight = 2;
    uint32 maxEntries = 3; // zero means unlimited
}
message GetAddressUtxosReply {
    string address = 6;
    bytes txid = 1;
    int32 index = 2;
    bytes script = 3;
    int64 valueZat = 4;
    uint64 height = 5;
}
message GetAddressUtxosReplyList {
    repeated GetAddressUtxosReply addressUtxos = 1;
}

// StandardFee is the reply to the GetStandardFee rpc: the ZIP 317
// marginal fee, in zatoshis per logical action, that wallets should
// pay for a transaction to be mined in the next block, as reported by
// the backend's `getstandardfee` RPC.
message StandardFee {
    uint64 feeZat = 1;           // standard fee in zatoshis per logical action
    uint32 estimatorVersion = 2; // backend fee-estimator version, 0 = static ZIP 317 marginal fee
}

service CompactTxStreamer {
    // Return the BlockID of the block at the tip of the best chain
    rpc GetLatestBlock(ChainSpec) returns (BlockID) {}

    // Return the compact block corresponding to the given block identifier.
    //
    // The returned `CompactBlock` includes transaction data for all value
    // pools, including transparent inputs (`vin`) and outputs (`vout`). This
    // differs from `GetBlockRange`, which supports filtering by pool type and
    // defaults to returning only shielded (Sapling, Orchard, and Ironwood)
    // data. Clients that require only data for specific pools should use
    // `GetBlockRange` with the appropriate `poolTypes` set.
    //
    // Note: the single null-outpoint input for coinbase transactions is
    // omitted from the `vin` field of the corresponding `CompactTx`. See the
    // documentation of the `CompactTx` message for details.
    rpc GetBlock(BlockID) returns (CompactBlock) {}

    // Return a list of consecutive compact blocks in the specified range,
    // which is inclusive of `range.end`.
    //
    // If range.start <= range.end, blocks are returned increasing height order;
    // otherwise blocks are returned in decreasing height order.
    rpc GetBlockRange(BlockRange) returns (stream CompactBlock) {}

    // !!! TODO: REMOVE THIS. DEPRECATED. Served by Zingod only because pepper-sync still calls it;
    // !!! delete once pepper-sync requests `GetBlockRange` with `poolTypes` instead.
    //
    // Return a stream of compact blocks for the specified range, where each
    // block contains only nullifier information for the shielded pools
    // (Sapling spend nullifiers, Orchard action nullifiers, and Ironwood action
    // nullifiers). Transparent transaction data, Sapling outputs, full
    // Orchard/Ironwood action data, and commitment tree sizes are not included.
    // Implementations MUST ignore any
    // `PoolType::TRANSPARENT` member of the `poolTypes` field of the request.
    //
    // Note: this method is deprecated; use `GetBlockRange` with the
    // appropriate `poolTypes` instead.
    rpc GetBlockRangeNullifiers(BlockRange) returns (stream CompactBlock) {
      option deprecated = true;
    }

    // Return the requested full (not compact) transaction (as from zcashd)
    rpc GetTransaction(TxFilter) returns (RawTransaction) {}

    // Submit the given transaction to the Zcash network
    rpc SendTransaction(RawTransaction) returns (SendResponse) {}

    // !!! TODO: REMOVE THIS. DEPRECATED. Served by Zingod only because pepper-sync still calls it;
    // !!! delete once pepper-sync calls `GetTaddressTransactions` (same request, same response).
    //
    // Return RawTransactions that match the given transparent address filter.
    //
    // Note: This function is misnamed, it returns complete `RawTransaction` values, not TxIds.
    // NOTE: this method is deprecated, please use GetTaddressTransactions instead.
    rpc GetTaddressTxids(TransparentAddressBlockFilter) returns (stream RawTransaction) {
      option deprecated = true;
    }

    // Return the transactions corresponding to the given t-address within the given block range.
    // Mempool transactions are not included in the results.
    rpc GetTaddressTransactions(TransparentAddressBlockFilter) returns (stream RawTransaction) {}

    rpc GetTaddressBalance(AddressList) returns (Balance) {}
    rpc GetTaddressBalanceStream(stream Address) returns (Balance) {}

    // Returns a stream of the compact transaction representation for transactions
    // currently in the mempool. The results of this operation may be a few
    // seconds out of date. If the `exclude_txid_suffixes` list is empty,
    // return all transactions; otherwise return all *except* those in the
    // `exclude_txid_suffixes` list (if any); this allows the client to avoid
    // receiving transactions that it already has (from an earlier call to this
    // RPC). The transaction IDs in the `exclude_txid_suffixes` list can be
    // shortened to any number of bytes to make the request more
    // bandwidth-efficient; if two or more transactions in the mempool match a
    // txid suffix, none of the matching transactions are excluded. Txid
    // suffixes in the exclude list that don't match any transactions in the
    // mempool are ignored.
    rpc GetMempoolTx(GetMempoolTxRequest) returns (stream CompactTx) {}

    // Return a stream of current Mempool transactions. This will keep the output stream open while
    // there are mempool transactions. It will close the returned stream when a new block is mined.
    rpc GetMempoolStream(Empty) returns (stream RawTransaction) {}

    // GetTreeState returns the note commitment tree state corresponding to the given block.
    // See section 3.7 of the Zcash protocol specification. It returns several other useful
    // values also (even though they can be obtained using GetBlock).
    // The block can be specified by either height or hash.
    rpc GetTreeState(BlockID) returns (TreeState) {}
    rpc GetLatestTreeState(Empty) returns (TreeState) {}

    // Returns a stream of information about roots of subtrees of the note commitment tree
    // for the specified shielded protocol (Sapling, Orchard, or Ironwood).
    rpc GetSubtreeRoots(GetSubtreeRootsArg) returns (stream SubtreeRoot) {}

    rpc GetAddressUtxos(GetAddressUtxosArg) returns (GetAddressUtxosReplyList) {}
    rpc GetAddressUtxosStream(GetAddressUtxosArg) returns (stream GetAddressUtxosReply) {}

    // Return information about this lightwalletd instance and the blockchain
    rpc GetLightdInfo(Empty) returns (LightdInfo) {}

    // Return the standard fee (the ZIP 317 marginal fee, in zatoshis per
    // logical action) that the backend recommends wallets pay for a
    // transaction to be mined in the next block. Requires a backend that
    // implements the `getstandardfee` RPC. Backends that lack it return
    // Unimplemented.
    rpc GetStandardFee(Empty) returns (StandardFee) {}
}

compact_formats.proto

// Copyright (c) 2019-2021 The Zcash developers
// Distributed under the MIT software license, see the accompanying
// file COPYING or https://www.opensource.org/licenses/mit-license.php .

syntax = "proto3";
package cash.z.wallet.sdk.rpc;
option go_package = "lightwalletd/walletrpc";
option swift_prefix = "";

// REMINDER: proto3 fields are all optional. A field that is not present will be set to its zero/false/empty
// value.

// Information about the state of the chain as of a given block.
message ChainMetadata {
    uint32 saplingCommitmentTreeSize = 1; // the size of the Sapling note commitment tree as of the end of this block
    uint32 orchardCommitmentTreeSize = 2; // the size of the Orchard note commitment tree as of the end of this block
    uint32 ironwoodCommitmentTreeSize = 3; // the size of the Ironwood note commitment tree as of the end of this block
}

// A compact representation of a Zcash block.
//
// CompactBlock is a packaging of ONLY the data from a block that's needed to:
//   1. Detect a payment to your Shielded address
//   2. Detect a spend of your Shielded notes
//   3. Update your witnesses to generate new spend proofs.
//   4. Spend UTXOs associated to t-addresses of your wallet.
//
// Currently, the `header` field should always be unset (empty). In the future, 
// the presence or absence of header data may be made dependent on request 
// parameters, although it is likely that such flexibility will only be provided via 
// newly-added service methods, not via existing APIs.
message CompactBlock {
    // Field 1 (`protoVersion`) was removed; see
    // https://github.com/zcash/lightwallet-protocol/issues/25
    reserved 1;
    reserved "protoVersion";
    uint64 height = 2;               // the height of this block
    bytes hash = 3;                  // if present, the ID (hash) of this block
    bytes prevHash = 4;              // if present, the ID (hash) of this block's predecessor
    uint32 time = 5;                 // if non-zero, Unix epoch time when the block was mined
    bytes header = 6;                // if present, full header (as returned by the getblock RPC)
    repeated CompactTx vtx = 7;      // zero or more compact transactions from this block
    ChainMetadata chainMetadata = 8; // information about the state of the chain as of this block
}

// A compact representation of a Zcash transaction.
//
// CompactTx contains the minimum information for a wallet to know if this transaction
// is relevant to it (either pays to it or spends from it) via shielded elements. Additionally,
// it can optionally include the minimum necessary data to detect payments to transparent addresses
// related to your wallet.
message CompactTx {
    // The index of the transaction within the block.
    uint64 index = 1;

    // The id of the transaction as defined in
    // [§ 7.1.1 ‘Transaction Identifiers’](https://zips.z.cash/protocol/protocol.pdf#txnidentifiers)
    // This byte array MUST be in protocol order and MUST NOT be reversed
    // or hex-encoded; the byte-reversed and hex-encoded representation is
    // exclusively a textual representation of a txid.
    bytes txid = 2;

    // The transaction fee: present if server can provide. In the case of a
    // stateless server and a transaction with transparent inputs, this will be
    // unset because the calculation requires reference to prior transactions.
    // If there are no transparent inputs, the fee will be calculable as:
    //    valueBalanceSapling + valueBalanceOrchard + valueBalanceIronwood
    //    + sum(vPubNew) - sum(vPubOld) - sum(tOut)
    uint32 fee = 3;

    repeated CompactSaplingSpend spends = 4;
    repeated CompactSaplingOutput outputs = 5;
    repeated CompactOrchardAction actions = 6;
    repeated CompactOrchardAction ironwoodActions = 9;

    // `CompactTxIn` values corresponding to the `vin` entries of the full transaction.
    //
    // Note: the single null-outpoint input for coinbase transactions is omitted. Light
    // clients can test `CompactTx.index == 0` to determine whether a `CompactTx`
    // represents a coinbase transaction, as the coinbase transaction is always the
    // first transaction in any block.
    repeated CompactTxIn vin = 7;

    // A sequence of transparent outputs being created by the transaction.
    repeated TxOut vout = 8;
}

// A compact representation of a transparent transaction input.
message CompactTxIn {
    // The id of the transaction that generated the output being spent. This
    // byte array must be in protocol order and MUST NOT be reversed or
    // hex-encoded.
    bytes prevoutTxid = 1;

    // The index of the output being spent in the `vout` array of the
    // transaction referred to by `prevoutTxid`.
    uint32 prevoutIndex = 2;
}

// A transparent output being created by the transaction.
//
// This contains identical data to the `TxOut` type in the transaction itself, and
// thus it is not "compact".
message TxOut {
    // The value of the output, in Zatoshis.
    uint64 value = 1;

    // The script pubkey that must be satisfied in order to spend this output.
    bytes scriptPubKey = 2;
}

// A compact representation of a [Sapling Spend](https://zips.z.cash/protocol/protocol.pdf#spendencodingandconsensus).
//
// CompactSaplingSpend is a Sapling Spend Description as described in 7.3 of the Zcash
// protocol specification.
message CompactSaplingSpend {
    bytes nf = 1;   // Nullifier (see the Zcash protocol specification)
}

// A compact representation of a [Sapling Output](https://zips.z.cash/protocol/protocol.pdf#outputencodingandconsensus).
//
// It encodes the `cmu` field, `ephemeralKey` field, and a 52-byte prefix of the
// `encCiphertext` field of a Sapling Output Description. Total size is 116 bytes.
message CompactSaplingOutput {
    bytes cmu = 1;          // Note commitment u-coordinate.
    bytes ephemeralKey = 2; // Ephemeral public key.
    bytes ciphertext = 3;   // First 52 bytes of ciphertext.
}

// A compact representation of an [Orchard Action](https://zips.z.cash/protocol/protocol.pdf#actionencodingandconsensus).
message CompactOrchardAction {
    bytes nullifier = 1;        // [32] The nullifier of the input note
    bytes cmx = 2;              // [32] The x-coordinate of the note commitment for the output note
    bytes ephemeralKey = 3;     // [32] An encoding of an ephemeral Pallas public key
    bytes ciphertext = 4;       // [52] The first 52 bytes of the encCiphertext field
}

Quick start

Requirements

  • A Zcash validator with JSON-RPC enabled: Zebra (zebrad, tested with 6.2.3) or Zakura (zakurad). Set [rpc] listen_addr in its config.
  • Local storage for the indexes, preferably NVMe (first sync).

Zingod uses these Zcash JSON-RPC methods: getblock, getblockchaininfo, getblockhash, getblockheader, getdeprecationinfo, getinfo, getpeerinfo, getrawmempool, getrawtransaction and sendrawtransaction. If the validator restricts its RPC methods, allow these. Zingod also calls checkadmission when the validator supports it.

Run with Docker

  1. Generate a config with every key at its default:

    docker run --rm zingodevops/zingod generate-config -o /dev/stdout > zingod.toml
    
  2. Edit zingod.toml:

    • Set network to mainnet, testnet or regtest.
    • Set jsonrpc_address under [[trusted_validators]] to your validator.
    • Set tls under [serve]. The container listens on all interfaces, so Zingod requires it: tls = "off" if wallets connect only from this host, or a certificate pair (TLS).
  3. Start it:

    docker run -d --name zingod -p 127.0.0.1:8137:8137 \
      -v ./zingod.toml:/etc/zingod/zingod.toml:ro \
      -v zingod-data:/var/lib/zingod \
      zingodevops/zingod
    
  4. Wait for the first sync. It takes about an hour on NVMe with the validator on the same host, and longer over a network. An index snapshot skips most of it. Check progress over HTTP on the gRPC port:

    curl -s http://localhost:8137/status                     # state and tip
    curl -sf http://localhost:8137/readyz && echo serving    # 200 once ready
    

Wallets can then connect to http://localhost:8137. See Docker for volumes, TLS mounts and Compose.

Run on a host

Building requires protoc on PATH.

cargo install --locked --git https://github.com/zingolabs/zingod zingod
zingod generate-config      # writes $XDG_CONFIG_HOME/zingod/zingod.toml
zingod start                # or: zingod start -c /path/to/zingod.toml

Run it under a service manager such as systemd. zingod --help lists every subcommand.

Try it on the Public Testnet

zebrad.toml and zingod.toml are a matching pair of configs for the Public Testnet, with zebrad’s JSON-RPC on 127.0.0.1:18232.

  1. Install zebrad: cargo install zebrad --locked.
  2. In zebrad.toml, replace <PATH_TO_ZEBRA> with a writable directory. In zingod.toml, replace <ZINGOD_DATA>.
  3. Run zebrad -c zebrad.toml start.
  4. In another shell, run zingod start -c zingod.toml.

Zingod serves gRPC on 127.0.0.1:8137.

Configuration

Zingod reads one TOML file: $XDG_CONFIG_HOME/zingod/zingod.toml, or the path given with -c. zingod generate-config writes a file with every key at its default.

Settings are applied in this order, later ones winning:

  1. Built-in defaults.
  2. The config file.
  3. Environment variables prefixed ZINGOD_CONFIG_, with __ between nested keys. For example, ZINGOD_CONFIG_SYNC__CONCURRENCY=64 sets concurrency under [sync].

Every subcommand validates the whole config and refuses to run on an unknown key or invalid value.

Default paths are under the XDG base directories. If there is no home directory (for example, HOME unset under a service manager), set each path explicitly: serve.state_dir, the path of every enabled index, and p2p.cache_dir if [p2p] is enabled.

Overview

Key or tablePurpose
networkmainnet (default), testnet or regtest
validator_clienthow Zingod talks to validators (below)
[[trusted_validators]]one table per validator: address, credentials, timeouts, connection budget, priority
[submission]how SendTransaction retries relaying
[p2p]Zingod’s own Zcash p2p peers (off by default)
[serve]gRPC listen address, TLS, gRPC-web, state directory, drain delay
[grpc]connection and stream limits, timeouts, trusted proxies, overload thresholds
[metrics]admin listener for metrics and probes (off by default)
[sync]block fetching and finality depth
[index.<name>]per index: enabled and path
[snapshot]bootstrap empty indexes from an index snapshot

Trusted validators do not vote. Each one adds failover, so two validators survive one going down.

At least one index must be enabled. utxo_set is enabled together with compact_block.

Validator client

validator_client sets how Zingod talks to every trusted validator:

  • legacy_rpc (default): JSON-RPC for everything. Works with zebrad, zakurad and zcashd.
  • grpc: blocks over zebrad’s indexer gRPC at each validator’s indexer_address, everything else over JSON-RPC. Requires a zebrad build whose indexer serves GetBlockRange (Zingod’s Zebra fork; stock zebrad does not). Do not use it with zakurad, whose GetBlockRange returns a different message type.

The indexer gRPC connection is plaintext unless [trusted_validators.indexer_tls] configures mutual TLS.

Reference

Every key, with its default and a description. This is the Public Testnet zingod.toml (download).

# zingod config paired with zebrad.toml (Testnet)
#
# - Env overrides + unknown-key rule: https://zingolabs.github.io/zingod/operators/configuration.html
# - `zingod generate-config` = every key at its default

# mainnet | testnet | regtest (default mainnet)
# - declared, never read off the validator (Zebra on regtest reports "test")
network = "testnet"

# How zingod talks to every [[trusted_validators]] entry (default legacy_rpc)
# - legacy_rpc = JSON-RPC for everything (zebrad, zakurad or zcashd); blocks arrive as hex inside
#   JSON
# - grpc = blocks over zebrad's indexer gRPC (GetBlock, GetBlockRange) as stored bytes, no hex
#   or JSON; everything else over JSON-RPC. Needs each validator's indexer_address and a zebrad
#   serving GetBlockRange: Zingod's zebra fork only (elicbarbieri/zebra perf/indexer-block-range)
# validator_client = "grpc"

# Admin listener (/metrics, /livez, /readyz). Omit listen_address to disable.
# [metrics]
# listen_address = "127.0.0.1:9999"

# Validators Zingod trusts (Zebra or Zakura JSON-RPC), at least one; no vote (`priority` orders who
# is asked)
# - tip = their most-work header (trusted: zebra validated it), served while any of them holds it
# - one listing admits a mempool transaction; submission: see [submission]
# - 2+ = survives one going down (no vote: more never weakens it)
# - down at boot or later = retried, never fatal
[[trusted_validators]]
jsonrpc_address = "127.0.0.1:18232"
# No credentials + a loopback address = no auth until the validator refuses, then its default
# cookie: <cache dir>/zebra/.cookie or <cache dir>/zakura/.cookie (both = the one written last)
# cookie_path = "<PATH_TO_ZEBRA>/.cookie"
# basic auth
# user = "xxxxxx"
# password = "xxxxxx"
# connect_timeout_secs = 2
# Silence before a request fails, never total duration (raise for a slow or lossy link)
# read_timeout_secs = 30
# Link budget to this validator (zingod never exceeds it): at least 5 connections, one for the poll,
# one reserved per lane (control, interactive, bulk) + one shared, the rest to whichever lane
# waits
# max_connections = 32
# Largest JSON-RPC reply read, 4 (one full block) ..= 32; batches are split to fit
# max_response_mib = 32
# zebrad's indexer gRPC (its `indexer_listen_addr`): tip + mempool push streams wake polling at
# once; polling slows to a 15 s reconcile while they are up. Unset = poll every second.
# validator_client = "grpc" fetches blocks here too (then required)
# indexer_address = "127.0.0.1:8230"
# Tier: every 0 asked before any 1 (a farther validator = 1: overflow + hedges only)
# priority = 0
# Mutual TLS to indexer_address (zakurad requires it off loopback: its [rpc.indexer_tls], whose
# client_ca_file signs cert_path). Unset = plaintext.
# [trusted_validators.indexer_tls]
# ca_path = "/etc/zingod/indexer-ca.pem"
# cert_path = "/etc/zingod/indexer-client.pem"
# key_path = "/etc/zingod/indexer-client-key.pem"
# Name on the validator's certificate (unset = the indexer_address host)
# server_name = "zakura-1.internal"

# [[trusted_validators]]
# jsonrpc_address = "127.0.0.1:18242"

# SendTransaction: a trusted validator's checkadmission answers the wallet (refused = that
# refusal, nothing sent); then one relay per attempt, through a random trusted validator, or
# the peers with [p2p] on (defaults shown)
[submission]
# Seconds an accepted transaction gets to be seen beyond the nodes relayed to before relaying again
propagation_threshold_secs = 15
# Relays per transaction at most; without a check, the wallet is answered at the first acceptance
max_attempts = 4

# Zingod's own peers on the Zcash p2p network (zebra-network), off by default: headers and blocks
# (checked by proof of work and merkle root: the tip keeps up while every trusted validator is
# down; finality too, once peers in more than 5 netgroups carry the chain), each peer's mempool
# polled (a transaction > 2/3 of them list from > 5 netgroups is served after Zingod's consensus
# checks: needs index.compact_block), and SendTransaction relays (zebra's own, to a third of the
# peers per attempt; then one trusted validator if still unseen). Zingod serves no peer (listener
# on loopback, no services advertised).
# [p2p]
# enabled = true
# Each peer holds one inbound slot on a public node: keep it modest
# peer_target = 8
# Empty = the network's DNS seeders (regtest has none: list peers)
# initial_peers = []
# cache_dir = "<ZINGOD_DATA>/peers"

[serve]
# CompactTxStreamer gRPC over HTTP/2
grpc_listen_address = "127.0.0.1:8137"
# serve browser (gRPC-web) clients too: permissive CORS (*) on this listener, native gRPC
# untouched; refuse origins with `zingod origin`. off = no CORS, browsers refused
grpc_web = false
# Max transparent receives one address request walks (all its addresses); over = RESOURCE_EXHAUSTED
# max_address_rows = 100000
# zingod's own state, kept across restarts (`zingod hold` writes its marker here);
# default <data dir>/zingod/state
# state_dir = "/var/lib/zingod"
# held: /readyz fails with "held" while still serving this long, then every open gRPC connection
# gets a GOAWAY; >= the balancer's down detection + DNS TTL
hold_goaway_delay_secs = 0

# TLS: unset = "off" on a loopback grpc_listen_address, refused at boot on any other
# - { cert_path, key_path } = PEM files, re-read on change (renewal = no restart)
# - "off" = plaintext (a TLS-terminating proxy in front; warned at boot off loopback)
# - "self_signed" = pair generated under <state_dir>/tls at first boot (fingerprint logged; only a
#   client that pins it connects)
# tls = { cert_path = "/etc/zingod/fullchain.pem", key_path = "/etc/zingod/key.pem" }

# Serving caps (defaults shown); every cap refuses rather than queues, except the read lanes
[grpc]
# excess accepts closed; + 1024 must fit the open-file limit or boot fails
max_connections = 4096
# per client: the peer, or the address a trusted proxy's PROXY header names
max_connections_per_ip = 128
# HTTP/2 SETTINGS_MAX_CONCURRENT_STREAMS: every stream on one connection, of any method (as
# lightwalletd's grpc-max-concurrent-streams)
max_streams_per_connection = 100
# work streams (every method but GetMempoolStream) across all connections; excess = UNAVAILABLE +
# retry hint
max_work_streams = 2048
# work streams admitted at once follow measured latency, max_work_streams the ceiling: a saturated
# node refuses new work within seconds (UNAVAILABLE + retry hint); false = always max_work_streams
adaptive_work_streams = true
# GetMempoolStream subscriptions (idle until the next block), counted apart from max_work_streams
max_subscriptions = 4096
# index reads in flight on the blocking pool, per lane; excess waits in its own lane
# - point: one block / tree state / root list; range: one GetBlockRange window; scan: one address
#   history
max_point_reads = 64
max_range_reads = 32
max_scan_reads = 4
# no-progress bounds, never a cap on a stream's length (the client's grpc-timeout is its deadline)
# data its client does not read for this long closes its connection
unread_timeout_secs = 300
# a stream with nothing to send for this long ends UNAVAILABLE (GetMempoolStream exempt)
silent_timeout_secs = 300
# GOAWAY after this long (±10% per connection): the client reconnects and re-resolves DNS, so a
# load balancer or a replaced node moves long-held wallet connections; 0 = never
max_connection_age_secs = 1800
# in-flight streams of a GOAWAY'd connection get this long to finish (a whole GetBlockRange sync)
max_connection_age_grace_secs = 600
# GOAWAY a connection with no open stream for this long (frees its per-client slot); 0 = never
max_connection_idle_secs = 1800
# proxies that open every connection with a PROXY header (v1/v2)
# - https://zingolabs.github.io/zingod/operators/network.html#behind-a-proxy
trusted_proxies = []
# SIGTERM/SIGINT closes the listener at once; open connections get this long to finish (GOAWAY
# sent). Out of a load balancer's rotation first = `zingod hold`
# - https://zingolabs.github.io/zingod/operators/maintenance.html#operator-hold
shutdown_grace_secs = 10

# When /readyz reports `Overloaded: <cause>` (0 = that cause off; `zingod overload` overrides live)
[grpc.overload]
# share of each minute zingod's cgroup waited (Linux PSI)
cpu_pressure_percent = 25
io_pressure_percent = 25

# One block-fetch pipeline, shared by every index (defaults shown; all non-zero)
[sync]
# Blocks below the tip kept reorg-able (in memory, served, written once buried); >= 1000 off regtest
finalised_depth = 1000
# Block fetches (and folds) in flight ahead of the next one needed, over every trusted validator
concurrency = 32
# MiB of finalized blocks one index may lag the fetch before it throttles the pipeline
queue_mib = 256

# The disk engine's map tables (one engine per process: every index shares merge_slots)
# [sync.lsm_store]
# Same-tier segments merged into one (>= 2; higher = less rewriting, larger merges)
# fanout = 16
# Merges doing disk I/O at once (the rest wait, smallest first)
# merge_slots = 4
# MiB/s every merge together may read + write (commits are never paced)
# merge_mib_per_sec = 200

# One table per index: `enabled` (default true; off = its methods UNIMPLEMENTED) and `path`
# (default $XDG_DATA_HOME/zingod/indexes/<name>); an omitted table takes both defaults
[index.compact_block]
# GetBlock, GetBlockRange (GetLatestBlock, GetLightdInfo = the served tip, either way)
# - its fee index (each CompactTx.fee) lives beside it, in `utxo_set`
enabled = true
path = "<ZINGOD_DATA>/compact_block"

[index.block_hash]
# GetBlock / GetTreeState by hash (by height unaffected)
enabled = true
path = "<ZINGOD_DATA>/block_hash"

[index.tree_state]
# GetTreeState, GetLatestTreeState, GetSubtreeRoots
enabled = true
path = "<ZINGOD_DATA>/tree_state"

[index.transparent_address]
# GetAddressUtxos(+Stream), GetTaddressBalance(+Stream), GetTaddressTransactions
enabled = true
path = "<ZINGOD_DATA>/transparent_address"

[index.txid_location]
# GetTransaction / GetTaddressTransactions bytes with no validator answering: the block it names,
# fetched from p2p peers
enabled = true
path = "<ZINGOD_DATA>/txid_location"

# Fill empty indexes from a published snapshot before boot (aria2c on PATH)
# [snapshot]
# manifest = "https://snapshots.zingolabs.dev/zingod-snapshot-1.0.0/manifest.json"
# connections = 8   # total across every archive

First sync

On first start, each index syncs from genesis. On Mainnet or Testnet this takes a while. After a restart, each index resumes where it stopped.

Zingod serves wallets while it syncs, at the height every index has reached (heights).

Storage

Keep the indexes on local storage, preferably NVMe. Zingod reads them through mmap, so network storage performs poorly.

Every index file carries page checksums. zingod verify finds a damaged index; delete its directory and Zingod resyncs it.

Index snapshot

A published snapshot fills empty indexes before Zingod starts, which skips most of the first sync. It needs aria2c on PATH; the Docker image and the Nix package include it.

[snapshot]
manifest = "https://snapshots.zingolabs.dev/zingod-snapshot-1.0.0/manifest.json"
connections = 8   # total across all archives, at most 16
  • Only enabled indexes whose directories are missing or empty are filled. Indexes holding data are left alone, and if none are empty nothing is downloaded.
  • Free space is checked before downloading. If it is short, startup stops and names the directory and the space needed.
  • Transient download errors are retried. A restart resumes each archive where it stopped.
  • Startup stops on a manifest for another network, a hash mismatch, a permanent download error, or an empty index the manifest has no archive for.

While the snapshot installs, the state is provisioning, with progress in detail (states).

The manifest format, for publishing your own snapshot:

{"network": "...", "height": 0,
 "indexes": {"<index>": {"archive": "...", "sha256": "...", "archive_bytes": 0, "installed_bytes": 0}}}

All archives come from one node at one height. Each is a zstd-compressed tar of the index directory’s contents, with no top-level directory. archive is resolved relative to the manifest URL, and installed_bytes is its unpacked size.

Monitoring

States

Zingod reports a single state. The same value appears in /readyz, /status, the zingod_state{state} metric and the State changed log line. The first matching row applies:

StateMeaning/readyz
heldan operator ran zingod hold503
downcannot serve: the server is stalled or overloaded, startup is waiting for a validator, or the tip is unverified and stale503
provisioningstarting up, or installing an index snapshot503
degradedserving a recent tip that no trusted validator has confirmed (p2p peers supplied it)200
syncingserving, but behind the best block; or no validator’s mempool is available yet503
servingserving at the best block200
  • Confirmed means a trusted validator has had the block on its best chain. One block past the last confirmation still counts, to cover the validator’s polling delay.
  • Recent means the block’s timestamp is within 24 block intervals of the current time.
  • A degraded node stays in load-balancer rotation so wallets keep being served while every trusted validator is down. Alert on the state, not on /readyz.
  • syncing never refuses requests. During a reorg, a node briefly reports syncing with the tip unvouched alarm raised.

detail adds context to the state (null until measured):

Statedetail
provisioning{phase, bytes, total_bytes, bytes_per_sec} for a snapshot install; else null
syncing{height, target, blocks_per_sec}: served height, best height, rate over ~60 s
serving{start, mbps}: gRPC egress over the last full 10-minute UTC window
degraded, down, held{reason, message}

reason is one of no_validator_reachable, tip_unvouched, tip_stale, runtime_stalled, overloaded, goaway_pending or goaway_sent.

Endpoints

ListenerPathResponse
gRPC/readyz{"state", "detail"}, 200 or 503
gRPC/statuspublic status (below), always 200
admin/readyzsame as on the gRPC listener
admin/livez503 after 30 seconds without a heartbeat from the server
admin/metricsPrometheus metrics

The gRPC listener answers these paths over HTTP/1.1 or HTTP/2 from the moment the process starts, before any gRPC method is available. Each connection may send 10 requests at once, refilling at one per second; beyond that it gets 429.

/status is safe to expose publicly. It contains no validator addresses, peers, disk details or mempool contents:

{"state": "serving", "detail": {"start": 1790000400, "mbps": 12.5},
 "chain": {"network": "mainnet", "tip": {"height": 3515406, "hash": "0000…"}, "served": 3515400,
           "validators": {"trusted": 2, "answering": 1}, "alarms": []},
 "build": {"version": "1.0.0"},
 "indexes": {"compact_block": ["GetBlock", "GetBlockRange", "GetBlockRangeNullifiers"], "utxo_set": []}}
FieldMeaning
chain.tipthe best block Zingod has verified
chain.servedthe height wallets are served at (null before startup completes)
chain.validatorstrusted validators configured, and how many are answering
chain.alarmsone message per raised alarm
indexeseach enabled index and the methods it serves

Alarms

Alarms appear in /status, and are logged at WARN when raised and INFO when cleared. They do not change what Zingod serves.

AlarmRaised when
stalea validator’s tip is 24 or more blocks behind the height expected from the clock
end of servicea validator’s release reaches its end-of-support height within a week
finality pausedmore than finalised_depth non-finalized blocks for 60 s, and finality not advancing
tip unvouchedthe best block is more than one block past the last one a trusted validator confirmed

Metrics

Every metric has HELP text. Histograms are Prometheus native histograms, available only in the protobuf format: enable scrape_native_histograms in Prometheus or Alloy. The text format carries every counter and gauge, but no histograms.

Disk space and disk I/O are not exported; use node exporter or cAdvisor.

Logging

zingod start logs to stdout in logfmt. RUST_LOG sets the filter; the default is zingod=info. An invalid filter stops startup.

ts=… level=info target=zingod::telemetry::progress message="Syncing blocks" height=3503023 target=3506659 bps=10 eta=5m49s component=ZingodNFS
  • Alert on WARN and ERROR. INFO is progress and lifecycle.
  • component names the subsystem: Zingod, Admin, IndexSnapshot, Traffic, ChainView, ZingodSync, ZingodNFS, GrpcSnapshot, Grpc, or an index name.
  • State changed is logged within a second of each state change, at ERROR for down, WARN for held and degraded, otherwise INFO. Not ready repeats every 60 seconds while the state is down, degraded or held.
  • While syncing, progress lines appear every 30 seconds. Block fetch stalled warns when syncing stops advancing. Serving the verified tip marks the end of the sync.

The Serving line

Once a minute, the gRPC server logs a Serving summary: rps (streams completed per second) and conns (open connections). It is logged at WARN if, during that minute, any request failed or was refused, took more than a second to its first response (slow), stalled (stalled), or a connection was refused at a limit (conns_refused). These fields appear only when non-zero.

  • INTERNAL, UNKNOWN and DATA_LOSS count as failures. UNAVAILABLE and RESOURCE_EXHAUSTED count as refusals. Other codes are client errors, counted in zingod_grpc_requests_total{code}.
  • The first failure of each minute is also logged as Request failed (ERROR), and the first refusal not caused by capacity as Request unavailable (WARN).

Maintenance

Operator hold

zingod hold takes a node out of rotation; zingod resume puts it back. A hold persists across restarts, deploys and reboots.

Use a hold before any planned stop. Stopping without one closes the gRPC listener immediately.

zingod hold            # returns once connections are drained, printing "safe to stop"
systemctl stop zingod
# ... maintenance ...
systemctl start zingod
zingod resume
  1. hold writes a marker file, held, in [serve] state_dir. It needs write access to that directory; there is no network endpoint.
  2. Within a second, the running node reports held and /readyz returns 503.
  3. After [serve] hold_goaway_delay_secs, every gRPC connection receives an HTTP/2 GOAWAY, and in-flight streams get max_connection_age_grace_secs to finish. Set the delay to at least your load balancer’s failure-detection time plus its DNS TTL.
  4. hold returns once the GOAWAYs are sent.

A held node keeps running and answering health checks and metrics. After resume, a node that fell behind reports syncing until it catches up. A stopped node is never reported as held, so dashboards can tell planned maintenance from a crash.

Both commands are idempotent.

Overload

A node that is persistently short of capacity reports down with reason overloaded, so load balancers move traffic away from it. Thresholds are set under [grpc.overload] (0 disables a cause). A cause is raised after its signal stays above the threshold for the enter time, and cleared after it stays below half the threshold for the exit time.

CauseSignal, per minuteEnter / exit
CPU pressure, IO pressureshare of time Zingod’s cgroup waited (Linux PSI)2 / 3 min
refusing workat least 10% of requests refused, at 1/s or more (exit: < 2%)5 / 5 min

Override the detector on a running node, without a restart:

zingod overload off                  # never report overloaded
zingod overload on                   # back to automatic
zingod overload set <key> <value>    # override a threshold, e.g. cpu_pressure_percent 40
zingod overload unset <key>          # back to the config's value

Overrides are stored in state_dir and persist across restarts. Zingod applies them within a second and logs a warning while any is in effect.

systemd

[Service]
Type=notify
ExecStart=/usr/bin/zingod start --config /etc/zingod/zingod.toml
Restart=always
TimeoutStartSec=15min
  • With Type=notify, systemctl start returns once Zingod is serving at the tip, or held. A deploy that cannot catch up shows as a failed unit.
  • While startup makes progress, Zingod extends the start timeout by 5 minutes at a time, so TimeoutStartSec only bounds a stalled start.
  • TimeoutStopSec must cover [grpc] shutdown_grace_secs plus the time to flush the indexes.
  • NOTIFY_SOCKET must be a filesystem path, not an abstract socket.

Stopping and restarting

On SIGINT or SIGTERM, Zingod closes the gRPC listener, sends GOAWAY to every connection, waits up to [grpc] shutdown_grace_secs for in-flight streams, flushes the indexes, and exits 0.

Zingod exits rather than serve data it cannot trust, so run it under a service manager that restarts it. It exits with code 1 when:

  • a validator entry is invalid, the gRPC address cannot be bound, or an index cannot be opened;
  • an index directory belongs to another network or format, or is truncated or corrupt;
  • an index no longer matches the chain (Diverged);
  • any internal task stops.

For a corrupt or mismatched index, delete its directory and Zingod resyncs it. An unreachable validator never stops Zingod.

Restarts are fast. Opening an index checks only its file lengths and last pages, then Zingod refetches the recent blocks above what was flushed (Fetching non-finalized blocks in the log).

zingod verify

zingod verify -c /etc/zingod/zingod.toml checks every page checksum of every enabled index. It only reads, so it is safe to run next to a live node.

It prints a JSON report to stdout and a summary per index to stderr; --help lists the exit codes. Per file, it reports lost, bad_sums and bad_pages. Bytes written by a live node after its last commit are reported as orphaned_bytes, not as corruption.

If any file is corrupt or lost, delete that index’s directory and let Zingod resync it.

Network exposure

Zingod has two listeners:

  • gRPC ([serve] grpc_listen_address) serves wallets, plus /readyz and /status, which are safe to expose publicly.
  • Admin ([metrics] listen_address, off by default) serves metrics and probes, with no authentication or encryption. Metrics include validator addresses. Keep it on loopback or a private network; Zingod warns at startup if it is bound to a public address.

Connections to validators are plain HTTP JSON-RPC.

Limits on connections, streams and reads are set under [grpc]; see the reference. When a limit is reached, the request is refused, not queued.

TLS

[serve] tls controls TLS on the gRPC listener:

ValueBehaviour
unsetplaintext on a loopback address; any other address refuses to start
{ cert_path, key_path }TLS with the given PEM files
"off"plaintext on any address, for use behind a TLS-terminating proxy (warns at startup)
"self_signed"TLS with a generated certificate, for clients that pin it
[serve.tls]
cert_path = "/etc/zingod/fullchain.pem"   # chain, leaf first
key_path = "/etc/zingod/key.pem"          # PKCS#8, PKCS#1 or SEC1

An invalid pair stops startup. Zingod checks the files every minute and picks up a renewed certificate without a restart; if the new pair fails to load, it keeps serving the old one.

With "self_signed", Zingod generates an ECDSA P-256 certificate with no expiry, stores it in <state_dir>/tls, and logs its SHA-256 fingerprint at each start. It covers localhost, the loopback addresses and the listen address. Wallets that verify against system roots, including zingolib, reject it, so public servers need a CA-issued certificate. To reissue it, for example after changing the listen address, delete cert.pem.

Browser clients (gRPC-web)

[serve] grpc_web = true serves browser wallets over gRPC-web on the gRPC listener, with CORS allowing any origin. Native gRPC is unaffected.

Enabling it makes the server usable from any website. A popular site embedding it can send load proportional to its own traffic.

zingod origin manages per-origin policy. Changes apply within a second and persist until changed:

CommandEffect
zingod origin block <domain>refuse requests from the origin with PERMISSION_DENIED
zingod origin allow <domain>serve the origin even when a parent domain is blocked
zingod origin reset <domain>remove the origin’s policy
zingod originlist the policies in effect
  • Policies apply to the registrable domain and its subdomains. The most specific policy wins, so allow app.example.com can override block example.com.
  • Every origin that is not blocked meets the same limits as any other client. Clients can set any Origin header, so an origin never lifts a limit.
  • zingod_grpc_origin_requests_total{verdict} counts browser requests by policy. To find an abusive origin, check your proxy or edge logs.

Over HTTP/1.1, browsers open one connection per in-flight stream, each counting against max_connections_per_ip. Serve over TLS so browsers use HTTP/2.

Stream timeouts

Streams have no total time limit. Two settings close streams that stop making progress:

  • unread_timeout_secs: the client stopped reading. The connection is closed.
  • silent_timeout_secs: the server has had nothing to send. The stream ends with UNAVAILABLE. GetMempoolStream is exempt.

Behind a proxy

Behind a proxy, every connection comes from the proxy’s address, so max_connections_per_ip would limit all clients together. To limit per client, list the proxy in trusted_proxies and have it send a PROXY protocol header (v1 or v2):

[serve]
tls = "off"   # the proxy terminates TLS

[grpc]
trusted_proxies = ["10.0.0.5/32"]

Connections from a trusted proxy must start with the header, or they are closed. Other addresses connect without one.

nginx must use the stream module, since grpc_pass in the http module cannot send the header:

stream {
    server {
        listen 443 ssl;
        ssl_certificate     /etc/ssl/zingod.pem;
        ssl_certificate_key /etc/ssl/zingod.key;
        ssl_alpn            h2;
        proxy_pass          127.0.0.1:8137;
        proxy_protocol      on;
    }
}

For HAProxy, use server zingod 127.0.0.1:8137 send-proxy-v2 in a mode tcp backend.

If your proxy cannot send the header, leave trusted_proxies empty and raise max_connections_per_ip to max_connections.

Open files

Each connection uses a file descriptor. At startup, Zingod raises its soft RLIMIT_NOFILE to the hard limit and refuses to start if max_connections plus 1024 does not fit. Raise the hard limit (LimitNOFILE= in systemd, --ulimit nofile= in Docker) or lower max_connections.

Docker

The image zingodevops/zingod runs zingod start as user zingod (UID and GID 1000), listens for gRPC on 0.0.0.0:8137, and has a built-in health check. It includes aria2c for index snapshots. To start a container, follow the quick start.

Other subcommands use the same config and data. Run them in the running container (docker exec zingod zingod hold) or in a one-off container with the same mounts (zingod verify).

Paths

PathContentsMount
/etc/zingod/zingod.tomlthe config filefile, read-only
/etc/zingod/tls/TLS certificates and keys, by conventiondirectory, read-only
/var/lib/zingod/indexes, state (including the hold marker), peer cachevolume, writable by UID 1000

A named volume gets the right ownership automatically. A bind mount must be writable by UID 1000: mkdir -p ./data && chown 1000:1000 ./data. Under rootless Podman, --userns=keep-id maps UID 1000 to your user.

Configuration

The image reads zingod.toml like any install, and ZINGOD_CONFIG_* environment variables override it (configuration). The file is required, because [[trusted_validators]] cannot be expressed in environment variables.

docker run ... -e ZINGOD_CONFIG_SYNC__CONCURRENCY=64 -e RUST_LOG=info,zingod_grpc=debug zingodevops/zingod

The image differs from a host install in one default: it listens on 0.0.0.0:8137. Because that is not a loopback address, [serve] tls must be set (TLS).

To expose metrics, set ZINGOD_CONFIG_METRICS__LISTEN_ADDRESS=0.0.0.0:9999 and publish that port only to your scraper.

TLS certificates

Mount the certificate directory, not the individual files. Zingod reloads renewed certificates, but a single-file bind mount keeps pointing at the old file after renewal.

docker run ... \
  -v /etc/letsencrypt/live/zingod.example.com:/etc/zingod/tls:ro \
  -e ZINGOD_CONFIG_SERVE__TLS__CERT_PATH=/etc/zingod/tls/fullchain.pem \
  -e ZINGOD_CONFIG_SERVE__TLS__KEY_PATH=/etc/zingod/tls/privkey.pem \
  zingodevops/zingod

The key must be readable by UID 1000.

Health check

The built-in check requests /readyz on 127.0.0.1:8137, over HTTPS without certificate verification, or over plaintext when tls = "off". It passes in the serving and degraded states, so a new node is unhealthy until it reaches the tip.

If the gRPC listener uses another port, override the check:

healthcheck:
  test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:9137/readyz"]

Compose

services:
  zingod:
    image: zingodevops/zingod:latest
    ports:
      - "127.0.0.1:8137:8137"
    environment:
      ZINGOD_CONFIG_SERVE__TLS: "off"
    volumes:
      - ./zingod.toml:/etc/zingod/zingod.toml:ro   # jsonrpc_address = "zebra:8232"
      - zingod-data:/var/lib/zingod
    depends_on:
      - zebra

  zebra:
    image: zfnd/zebra:latest
    volumes:
      - zebra-data:/home/zebra/.cache/zebra
    # set [rpc] listen_addr so zingod can reach it

volumes:
  zingod-data:
  zebra-data:

jsonrpc_address accepts a hostname, so a Compose service name works.

Building the image

docker build -t zingod .
docker build -t zingod --build-arg CARGO_PROFILE=profiling .   # with symbols and frame pointers

The build produces a static musl binary on Alpine. Base images are pinned by digest.

Migrating from Zaino

Zingod 1.0 replaces Zaino 0.10. It is a new daemon, not an upgrade: it reads nothing from a Zaino install, so set it up fresh and delete Zaino’s data once Zingod is serving.

Zaino 0.10Zingod 1.0
repo zingolabs/zaino, tags zainod-X.Y.Zrepo zingolabs/zingod, tags vX.Y.Z
binary zainodbinary zingod
image zingodevops/zainod, /app/config, /app/dataimage zingodevops/zingod, /etc/zingod/zingod.toml, /var/lib/zingod
$XDG_CONFIG_HOME/zaino/zainod.toml, ZAINO_* variables$XDG_CONFIG_HOME/zingod/zingod.toml, ZINGOD_CONFIG_* variables
one LMDB databaseone directory per index
cargo features (prometheus, no_tls_*, …)none; every feature is always built

Config

Start from zingod generate-config; unknown keys are rejected, so a Zaino config will not load. Carry these values over:

Zaino 0.10Zingod 1.0
network = "Mainnet"network = "mainnet" (lowercase)
[validator_settings] validator_jsonrpc_listen_address, validator_cookie_path, validator_user, validator_passwordone [[trusted_validators]] table per validator: jsonrpc_address, cookie_path, user, password
grpc_settings.listen_address, [grpc_settings.tls]serve.grpc_listen_address, serve.tls (TLS)
metrics_endpointmetrics.listen_address
backend, zebra_db_path, ephemeral_finalised_state, json_server_settings, storage, mempool, donation_addressremoved

Data

Zingod builds its indexes on first start, or installs them from an index snapshot. Plan for the first sync.

APIs

  • Zingod serves no JSON-RPC. Zaino’s JSON-RPC server and port 8237 are gone; send node RPC to the validator.
  • zaino.index.v1.IndexedTipService, GetBlockNullifiers and Ping return UNIMPLEMENTED. Poll GetLatestBlock instead, and use GetBlockRange with poolTypes.
  • GetBlockRangeNullifiers and GetTaddressTxids still work but are deprecated.
  • See methods and status codes for other behaviour wallets may notice.

Metrics

All zaino_* and zainod_* metrics are gone; process_* remains. Closest replacements:

ZainoZingod
zaino_chain_tip_heightzingod_best_tip
zaino_sync_fetched_heightzingod_fetch_height
zaino_sync_finalized_heightzingod_index_finalized_height{index}
zaino_grpc_request_duration_secondszingod_grpc_duration_seconds{method}
zaino_rpc_outbound_*zingod_validator_rpc_*
zaino_mempool_transactionszingod_chainview_mempool_transactions{state}

Histograms are native histograms (metrics).

Logs

Logs are logfmt on stdout, filtered by RUST_LOG (default zingod=info). ZAINOLOG_* is ignored.

Architecture

How zingod turns a Zcash chain into the lightwalletd gRPC service, and why it is built this way. Crate APIs live in each crate’s usage.md: what answers each method in zingod-grpc, storage in zingod-persistence.

1. Who owns what

  • The validator (zebrad, zakura) owns consensus: valid blocks, valid transactions, the mempool.
  • The wallet owns keys.
  • Zingod owns every answer derived from the chain.

Data a validator needs for consensus stays there and is not copied; everything derived is a Zingod index and never falls back to a validator. So:

  • Block, tree-state and transparent-address methods come from an index or not at all. A disabled index answers UNIMPLEMENTED naming it.
  • GetTransaction goes to a validator: raw transaction bytes are consensus data it already stores (why). With no validator answering, the txid-location index names the transaction’s block, fetched from p2p peers and checked against the hash our indexes hold (which does not cover v5 proofs and signatures). GetTaddressTransactions works the same way: the index names the txids, GetTransaction’s path supplies the bytes.
  • Zingod does not validate transactions. That would be a second mempool verifier (proofs, nullifiers, scripts, conflicts) tracking every network upgrade. Validators decide; Zingod checks only what the bytes, the next block and its own UTXO set decide (§5).

The principle behind the rest: verify what Zingod can, trust validators for as little as possible, take everything else from as many peers as possible.

2. The map

  trusted validators (1–N, configured)             peers ([p2p] on; default off)
  JSON-RPC or gRPC, + zebrad push streams          zebra-network peer set
                    │                                          │
                    └──────────────┬───────────────────────────┘
                                   ▼
                    zingod-traffic  TrafficBalancer      every request: lanes, hedge, blame
                                   │
       ┌───────────────────────────┼─────────────────────────────────────┐
       ▼                           ▼                                     ▼
  HeaderSync (chainview)      ObservationMerge + PeerMempool       ChainView::validate_transaction
  → HeaderChain               → mempool view                       → broadcast_transaction
                                                                   (SendTransaction)
       │ watch<HeaderChainSnapshot> │
       ├──────────────────┐         │
       ▼                  ▼         │
  FinalizedFollower        Nfs (RAM)    │
  (zingod-sync)         tip overlay  │
       │ FinalizedStream     │ NfsPublication
       ▼                     ▼      ▼
  index writers   ──▶ zingod-grpc-snapshot Publisher ──▶ one Arc<GrpcSnapshot> per request
  (disk)  durable views                                  → zingod-grpc, /status, /readyz, /metrics

Two paths fold blocks, and their windows overlap:

  • Finalized path (disk): blocks at or below the finalized tip, folded and committed by each index writer. Writers never see a reorg.
  • Tip path (RAM, the NFS): every block above the indexes’ durable tips, finalized blocks not yet durable included, folded in memory and served at once. A block leaves once every index has committed it. Reorgs live here alone.

Both run the same fold per index. While the NFS folds, a block above the lowest durable tip is folded on both paths: in RAM by the NFS, and by its writer once finalized. When the NFS stands down (first sync, an index enabled late: §7) the finalized path folds alone.

3. Sources and what each is trusted for

A trusted validator is one the operator configured ([[trusted_validators]]). A peer is any node on the p2p network: discovered, unauthenticated, free to multiply. Nothing a peer says counts until Zingod has checked it.

NeedTrusted validatorsPeersCheck
Headers✓ (getblockheader runs)✓ (FindHeaders), after the anchortrusted: link + work; peer: every rule
Anchor (first sync)✓ only sourceheaders below our durable tip (restart)each hash = the one our indexes hold
Blocks with a header✓✓ (by hash)hash + merkle root + coinbase height
Blocks below the headers✓ only source (by height)—coinbase height (body trusted)
Mempool✓ (listing + fees)✓ (quorum of listings)txid recomputed; §5
Mined tx by id✓ only source—txid recomputed
Finality✓ (vouch)✓ (> 5 netgroups, no difficulty drop)§4
Submission✓ check (checkadmission)✓ relays (zebra’s, ~⅓ of peers)§6

Why peers never decide the tip. A peer count proves nothing an attacker cannot buy: anyone can run ninety. Proof of work costs real mining however many peers present it. So peers are safe sources of headers and of anything checkable against them, and weak sources of claims. Two claims are taken from peers anyway, so Zingod keeps serving with every trusted validator down: finality and mempool admission. Both count only outbound peers (Zingod chose them) spread over more than 5 netgroups (/16 IPv4, /32 IPv6; one operator ≈ one netgroup).

4. The best chain

The Validated Header tree (zingod-header-chain, memory only) holds every valid branch above the finalized tip; the best tip is the leaf with the most cumulative work. HeaderSync (zingod-chainview) is its only writer and publishes an immutable HeaderChainSnapshot on a watch that the NFS, the finalized path and the GrpcSnapshot publisher read.

  finalized tip                          best tip (most work)
  ───●──●──●──●──●──●──●──●──●──●──●  A  ◀─
                 └──●──●──●            B  kept: may still win; overtaking A = a reorg
  • Anchor. No history from genesis. With no chain, or a trusted validator so far ahead its anchor lies above our fetch ceiling, the tree anchors at that validator’s claim − max(depth, 113) (113 = the difficulty rule’s context). With no trusted validator answering (a restart mid-outage, [p2p] on), the indexes’ own chain anchors instead: the 113 blocks below their durable tip, each peer header matched to the hash the indexes hold. A first sync has no such chain: only a trusted validator anchors it.
  • Trusted headers arrive in runs and enter on their parent link and nBits work alone: the validator already ran the rules.
  • Peer headers ([p2p] on, once anchored) run every consensus rule first (zingod_header_chain::validate, the rules). A header from the future is deferred, not blamed; one failing a rule gets its sender blamed.
  • Bounds. Side branches are capped in tips and nodes (past either, the lowest-work side leaf goes), and only a window of finalized headers is kept (limits). depth = [sync] finalised_depth (default and floor).

Finality

A header proves work, not that its block’s transactions are valid. A finalized block is never reorged (the indexes hold it on disk), so finality is where Zingod stops being able to change its mind.

Every header node records its witnesses: who once had that block on its best chain, and so every ancestor too, for good. Each block carries all of its own accounting; nothing about it is kept anywhere else.

  • Vouched. A trusted validator witnessed it (zebra commits only valid blocks). Vouches come from the last header of each trusted run, each validator’s claimed tip, and its getblockhash answer at our best. Peer headers never vouch.
  • Peer-quorate ([p2p] on). Live outbound peers in > 5 netgroups witnessed it by announcing it or a descendant.
  • finalized = min(highest settled, best − depth), where settled = vouched, or peer-quorate while no unvouched best-chain block has under half the work of the last vouched block. An attacker who eclipses us can feed a private chain, but its difficulty retargets to their hashrate, which shows up as that collapse. Both peer checks fail safe: a thin peer set or a real hashrate drop only delays finality.
  • Neither holding: finality waits, the finality paused alarm rises (§11), serving continues from RAM. A trusted validator’s return vouches and finality resumes.

A reorg, step by step

Peers are first to every reorg: their headers enter on the rules alone, while a trusted validator downloads and fully verifies the new branch before it switches, and Zingod learns of that on its next poll (§9). Meanwhile the served tip drops to the fork point (§7) and synced closes (§8).

 network          ─── N ──┬── N+1              branch A, Zingod + the validator on it
                          └── N+1' ── N+2      branch B, more work, wins

 ──────────────────────────────────────────────────────────────────────────────────▶ time
                   t0     │ L2: validator verifies B │ L3 │   L4: fetch N+1', N+2     │
                    │     │                          │    │                           │
 peers ─────────────┤ announce B headers                                              │
                    │                                                                 │
 best ─── N+1 ──────┤ N+2  (every header rule: PoW, difficulty, MTP, Equihash)        │
                    │                                                                 │
 served ── N+1 ─────┤ N ◀── fork point, vouched: what wallets get ───────────────────▶ N+2
                    │                                                                 │
 validator ── N+1 ──┼────────────────────────────▶ N+2                                │
                    │                              │    │                             │
 vouched ─── N+1 ───┤ N ◀── last shared block ─────┼────▶ N+2  (poll, or push hint)   │
latencywhatbound
L1peers announce, headers validated, best switchesmilliseconds
L2the validator downloads and verifies the branch, switchesits verification time
L3Zingod sees the validator’s new tip (the vouch)one poll; ~0 with the push stream (§9)
L4Zingod fetches the branch’s blocks by hash and folds themblock fetch

Through the window Zingod serves the fork point, a vouched block behind an unvouched best: a currency gap, not a trust gap, and the state rule reads it so (states). The tip unvouched alarm (§11) carries the unvouched best. L2 is the validator’s job and no change on Zingod’s side removes it.

5. The mempool view

Every transaction the view knows is a sighting: its bytes (fetched once, txid recomputed), its fee, a bitset of trusted validators listing it, a quorum flag and an ours flag.

Servable = listed by a trusted validator (not Down or catching up), admitted by a peer quorum, or ours (accepted from our own submission). One trusted listing is enough: each trusted validator is trusted.

Peer quorum ([p2p] on). Each live outbound peer is polled for its whole mempool, and each listing counts while fresh (intervals). A transaction is admitted when:

  • more than 2/3 of the counted listings hold its exact id (txid + auth digest), from > 5 netgroups;
  • its bytes decode, its expiry and (v5+) branch id fit the block above the verified best, it is not coinbase;
  • every transparent prevout resolves in the served UTXO set or a held transaction’s outputs, and the resulting fee is not negative.

It stays while more than 1/3 still list it. No fee floor (zebrad’s and zakura’s differ), no proofs, signatures or scripts: each lister is a full node that ran those. A consensus refusal is remembered (REFUSED_TTL); an unknown prevout is retried next round. With honest peers two conflicting transactions are never both admitted (one node’s mempool holds at most one, and > 2/3 + > 1/3 must overlap); a peer listing both can get both admitted and kept (mempool_model_tests.rs). Cost to an attacker: > 2/3 of our outbound peers across 6 netgroups, for a served transaction that never mines.

Transaction inv announcements are not used: nodes announce each transaction to a third of their peers, once, so silence means nothing.

  • Leaving. A transaction goes when nothing lists it and no quorum keeps it. A new tip clears nothing; an unlisted ours goes at the next move of the best block.
  • Gate. GetMempoolTx / GetMempoolStream answer only while a Live trusted validator holds the verified best, or the polled peers span > 5 netgroups. Otherwise UNAVAILABLE, never an unverified mempool.

6. Submission

SendTransaction asks one trusted validator for a verdict without a broadcast, answers the wallet from it, then relays through the peers and watches the transaction spread. Relaying every wallet transaction through the same validators would tell any observer which transactions came through Zingod, and let one validator censor every wallet behind it. The relay follows Dandelion++’s originator with its fail-safe timer.

precheck (nothing contacted): decodes (-22); expiry ≥ next block, v5+ branch = next block's (-25)
check: checkadmission on a random Live trusted validator holding the verified best
       refused → wallet gets that refusal at once, nothing relayed
       admissible → wallet answered Ok(txid) at once; transaction becomes `ours` (servable)
       unsupported / unanswered → next holder; none → unchecked (below)
attempt: peers — zebra's own relay, PushTransaction to ⌈ready/3⌉ random ready peers
         (no peers: a random untried trusted validator, sendrawtransaction)
unseen past propagation_threshold → next attempt, up to max_attempts
then, still unspread: one trusted sendrawtransaction (the fail-safe)
unchecked: Ok at the first acceptance (trusted Ok | any trusted listing | peer quorum);
           never accepted → that trusted push is the verdict; else UNAVAILABLE
  • checkadmission (Zingod’s zebra fork) runs sendrawtransaction’s checks, the fee policy and mempool conflicts included, without admitting or relaying. It goes to a holder of the verified best only (a validator behind it refuses what spends a fresh output). A validator without it (zcashd, stock zebrad, zakurad) answers -32601: never a failure, the next holder is asked. A refusal’s wire shape: SendTransaction.
  • A peer relay gets no reply and zebra names no recipient. Spread = a trusted validator never pushed to lists it, or a peer quorum admits it (past the ~⅓ one relay reaches). The job ends at spread, when nothing outside its trusted entries can be watched, or when attempts run out.
  • Relays ride the peer set’s existing connections: a zebrad peer drops a second connection from our IP, so a fresh connection per push would never deliver. With [p2p] off every attempt is a trusted sendrawtransaction.
  • [submission] sets propagation_threshold_secs and max_attempts (SubmitPolicy).

7. Indexing

Finalized path (zingod-sync)

FinalizedFollower streams lowest durable tip + 1 ..= finalized tip into one FinalizedStream<Block>:

  • Heights the header chain has a header for: fetched by hash from any member (peers included), checked against the header (hash, merkle root, coinbase height).
  • Below the header chain (bulk sync, a restart from an old index): by height, trusted only, batched, checked by coinbase height only. A trusted validator’s body is trusted.
  • Each block must link to every durable tip it extends and to the block sent before it; otherwise the follower stops and the process exits. Nothing is skipped.
  • Queues are byte-bounded and a send reserves room in every queue first, so the slowest writer paces the fetch and memory stays bounded. A dropped queue or failed commit panics.
  • Utxo-set resolves each transaction’s fee and feeds compact-block through a FeeSink.
  • A writer commits when its buffer fills, after a run ending at the finalized tip, and at shutdown.

Tip path (zingod-nfs)

  • Root = the lowest durable tip across indexes.
  • The NFS folds only while the header just above the root is held and best is within 2 · depth of the root. Otherwise (first sync, an index enabled late) the finalized path does the work and the root is served.
  • Each best-chain block above the root is fetched by hash, checked against its header, and folded over the indexes’ durable views (utxo-set first: its fees feed compact-block).
  • Served tip = the deepest folded best-chain node, else the root. A reorg moves it to the fork point at once, then forward as the new branch folds. Side branches stay until the header chain drops them, so switching back costs nothing.
  • A node goes once every index has committed its height.

8. Serving: one GrpcSnapshot

zingod-grpc-snapshot’s publisher combines the NFS’s latest NfsPublication and the chain view’s latest publish into one Arc<GrpcSnapshot>. Every gRPC request or stream, probe, scrape and state log loads one and reads only it, so no answer mixes two moments.

  • No height named → the served tip, which every enabled index holds. A height named → up to that index’s own durable tip.
  • synced opens when the served tip is the best block and no answering trusted validator claims a higher tip (ChainViewSnapshot::claimed_past_best: one already shown off our chain does not count, one not polled since best moved does), and stays while it is on the best chain within depth of it. At boot, header sync anchors depth below a validator’s claim, so it stays shut until the header chain reaches that claim and the NFS has fetched the window.
  • The daemon’s state (serving, degraded, down, …) is judged from the same load: states.

GetMempoolStream

The protocol says “close the stream when a new block is mined”; Zingod makes that exact. A stream is bound to the served tip’s hash: it opens with the servable mempool, sends each new transaction once (every message height: 0, never one mined in a block the served tip extends), and ends OK once the served tip moves, never on an empty mempool. It ends only after the GrpcSnapshot already serves the new tip, so the GetLatestBlock a wallet sends next sees it. One append-only log per served tip is shared by every subscriber, so per-transaction work never scales with subscribers.

9. Talking to validators and peers

Every request goes through one TrafficBalancer (zingod-traffic); lanes, intervals, hedge delays and bench times are in its guide.

  • Members. Each trusted validator, in priority tiers (0 first); then the whole peer set as one member below every tier (zebra picks the peer). Peers answer headers and blocks only.
  • Failure. A round asks each member once, best tier first; slow answers are hedged to another member. Blocks are retried until answered; everything else ends unanswered after one round.
  • Blame. The balancer cannot judge a value; the caller checks it and reports a bad one, which benches that member. Absent is never blame. The peer set is never benched: zebra scores the lying peer instead.
  • Polling. Each trusted validator is polled on its own permit, so no request delays tip following, and sooner on a zebrad push event. Consecutive failures make it Down, probed until it answers. A failing validator never stops the process.
  • Load on a zebrad = max_connections. There is no request-rate or byte-rate limit.

10. When things fail

DownChain + indexesMempool, SendTransactionFinality
one trusted validator✓✓✓
every trusted, [p2p] on✓ near the tip (peers); bulk sync below the headers stalls✓ quorum; GetTransaction from the peers’ blockon peers’ word, else paused
every trusted, [p2p] offtip stopsUNAVAILABLEpaused
every trusted, restart, [p2p] on✓ anchored on the indexes’ own chain, then as aboveas aboveas above
every trusted, first syncnothing (only a trusted validator anchors)UNAVAILABLE—
every peer✓ (trusted)✓✓

11. Telemetry

Observation only: none of it changes a tip or what is served. State is read from one GrpcSnapshot at scrape (zingod_* and zingod_chainview_* gauges, /status); events are emitted where they happen. Alarms log once WARN when raised, once INFO when cleared (the alarms; stale = STALE_TIP_BLOCKS).

12. Seams

A trait exists only at a seam: a boundary with a second implementation (a real one or a test double), or one that keeps a dependency out of the crates above it.

  • zingod_source::ChainDataSource: JSON-RPC or indexer gRPC; tests drive the real client against a MockValidator.
  • zingod_chainview::PeerSource, zingod_traffic::PeerTransport: zingod’s ZebraPeers, which keeps zebra-network out of both crates.
  • zingod_chainview::UnspentOutputs: the chain view sits below the indexes.
  • zingod_persistence::PersistenceEngine and fs::Fs: indexes see the port, never the engine; SimFs tests crashes.

Not seams: the gRPC service, each index, and the validator set (TrafficBalancer is not a ChainDataSource, so a pool is never mistaken for one validator).

What the clients actually call

What the two wallet sync engines ask for, how often, and where answering slightly wrong is worse than not answering. Read at the source; references name a function or file, not a line.

  • librustzcash: zcash_client_backend/src/sync.rs, the only RPC caller in that workspace. It backs Zashi and zallet.
  • pepper-sync: zingolib/pepper-sync, all traffic through client/fetch.rs. It backs zingolib and Zingo mobile.

Where Zingod answers each method from: zingod-grpc; the wallet-facing contract: the book.

The call matrix

RPClibrustzcashpepper-sync
GetLatestBlockevery loop iterationonce per session
GetBlockRangeper batch, dominantper scan range, dominant
GetBlocknot calledreorg check and backfill
GetTreeState1:1 with every GetBlockRange, at scan_range.start - 1 (download_chain_state)birthday and fallbacks
GetSubtreeRootsthree pools at startup, ironwood includedthree pools per session, two passes each
GetTransactionnot calledper scan target
GetTaddressTransactionsnot callednot called (it uses GetTaddressTxids)
GetTaddressTxids (deprecated alias)not calledper known t-address, and gap-limit discovery
GetBlockRangeNullifiers (deprecated alias)not calledper ScannedWithoutMapping range
GetAddressUtxosStreamper account per iterationdead code
GetMempoolStreamnot calledbackground task, retries forever
SendTransactionthe embedder’s jobzingolib
GetLightdInfonever calledzingolib server selection only

GetTreeState is 1:1 with GetBlockRange, the highest-volume method after block fetch itself: a librustzcash wallet cannot complete a single sync batch without it.

Two deprecated methods, served until pepper-sync moves off them

GetBlockRangeNullifiers and GetTaddressTxids are deprecated aliases, marked TODO: REMOVE at every site. pepper-sync calls both on every sync and treats any non-OK status as a fatal SyncError, so without them no pepper-sync wallet syncs against Zingod. They go when pepper-sync calls the replacements.

  • GetBlockRangeNullifiers is superseded by GetBlockRange with shielded poolTypes, a superset. Its shape is not a poolTypes subset (it drops chainMetadata and sapling outputs, and keeps only each action’s nullifier), so Zingod re-projects the stored compact blocks rather than keep a second projection that could disagree.
  • GetTaddressTxids is misnamed and deprecated by the proto itself (“please use GetTaddressTransactions instead”).

Correctness landmines

Ironwood tree state must be served correctly or wallets silently mis-sync. Both clients map an empty TreeState.{orchard,ironwood}_tree to an empty commitment tree with no error (TreeState::orchard_tree / ironwood_tree in zcash_client_backend/src/proto.rs; pepper-sync/src/witness.rs), indistinguishable from a pre-activation height; GetSubtreeRoots is swallowed into Vec::new() the same way. So Zingod serves each pool from its activation on as the real tree (000000 when empty), and "" only below activation (tree-state guide).

The wire format is a CommitmentTree, not a frontier. TreeState.*_tree is the hex of zcash_primitives::merkle_tree::write_commitment_tree; both clients parse it with read_commitment_tree, then .to_frontier().

Neither client sends a block hash. Both pass hash: vec![] to GetTreeState.

GetSubtreeRoots costs one extra empty round trip. pepper-sync re-requests until a pass returns zero roots, so a session makes six calls, not three.

GetBlockRange has no client-side size limit. pepper-sync requests a whole scan range in one call, and scan ranges follow subtree boundaries (2^16 notes, not blocks): a mainnet wallet sync asked for 158,853 blocks in one call. Any server-side maximum breaks zingolib for some birthday, and because it retries a stream error unboundedly, it presents as a hang. So Zingod has no range cap, and never answers short: a wallet given fewer blocks than it asked for reads that as the end of the chain.

Do not implement GetAddressUtxos paging naively. startHeight is not a unique resume key (lightwallet-protocol issue #33), so paging can stall. No current client pages.

pepper-sync refuses a server below protocol v0.5.0. It rejects any lightwalletProtocolVersion below v0.5.0, an empty one included (check_lightwallet_protocol_version in pepper-sync/src/client.rs). Zingod reports its vendored protos’ release (zingod_proto::LIGHTWALLET_PROTOCOL_VERSION), so a new upstream release moves the reported version with it.

Transparent: address queries and the private path

pepper-sync discovers transparent funds by sending each derived address to the server, one per round trip, until gap_limit consecutive addresses come back empty (address_discovery in pepper-sync/src/sync/transparent.rs), and re-pays it every session. The server learns every address a wallet derives, which are used, and the order that links them to one wallet.

The private alternative: request the TRANSPARENT pool in BlockRange.poolTypes and match scripts locally (lightwallet-protocol v0.4.0). Zingod serves it. CompactTxIn carries no prevout value, so compact blocks alone cannot yield a transparent balance: the address RPCs stay even once clients move to the private path. As lightwalletd does, a coinbase’s vin is omitted (a client detects it by CompactTx.index == 0).

Never silently prune a requested TRANSPARENT. If Zingod cannot serve it, return an error. Silent pruning costs a user their privacy with nothing in the logs.

The contract is the proto, not a ZIP

ZIP-307 covers Sapling only and ZIP-314 is Reserved with an empty body, so the .proto comments and CHANGELOG in zcash/lightwallet-protocol are the normative source, and LightdInfo.lightwalletProtocolVersion is how a server states what it supports.

Testing

Zingod has two kinds of test. The unit and model tests in packages/* run on the host with cargo nextest run --workspace and need no validator, although they are not network-free (zingod-grpc, for example, binds a loopback socket). Their blocks, headers and validators come from one builder, MockChain. The live tests in live-tests/ run a deployed zingod against a real zebrad on Kubernetes, driven by ztest.

SetWhat it covers
live-tests/clientlesszingod’s gRPC answers checked against the validator as oracle, with no wallet
live-tests/e2ea real wallet syncing and sending through zingod’s gRPC to a live validator
live-tests/non-finalized-statereorgs forced on a regtest zebrad, checked across every index (README)
live-tests/syncthe long-running mainnet profiles zingod_index_construction and zingod_index_crash_consistency

live-tests/ is a standalone Cargo workspace, and live-tests/sync/ a second one nested inside it, each with its own lock. The rules for writing live tests: live-tests/CLAUDE.md.

Model tests in bulk

Every model suite runs at its in-code case count under cargo nextest run --workspace. To run them harder, use the models nextest profile (.config/nextest.toml), which selects only the model suites and lifts the default profile’s 60 s per-test kill, and set PROPTEST_CASES:

PROPTEST_CASES=5000 cargo nextest run --profile models

PROPTEST_MAX_SHRINK_ITERS shrinks a failure harder and PROPTEST_VERBOSE=1 prints each case. Raising PROPTEST_CASES across the whole workspace also scales transactions_decode_to_their_source (zingod-source), which is slow.

A new model suite joins the profile by its name: its test path contains model or random_histories. Its oracle is written independently of the code under test, and moves the model cannot apply are clamped rather than filtered out, so shrinking stays effective. A failing case is saved and replays on every run: beside an integration test as <test>.proptest-regressions, or under the crate’s proptest-regressions/. Commit it with the fix.

Heavy run

A change to zingod-persistence, zingod-header-chain, zingod-nfs, zingod-grpc-snapshot or zingod-traffic is done only once this passes for that crate. Each round runs the crate’s whole suite on fresh seeds, so the timing-sensitive tests (background merges, failed I/O, crash states) repeat alongside the models. Then the crates built on it run once:

crate=zingod-persistence
end=$((SECONDS + 180)); round=0
while [ $SECONDS -lt $end ]; do
  round=$((round + 1))
  PROPTEST_CASES=1000 cargo nextest run -p "$crate" --all-features --profile heavy \
    || { echo "$crate: FAILED in round $round"; break; }
  echo "$crate: round $round passed"
done

cargo nextest run --workspace -E "rdeps($crate)"   # after the loop passes
  • --profile heavy lifts the default profile’s 60 s kill (at 1000 cases a model test outlives it): a test still running at 300 s is reported slow, never killed.
  • --all-features builds the crate with every optional feature on.
  • Lengthen the run by raising 180, never shorten it. Keep the test profile (no --release): overflow checks stay on.
  • A failure saves its shrunk case (above). Fix the bug and rerun the whole block.

Commands

cargo nextest run --workspace              # from the repo root

cd live-tests
ztest run                                  # clientless, e2e and non-finalized-state
ztest run -p clientless                    # one partition
ztest run --rerun latest -p clientless     # the last run's failures
ztest run --no-cleanup -p clientless       # keep each test's namespace (pods, logs) to inspect

cd sync                                    # now in live-tests/sync
ztest sync start zingod_index_construction --watch
ztest sync start zingod_index_crash_consistency

ztest run MUST be launched from live-tests/ and ztest sync from live-tests/sync/: neither finds its tests from any other directory. ztest run takes cargo nextest run arguments, but its own flags such as --rerun and --cluster MUST come before them.

The ztest CLI

Both live workspaces depend on the published ztest crate (ztest = "0.2.0" in each Cargo.toml). Install the matching CLI from crates.io:

cargo install --locked ztest_cli

The sync profiles build zebrad’s image from a zebra checkout beside this repository.

Cluster setup

Any reachable Kubernetes cluster works. For a local kind cluster:

kind create cluster --name ztest
ztest cluster add kind --kind ztest --set-default   # profile "kind" on kind cluster "ztest"
ztest cluster setup                                 # provisions ztest's resources; takes a few minutes
ztest cluster check                                 # read-only readiness report

The sync profiles restore validator chain-state snapshots of 40 to 200 GB. A CSI driver with copy-on-write VolumeSnapshot support makes that restore fast (without one the snapshot is copied); the other live tests need neither. Pick the driver when you add the cluster profile:

ztest cluster add kind --kind ztest --storage-driver topolvm.io --set-default                    # LVM thin pools
ztest cluster add kind --kind ztest --storage-driver rook-ceph.cephfs.csi.ceph.com --set-default # shared cluster

Updating Zebra

Zingod’s validator path links no Zebra crate. zingod-source fetches each block as consensus bytes (JSON-RPC getblock <hash> 0, or the indexer gRPC), decodes it with librustzcash, and parses every other JSON-RPC result itself (packages/zingod-source/src/json.rs). The one Zebra link is zingod-peers, which embeds zebra-network (and zebra-chain) as Zingod’s p2p layer, pinned by git rev in the root Cargo.toml.

So keeping up with Zebra means testing against each new zebrad release, plus moving the zebra-* rev when the p2p layer needs it. A spread between the latest zebrad and the one the live suite tests is high-priority tech debt. When checking a JSON-RPC response’s shape, read the tagged Zebra source for the node version you target, not the crates.io crates.

Moving to a new zebrad

  1. Take a baseline on dev with cargo nextest run --workspace and the live suite.
  2. Bump the version in every Validator::zebrad("…") under live-tests/, and the zebrad version stated in the requirements.
  3. Check out the matching tag in the zebra checkout beside this repository: the sync profiles build zebrad’s image from it.
  4. Run the live tests (testing.md). A changed response shape shows up as a parse error in zingod-source, which is where it is fixed.

Pinning unreleased dependencies

librustzcash’s crates (zcash_primitives, zcash_protocol, zcash_transparent, zcash_address, zcash_encoding, equihash) and the crates under them (orchard, sapling-crypto, incrementalmerkletree) track upstream closely. An unpublished change is pinned through the root [patch.crates-io] (Cargo honours it only in the workspace root), by git rev, or by path to a sibling checkout while a change spans several repositories. Patching one of these crates usually forces patching the others, so every crate agrees on one version of the types that cross between them; the zebra-* dependencies point at a zebra fork built on the same forks for the same reason. Comment each patch with why it is not a published version and when to drop it.