Zingod
Zingod is a Zcash indexer for light wallets. It follows the chain from one or more trusted
validators (Zebra or
Zakura), optionally also from Zcash p2p peers, builds its
own indexes, and serves the lightwalletd-compatible CompactTxStreamer gRPC service.
The validator validates. Zingod indexes and serves.
- Wallet developers: connecting, how each method behaves, and the status codes to handle.
- Operators: installing, configuring, monitoring and exposing a Zingod server.
- Contributors: how Zingod is built, and how it is tested.
Source and releases: github.com/zingolabs/zingod.
Connecting
Zingod serves the CompactTxStreamer gRPC service from the
LightWallet Protocol, the same service as lightwalletd. A wallet that works with
lightwalletd works with Zingod without changes. Zingod does not serve JSON-RPC.
Transport
gRPC over HTTP/2 on a single port, 8137 by default.
Public servers use TLS with a CA-issued certificate. Plaintext is for a wallet on the same host, or for a server behind the operator’s own TLS-terminating proxy. An operator can also run a self-signed certificate; only clients that pin its fingerprint can connect to it.
Browser clients
If the operator enables gRPC-web, browsers can connect on the same port. CORS allows any origin. With gRPC-web off, browser requests fail CORS. Native gRPC clients work either way.
- Over HTTP/1.1, each in-flight stream uses its own connection, and connections are limited per client address. Connect over TLS so the browser can use HTTP/2.
- Operators can block an origin. Requests from a blocked origin fail with
PERMISSION_DENIED.
Server info
GetLightdInfo is answered from Zingod’s own state, so it works even when the server’s
validators are unreachable.
| Field | Value |
|---|---|
vendor | zingod |
chainName | main, test or regtest |
blockHeight | the served tip (see heights) |
estimatedHeight | the best block Zingod has verified; higher than blockHeight while syncing |
consensusBranchId | the consensus branch at blockHeight |
upgradeName, upgradeHeight | the next network upgrade after blockHeight, or empty and 0 |
lightwalletProtocolVersion | the protocol release Zingod implements |
Connection limits
Defaults are shown; operators can change them.
| Limit | Default |
|---|---|
| concurrent streams per connection | 100 |
| connections per client address | 128 |
| connection lifetime | 30 minutes (±10%), then HTTP/2 GOAWAY |
| idle connection | closed after 30 minutes |
| in-flight streams after a GOAWAY | 10 minutes to finish |
| client not reading a stream | connection closed after 5 minutes |
| stream with nothing to send | ends UNAVAILABLE after 5 minutes (except GetMempoolStream) |
Streams have no total time limit, so a full-chain GetBlockRange runs for as long as the client
keeps reading. gRPC libraries reconnect after a GOAWAY on their own. A server being taken out of
service also sends GOAWAY, so reconnecting is how a wallet moves to another server.
Health check
The gRPC port also answers two HTTP paths, useful for picking a server:
GET /readyz:200when the server is ready for wallets, otherwise503.GET /status: JSON with the server’s state, tip and served methods (format).
Methods
The protocol files define every method. This page covers only behaviour the protocol leaves open, and the places where Zingod differs from lightwalletd.
Heights
The served tip is the block Zingod currently answers at. Every index holds it.
- Methods that take no height answer at the served tip:
GetLatestBlock,GetLatestTreeState,GetSubtreeRoots, the address methods, andGetLightdInfo.blockHeight. A wallet that only asks for heights up toGetLatestBlockalways gets an answer. - Methods that take a height (
GetBlock,GetBlockRange,GetTreeState) answer up to the tip of the index they read. While the server is syncing, that can be above the served tip. - If a
BlockIDhas both a height and a hash, the hash is used. - Unmined transactions are reported at height 0.
Blocks
GetBlockRange has no limit on range size, and it never returns fewer blocks than asked
for without an error. If the range extends past the tip, the stream sends every block up to the
tip and then ends with OUT_OF_RANGE. If the range starts past the tip, it fails with
OUT_OF_RANGE before sending anything.
GetBlock returns every pool, including transparent inputs and outputs, as lightwalletd
does. It takes no poolTypes.
CompactTx.fee is filled in for mined and mempool transactions. lightwalletd leaves it 0.
Zingod reports 0 for coinbase transactions, for a wallet’s own transaction before another node
lists it, and for fees of 2^32 zatoshis or more, which do not fit the uint32 field.
Tree state
GetTreeState answers for exactly the requested height. The block hash is in display order.
Each tree field holds a hex-encoded CommitmentTree:
""if the pool is not active at that height."000000"if the pool is active but empty.
GetSubtreeRoots either streams the whole answer or fails before the first message. A
startIndex past the last root returns an empty stream. maxEntries 0 returns every root. An
unknown shieldedProtocol fails with UNIMPLEMENTED.
Transactions
GetTransaction accepts only a txid (TxFilter.hash). A (block, index) filter fails with
INVALID_ARGUMENT. Zingod asks its validators for the transaction and returns NOT_FOUND only
when every validator asked reports it missing. If no validator responds, Zingod serves the
transaction from its mempool or fetches its block from p2p peers; if neither works, the call fails
with UNAVAILABLE.
SendTransaction reports a rejected transaction in the response, as lightwalletd does. The
call returns OK with:
errorCode | Meaning |
|---|---|
0 | accepted; errorMessage is the txid, quoted, in display order |
-22 | the transaction could not be parsed |
-25 | the transaction has expired, or targets the wrong consensus branch |
| other | the validator rejected it; errorMessage is the validator’s message verbatim |
The call itself fails with UNAVAILABLE only when no validator accepted the transaction and at
least one could not be reached. Retry in that case.
Mempool
The mempool Zingod serves contains transactions listed by a trusted validator, transactions a
quorum of its p2p peers agree on, and transactions submitted through this server. Both mempool
methods fail with UNAVAILABLE when no trusted validator at the best block is reachable and
Zingod has too few p2p peers for a quorum.
GetMempoolStream starts with the current mempool, then sends each new transaction once,
each with height 0. The stream ends with OK when a new block arrives. By then the server already
serves that block, so a GetLatestBlock sent next returns it.
GetMempoolTx accepts poolTypes like GetBlockRange.
Transparent addresses
The address methods (GetAddressUtxos, GetAddressUtxosStream, GetTaddressBalance,
GetTaddressBalanceStream, GetTaddressTransactions) answer at the served tip.
- A request covering more than 100,000 received outputs (the default limit) fails with
RESOURCE_EXHAUSTED. Send fewer addresses per request. - An address for another network fails with
INVALID_ARGUMENT. - Do not page
GetAddressUtxosbystartHeight. It is not a unique resume key and paging can stall (lightwallet-protocol#33).
Address queries tell the server which addresses belong to one wallet. To find transparent
activity privately, request the TRANSPARENT pool in GetBlockRange and match scripts locally.
Coinbase inputs are omitted (a coinbase is the transaction at CompactTx.index 0), and
CompactTxIn carries no value, so balances still require the address methods.
Fees
GetStandardFee returns the ZIP 317 marginal fee per logical action, in zatoshis, for the
next block. The estimator version is 0.
Pool filtering
poolTypes on GetBlockRange and GetMempoolTx selects the pools returned:
- Empty: Sapling, Orchard and Ironwood. Transparent data is excluded.
- Otherwise: exactly the listed pools.
POOL_TYPE_INVALIDor an unknown value fails withINVALID_ARGUMENT.
GetBlockRangeNullifiers ignores TRANSPARENT, as the protocol specifies.
Deprecated and unsupported
GetBlockRangeNullifiersandGetTaddressTxidsare deprecated and will be removed. UseGetBlockRangewith shieldedpoolTypes, andGetTaddressTransactions.GetBlockNullifiersandPingare not supported and returnUNIMPLEMENTED.- Operators can disable individual indexes. Methods that depend on a disabled index return
UNIMPLEMENTED./statuslists the methods each enabled index serves.
Status codes
How to handle each gRPC status Zingod returns. A rejected SendTransaction is not an error
status; it is reported in the response (details).
| Code | When | Client action |
|---|---|---|
UNAVAILABLE | the server is starting, at capacity, or cannot reach its data sources; a stream had nothing to send for 5 minutes | retry with backoff |
NOT_FOUND | the height, hash or txid is not in the chain, or is above the index’s tip | do not retry the same request |
OUT_OF_RANGE | GetBlockRange reached the tip, after sending every block up to it | retry once the tip advances |
INVALID_ARGUMENT | malformed request, height above the protocol maximum, invalid or wrong-network address, invalid poolTypes | fix the request |
RESOURCE_EXHAUSTED | request body too large (64 KiB; 2 MB + 1 KiB for SendTransaction), or too many address outputs in one request | send less per request |
DEADLINE_EXCEEDED | the request’s grpc-timeout passed, or the request body took more than 30 seconds to arrive | retry with a longer deadline |
UNIMPLEMENTED | the method or shieldedProtocol is unsupported, or its index is disabled | do not retry |
PERMISSION_DENIED | the browser origin is blocked by the operator | do not retry |
INTERNAL | server fault | report to the operator |
When the server is at capacity, the UNAVAILABLE response carries grpc-retry-pushback-ms
(250). Wait at least that long before retrying.
Protocol files
The wallet API is defined by the
LightWallet Protocol .proto files and their
CHANGELOG. There is no ZIP for it. Zingod implements one protocol release and reports it in
GetLightdInfo.lightwalletProtocolVersion.
The files below are the ones Zingod builds from. They differ from upstream in service.proto:
PingandGetBlockNullifiersare removed, and returnUNIMPLEMENTED.GetBlockRangeNullifiersandGetTaddressTxidsare marked deprecated.
service.proto
// Copyright (c) 2019-2020 The Zcash developers
// Distributed under the MIT software license, see the accompanying
// file COPYING or https://www.opensource.org/licenses/mit-license.php .
syntax = "proto3";
package cash.z.wallet.sdk.rpc;
option go_package = "lightwalletd/walletrpc";
option swift_prefix = "";
import "compact_formats.proto";
// An identifier for a Zcash value pool.
enum PoolType {
POOL_TYPE_INVALID = 0;
TRANSPARENT = 1;
SAPLING = 2;
ORCHARD = 3;
IRONWOOD = 4;
}
// A BlockID message contains identifiers to select a block: a height or a
// hash. Support for specification by hash is not mandatory. (If `hash` is
// non-empty, the rpc may return an error.) This field is present to support
// a possible future upgrade.
message BlockID {
uint64 height = 1;
bytes hash = 2;
}
// BlockRange specifies a series of blocks from start to end inclusive.
// Both BlockIDs must be heights; specification by hash is not yet supported.
//
// If no pool types are specified, the server should default to the legacy
// behavior of returning only data relevant to the shielded (Sapling, Orchard,
// and Ironwood) pools; otherwise, the server should prune `CompactBlock`s
// returned to include only data relevant to the requested pool types. Clients MUST
// verify that the version of the server they are connected to are capable
// of returning pruned and/or transparent data before setting `poolTypes`
// to a non-empty value.
message BlockRange {
BlockID start = 1;
BlockID end = 2;
repeated PoolType poolTypes = 3;
}
// A TxFilter contains the information needed to identify a particular
// transaction: either a block and an index, or a direct transaction hash.
// Currently, only specification by hash is supported.
message TxFilter {
BlockID block = 1; // block identifier, height or hash
uint64 index = 2; // index within the block
bytes hash = 3; // transaction ID (hash, txid)
}
// RawTransaction contains the complete transaction data. It also optionally includes
// the block height in which the transaction was included, or, when returned
// by GetMempoolStream(), the latest block height.
//
// FIXME: the documentation here about mempool status contradicts the documentation
// for the `height` field. See https://github.com/zcash/librustzcash/issues/1484
message RawTransaction {
// The serialized representation of the Zcash transaction.
bytes data = 1;
// The height at which the transaction is mined, or a sentinel value.
//
// Due to an error in the original protobuf definition, it is necessary to
// reinterpret the result of the `getrawtransaction` RPC call. Zcashd will
// return the int64 value `-1` for the height of transactions that appear
// in the block index, but which are not mined in the main chain. Here, the
// height field of `RawTransaction` was erroneously created as a `uint64`,
// and as such we must map the response from the zcashd RPC API to be
// representable within this space. Additionally, the `height` field will
// be absent for transactions in the mempool, resulting in the default
// value of `0` being set. Therefore, the meanings of the `height` field of
// the `RawTransaction` type are as follows:
//
// * height 0: the transaction is in the mempool
// * height 0xffffffffffffffff: the transaction has been mined on a fork that
// is not currently the main chain
// * any other height: the transaction has been mined in the main chain at the
// given height
uint64 height = 2;
}
// A SendResponse encodes an error code and a string. It is currently used
// only by SendTransaction(). If error code is zero, the operation was
// successful; if non-zero, it and the message specify the failure.
message SendResponse {
int32 errorCode = 1;
string errorMessage = 2;
}
// Chainspec is a placeholder to allow specification of a particular chain fork.
message ChainSpec {}
// Empty is for gRPCs that take no arguments, currently only GetLightdInfo.
message Empty {}
// LightdInfo returns various information about this lightwalletd instance
// and the state of the blockchain.
message LightdInfo {
string version = 1;
string vendor = 2;
bool taddrSupport = 3; // true
string chainName = 4; // either "main" or "test"
uint64 saplingActivationHeight = 5; // depends on mainnet or testnet
string consensusBranchId = 6; // protocol identifier, see consensus/upgrades.cpp
uint64 blockHeight = 7; // latest block on the best chain
string gitCommit = 8;
string branch = 9;
string buildDate = 10;
string buildUser = 11;
uint64 estimatedHeight = 12; // less than tip height if zcashd is syncing
string zcashdBuild = 13; // example: "v4.1.1-877212414"
string zcashdSubversion = 14; // example: "/MagicBean:4.1.1/"
string donationAddress = 15; // Zcash donation UA address
string upgradeName = 16; // name of next pending network upgrade, empty if none scheduled
uint64 upgradeHeight = 17; // height of next pending upgrade, zero if none is scheduled
string lightwalletProtocolVersion = 18; // version of https://github.com/zcash/lightwallet-protocol served by this server
}
// TransparentAddressBlockFilter restricts the results of the GRPC methods that
// use it to the transactions that involve the given address and were mined in
// the specified block range. Non-default values for both the address and the
// block range must be specified. Mempool transactions are not included.
//
// The `poolTypes` field of the `range` argument should be ignored.
// Implementations MAY consider it an error if any pool types are specified.
message TransparentAddressBlockFilter {
string address = 1; // t-address
BlockRange range = 2; // start, end heights only
}
message Address {
string address = 1;
}
message AddressList {
repeated string addresses = 1;
}
message Balance {
int64 valueZat = 1;
}
// Request parameters for the `GetMempoolTx` RPC.
message GetMempoolTxRequest {
// A list of transaction ID byte string suffixes that should be excluded
// from the response. These suffixes may be produced either directly from
// the underlying txid bytes, or, if the source values are encoded txid
// strings, by truncating the hexadecimal representation of each
// transaction ID to an even number of characters, and then hex-decoding
// and then byte-reversing this value to obtain the byte representation.
repeated bytes exclude_txid_suffixes = 1;
// We reserve field number 2 for a potential future `exclude_txid_prefixes`
// field.
reserved 2;
// The server must prune `CompactTx`s returned to include only data
// relevant to the requested pool types. If no pool types are specified,
// the server should default to the legacy behavior of returning only data
// relevant to the shielded (Sapling, Orchard, and Ironwood) pools.
repeated PoolType poolTypes = 3;
}
// The TreeState is derived from the Zcash z_gettreestate rpc.
message TreeState {
string network = 1; // "main" or "test"
uint64 height = 2; // block height
string hash = 3; // block id
uint32 time = 4; // Unix epoch time when the block was mined
string saplingTree = 5; // sapling commitment tree state
string orchardTree = 6; // orchard commitment tree state
string ironwoodTree = 7; // ironwood commitment tree state
}
enum ShieldedProtocol {
sapling = 0;
orchard = 1;
ironwood = 2;
}
message GetSubtreeRootsArg {
uint32 startIndex = 1; // Index identifying where to start returning subtree roots
ShieldedProtocol shieldedProtocol = 2; // Shielded protocol to return subtree roots for
uint32 maxEntries = 3; // Maximum number of entries to return, or 0 for all entries.
}
message SubtreeRoot {
bytes rootHash = 2; // The 32-byte Merkle root of the subtree.
bytes completingBlockHash = 3; // The hash of the block that completed this subtree.
uint64 completingBlockHeight = 4; // The height of the block that completed this subtree in the main chain.
}
// Results are sorted by height, which makes it easy to issue another
// request that picks up from where the previous left off.
message GetAddressUtxosArg {
repeated string addresses = 1;
uint64 startHeight = 2;
uint32 maxEntries = 3; // zero means unlimited
}
message GetAddressUtxosReply {
string address = 6;
bytes txid = 1;
int32 index = 2;
bytes script = 3;
int64 valueZat = 4;
uint64 height = 5;
}
message GetAddressUtxosReplyList {
repeated GetAddressUtxosReply addressUtxos = 1;
}
// StandardFee is the reply to the GetStandardFee rpc: the ZIP 317
// marginal fee, in zatoshis per logical action, that wallets should
// pay for a transaction to be mined in the next block, as reported by
// the backend's `getstandardfee` RPC.
message StandardFee {
uint64 feeZat = 1; // standard fee in zatoshis per logical action
uint32 estimatorVersion = 2; // backend fee-estimator version, 0 = static ZIP 317 marginal fee
}
service CompactTxStreamer {
// Return the BlockID of the block at the tip of the best chain
rpc GetLatestBlock(ChainSpec) returns (BlockID) {}
// Return the compact block corresponding to the given block identifier.
//
// The returned `CompactBlock` includes transaction data for all value
// pools, including transparent inputs (`vin`) and outputs (`vout`). This
// differs from `GetBlockRange`, which supports filtering by pool type and
// defaults to returning only shielded (Sapling, Orchard, and Ironwood)
// data. Clients that require only data for specific pools should use
// `GetBlockRange` with the appropriate `poolTypes` set.
//
// Note: the single null-outpoint input for coinbase transactions is
// omitted from the `vin` field of the corresponding `CompactTx`. See the
// documentation of the `CompactTx` message for details.
rpc GetBlock(BlockID) returns (CompactBlock) {}
// Return a list of consecutive compact blocks in the specified range,
// which is inclusive of `range.end`.
//
// If range.start <= range.end, blocks are returned increasing height order;
// otherwise blocks are returned in decreasing height order.
rpc GetBlockRange(BlockRange) returns (stream CompactBlock) {}
// !!! TODO: REMOVE THIS. DEPRECATED. Served by Zingod only because pepper-sync still calls it;
// !!! delete once pepper-sync requests `GetBlockRange` with `poolTypes` instead.
//
// Return a stream of compact blocks for the specified range, where each
// block contains only nullifier information for the shielded pools
// (Sapling spend nullifiers, Orchard action nullifiers, and Ironwood action
// nullifiers). Transparent transaction data, Sapling outputs, full
// Orchard/Ironwood action data, and commitment tree sizes are not included.
// Implementations MUST ignore any
// `PoolType::TRANSPARENT` member of the `poolTypes` field of the request.
//
// Note: this method is deprecated; use `GetBlockRange` with the
// appropriate `poolTypes` instead.
rpc GetBlockRangeNullifiers(BlockRange) returns (stream CompactBlock) {
option deprecated = true;
}
// Return the requested full (not compact) transaction (as from zcashd)
rpc GetTransaction(TxFilter) returns (RawTransaction) {}
// Submit the given transaction to the Zcash network
rpc SendTransaction(RawTransaction) returns (SendResponse) {}
// !!! TODO: REMOVE THIS. DEPRECATED. Served by Zingod only because pepper-sync still calls it;
// !!! delete once pepper-sync calls `GetTaddressTransactions` (same request, same response).
//
// Return RawTransactions that match the given transparent address filter.
//
// Note: This function is misnamed, it returns complete `RawTransaction` values, not TxIds.
// NOTE: this method is deprecated, please use GetTaddressTransactions instead.
rpc GetTaddressTxids(TransparentAddressBlockFilter) returns (stream RawTransaction) {
option deprecated = true;
}
// Return the transactions corresponding to the given t-address within the given block range.
// Mempool transactions are not included in the results.
rpc GetTaddressTransactions(TransparentAddressBlockFilter) returns (stream RawTransaction) {}
rpc GetTaddressBalance(AddressList) returns (Balance) {}
rpc GetTaddressBalanceStream(stream Address) returns (Balance) {}
// Returns a stream of the compact transaction representation for transactions
// currently in the mempool. The results of this operation may be a few
// seconds out of date. If the `exclude_txid_suffixes` list is empty,
// return all transactions; otherwise return all *except* those in the
// `exclude_txid_suffixes` list (if any); this allows the client to avoid
// receiving transactions that it already has (from an earlier call to this
// RPC). The transaction IDs in the `exclude_txid_suffixes` list can be
// shortened to any number of bytes to make the request more
// bandwidth-efficient; if two or more transactions in the mempool match a
// txid suffix, none of the matching transactions are excluded. Txid
// suffixes in the exclude list that don't match any transactions in the
// mempool are ignored.
rpc GetMempoolTx(GetMempoolTxRequest) returns (stream CompactTx) {}
// Return a stream of current Mempool transactions. This will keep the output stream open while
// there are mempool transactions. It will close the returned stream when a new block is mined.
rpc GetMempoolStream(Empty) returns (stream RawTransaction) {}
// GetTreeState returns the note commitment tree state corresponding to the given block.
// See section 3.7 of the Zcash protocol specification. It returns several other useful
// values also (even though they can be obtained using GetBlock).
// The block can be specified by either height or hash.
rpc GetTreeState(BlockID) returns (TreeState) {}
rpc GetLatestTreeState(Empty) returns (TreeState) {}
// Returns a stream of information about roots of subtrees of the note commitment tree
// for the specified shielded protocol (Sapling, Orchard, or Ironwood).
rpc GetSubtreeRoots(GetSubtreeRootsArg) returns (stream SubtreeRoot) {}
rpc GetAddressUtxos(GetAddressUtxosArg) returns (GetAddressUtxosReplyList) {}
rpc GetAddressUtxosStream(GetAddressUtxosArg) returns (stream GetAddressUtxosReply) {}
// Return information about this lightwalletd instance and the blockchain
rpc GetLightdInfo(Empty) returns (LightdInfo) {}
// Return the standard fee (the ZIP 317 marginal fee, in zatoshis per
// logical action) that the backend recommends wallets pay for a
// transaction to be mined in the next block. Requires a backend that
// implements the `getstandardfee` RPC. Backends that lack it return
// Unimplemented.
rpc GetStandardFee(Empty) returns (StandardFee) {}
}
compact_formats.proto
// Copyright (c) 2019-2021 The Zcash developers
// Distributed under the MIT software license, see the accompanying
// file COPYING or https://www.opensource.org/licenses/mit-license.php .
syntax = "proto3";
package cash.z.wallet.sdk.rpc;
option go_package = "lightwalletd/walletrpc";
option swift_prefix = "";
// REMINDER: proto3 fields are all optional. A field that is not present will be set to its zero/false/empty
// value.
// Information about the state of the chain as of a given block.
message ChainMetadata {
uint32 saplingCommitmentTreeSize = 1; // the size of the Sapling note commitment tree as of the end of this block
uint32 orchardCommitmentTreeSize = 2; // the size of the Orchard note commitment tree as of the end of this block
uint32 ironwoodCommitmentTreeSize = 3; // the size of the Ironwood note commitment tree as of the end of this block
}
// A compact representation of a Zcash block.
//
// CompactBlock is a packaging of ONLY the data from a block that's needed to:
// 1. Detect a payment to your Shielded address
// 2. Detect a spend of your Shielded notes
// 3. Update your witnesses to generate new spend proofs.
// 4. Spend UTXOs associated to t-addresses of your wallet.
//
// Currently, the `header` field should always be unset (empty). In the future,
// the presence or absence of header data may be made dependent on request
// parameters, although it is likely that such flexibility will only be provided via
// newly-added service methods, not via existing APIs.
message CompactBlock {
// Field 1 (`protoVersion`) was removed; see
// https://github.com/zcash/lightwallet-protocol/issues/25
reserved 1;
reserved "protoVersion";
uint64 height = 2; // the height of this block
bytes hash = 3; // if present, the ID (hash) of this block
bytes prevHash = 4; // if present, the ID (hash) of this block's predecessor
uint32 time = 5; // if non-zero, Unix epoch time when the block was mined
bytes header = 6; // if present, full header (as returned by the getblock RPC)
repeated CompactTx vtx = 7; // zero or more compact transactions from this block
ChainMetadata chainMetadata = 8; // information about the state of the chain as of this block
}
// A compact representation of a Zcash transaction.
//
// CompactTx contains the minimum information for a wallet to know if this transaction
// is relevant to it (either pays to it or spends from it) via shielded elements. Additionally,
// it can optionally include the minimum necessary data to detect payments to transparent addresses
// related to your wallet.
message CompactTx {
// The index of the transaction within the block.
uint64 index = 1;
// The id of the transaction as defined in
// [§ 7.1.1 ‘Transaction Identifiers’](https://zips.z.cash/protocol/protocol.pdf#txnidentifiers)
// This byte array MUST be in protocol order and MUST NOT be reversed
// or hex-encoded; the byte-reversed and hex-encoded representation is
// exclusively a textual representation of a txid.
bytes txid = 2;
// The transaction fee: present if server can provide. In the case of a
// stateless server and a transaction with transparent inputs, this will be
// unset because the calculation requires reference to prior transactions.
// If there are no transparent inputs, the fee will be calculable as:
// valueBalanceSapling + valueBalanceOrchard + valueBalanceIronwood
// + sum(vPubNew) - sum(vPubOld) - sum(tOut)
uint32 fee = 3;
repeated CompactSaplingSpend spends = 4;
repeated CompactSaplingOutput outputs = 5;
repeated CompactOrchardAction actions = 6;
repeated CompactOrchardAction ironwoodActions = 9;
// `CompactTxIn` values corresponding to the `vin` entries of the full transaction.
//
// Note: the single null-outpoint input for coinbase transactions is omitted. Light
// clients can test `CompactTx.index == 0` to determine whether a `CompactTx`
// represents a coinbase transaction, as the coinbase transaction is always the
// first transaction in any block.
repeated CompactTxIn vin = 7;
// A sequence of transparent outputs being created by the transaction.
repeated TxOut vout = 8;
}
// A compact representation of a transparent transaction input.
message CompactTxIn {
// The id of the transaction that generated the output being spent. This
// byte array must be in protocol order and MUST NOT be reversed or
// hex-encoded.
bytes prevoutTxid = 1;
// The index of the output being spent in the `vout` array of the
// transaction referred to by `prevoutTxid`.
uint32 prevoutIndex = 2;
}
// A transparent output being created by the transaction.
//
// This contains identical data to the `TxOut` type in the transaction itself, and
// thus it is not "compact".
message TxOut {
// The value of the output, in Zatoshis.
uint64 value = 1;
// The script pubkey that must be satisfied in order to spend this output.
bytes scriptPubKey = 2;
}
// A compact representation of a [Sapling Spend](https://zips.z.cash/protocol/protocol.pdf#spendencodingandconsensus).
//
// CompactSaplingSpend is a Sapling Spend Description as described in 7.3 of the Zcash
// protocol specification.
message CompactSaplingSpend {
bytes nf = 1; // Nullifier (see the Zcash protocol specification)
}
// A compact representation of a [Sapling Output](https://zips.z.cash/protocol/protocol.pdf#outputencodingandconsensus).
//
// It encodes the `cmu` field, `ephemeralKey` field, and a 52-byte prefix of the
// `encCiphertext` field of a Sapling Output Description. Total size is 116 bytes.
message CompactSaplingOutput {
bytes cmu = 1; // Note commitment u-coordinate.
bytes ephemeralKey = 2; // Ephemeral public key.
bytes ciphertext = 3; // First 52 bytes of ciphertext.
}
// A compact representation of an [Orchard Action](https://zips.z.cash/protocol/protocol.pdf#actionencodingandconsensus).
message CompactOrchardAction {
bytes nullifier = 1; // [32] The nullifier of the input note
bytes cmx = 2; // [32] The x-coordinate of the note commitment for the output note
bytes ephemeralKey = 3; // [32] An encoding of an ephemeral Pallas public key
bytes ciphertext = 4; // [52] The first 52 bytes of the encCiphertext field
}
Quick start
Requirements
- A Zcash validator with JSON-RPC enabled: Zebra
(
zebrad, tested with 6.2.3) or Zakura (zakurad). Set[rpc] listen_addrin its config. - Local storage for the indexes, preferably NVMe (first sync).
Zingod uses these Zcash JSON-RPC methods: getblock,
getblockchaininfo, getblockhash, getblockheader, getdeprecationinfo, getinfo,
getpeerinfo, getrawmempool, getrawtransaction and sendrawtransaction. If the validator
restricts its RPC methods, allow these. Zingod also calls checkadmission when the validator
supports it.
Run with Docker
-
Generate a config with every key at its default:
docker run --rm zingodevops/zingod generate-config -o /dev/stdout > zingod.toml -
Edit
zingod.toml:- Set
networktomainnet,testnetorregtest. - Set
jsonrpc_addressunder[[trusted_validators]]to your validator. - Set
tlsunder[serve]. The container listens on all interfaces, so Zingod requires it:tls = "off"if wallets connect only from this host, or a certificate pair (TLS).
- Set
-
Start it:
docker run -d --name zingod -p 127.0.0.1:8137:8137 \ -v ./zingod.toml:/etc/zingod/zingod.toml:ro \ -v zingod-data:/var/lib/zingod \ zingodevops/zingod -
Wait for the first sync. It takes about an hour on NVMe with the validator on the same host, and longer over a network. An index snapshot skips most of it. Check progress over HTTP on the gRPC port:
curl -s http://localhost:8137/status # state and tip curl -sf http://localhost:8137/readyz && echo serving # 200 once ready
Wallets can then connect to http://localhost:8137. See Docker for volumes, TLS
mounts and Compose.
Run on a host
Building requires protoc on PATH.
cargo install --locked --git https://github.com/zingolabs/zingod zingod
zingod generate-config # writes $XDG_CONFIG_HOME/zingod/zingod.toml
zingod start # or: zingod start -c /path/to/zingod.toml
Run it under a service manager such as systemd. zingod --help lists
every subcommand.
Try it on the Public Testnet
zebrad.toml and zingod.toml are a matching pair of configs for
the Public Testnet, with zebrad’s JSON-RPC on 127.0.0.1:18232.
- Install zebrad:
cargo install zebrad --locked. - In
zebrad.toml, replace<PATH_TO_ZEBRA>with a writable directory. Inzingod.toml, replace<ZINGOD_DATA>. - Run
zebrad -c zebrad.toml start. - In another shell, run
zingod start -c zingod.toml.
Zingod serves gRPC on 127.0.0.1:8137.
Configuration
Zingod reads one TOML file: $XDG_CONFIG_HOME/zingod/zingod.toml, or the path given with -c.
zingod generate-config writes a file with every key at its default.
Settings are applied in this order, later ones winning:
- Built-in defaults.
- The config file.
- Environment variables prefixed
ZINGOD_CONFIG_, with__between nested keys. For example,ZINGOD_CONFIG_SYNC__CONCURRENCY=64setsconcurrencyunder[sync].
Every subcommand validates the whole config and refuses to run on an unknown key or invalid value.
Default paths are under the XDG base directories. If there is no home directory (for example,
HOME unset under a service manager), set each path explicitly: serve.state_dir, the path of
every enabled index, and p2p.cache_dir if [p2p] is enabled.
Overview
| Key or table | Purpose |
|---|---|
network | mainnet (default), testnet or regtest |
validator_client | how Zingod talks to validators (below) |
[[trusted_validators]] | one table per validator: address, credentials, timeouts, connection budget, priority |
[submission] | how SendTransaction retries relaying |
[p2p] | Zingod’s own Zcash p2p peers (off by default) |
[serve] | gRPC listen address, TLS, gRPC-web, state directory, drain delay |
[grpc] | connection and stream limits, timeouts, trusted proxies, overload thresholds |
[metrics] | admin listener for metrics and probes (off by default) |
[sync] | block fetching and finality depth |
[index.<name>] | per index: enabled and path |
[snapshot] | bootstrap empty indexes from an index snapshot |
Trusted validators do not vote. Each one adds failover, so two validators survive one going down.
At least one index must be enabled. utxo_set is enabled together with compact_block.
Validator client
validator_client sets how Zingod talks to every trusted validator:
legacy_rpc(default): JSON-RPC for everything. Works with zebrad, zakurad and zcashd.grpc: blocks over zebrad’s indexer gRPC at each validator’sindexer_address, everything else over JSON-RPC. Requires a zebrad build whose indexer servesGetBlockRange(Zingod’s Zebra fork; stock zebrad does not). Do not use it with zakurad, whoseGetBlockRangereturns a different message type.
The indexer gRPC connection is plaintext unless [trusted_validators.indexer_tls] configures
mutual TLS.
Reference
Every key, with its default and a description. This is the
Public Testnet zingod.toml
(download).
# zingod config paired with zebrad.toml (Testnet)
#
# - Env overrides + unknown-key rule: https://zingolabs.github.io/zingod/operators/configuration.html
# - `zingod generate-config` = every key at its default
# mainnet | testnet | regtest (default mainnet)
# - declared, never read off the validator (Zebra on regtest reports "test")
network = "testnet"
# How zingod talks to every [[trusted_validators]] entry (default legacy_rpc)
# - legacy_rpc = JSON-RPC for everything (zebrad, zakurad or zcashd); blocks arrive as hex inside
# JSON
# - grpc = blocks over zebrad's indexer gRPC (GetBlock, GetBlockRange) as stored bytes, no hex
# or JSON; everything else over JSON-RPC. Needs each validator's indexer_address and a zebrad
# serving GetBlockRange: Zingod's zebra fork only (elicbarbieri/zebra perf/indexer-block-range)
# validator_client = "grpc"
# Admin listener (/metrics, /livez, /readyz). Omit listen_address to disable.
# [metrics]
# listen_address = "127.0.0.1:9999"
# Validators Zingod trusts (Zebra or Zakura JSON-RPC), at least one; no vote (`priority` orders who
# is asked)
# - tip = their most-work header (trusted: zebra validated it), served while any of them holds it
# - one listing admits a mempool transaction; submission: see [submission]
# - 2+ = survives one going down (no vote: more never weakens it)
# - down at boot or later = retried, never fatal
[[trusted_validators]]
jsonrpc_address = "127.0.0.1:18232"
# No credentials + a loopback address = no auth until the validator refuses, then its default
# cookie: <cache dir>/zebra/.cookie or <cache dir>/zakura/.cookie (both = the one written last)
# cookie_path = "<PATH_TO_ZEBRA>/.cookie"
# basic auth
# user = "xxxxxx"
# password = "xxxxxx"
# connect_timeout_secs = 2
# Silence before a request fails, never total duration (raise for a slow or lossy link)
# read_timeout_secs = 30
# Link budget to this validator (zingod never exceeds it): at least 5 connections, one for the poll,
# one reserved per lane (control, interactive, bulk) + one shared, the rest to whichever lane
# waits
# max_connections = 32
# Largest JSON-RPC reply read, 4 (one full block) ..= 32; batches are split to fit
# max_response_mib = 32
# zebrad's indexer gRPC (its `indexer_listen_addr`): tip + mempool push streams wake polling at
# once; polling slows to a 15 s reconcile while they are up. Unset = poll every second.
# validator_client = "grpc" fetches blocks here too (then required)
# indexer_address = "127.0.0.1:8230"
# Tier: every 0 asked before any 1 (a farther validator = 1: overflow + hedges only)
# priority = 0
# Mutual TLS to indexer_address (zakurad requires it off loopback: its [rpc.indexer_tls], whose
# client_ca_file signs cert_path). Unset = plaintext.
# [trusted_validators.indexer_tls]
# ca_path = "/etc/zingod/indexer-ca.pem"
# cert_path = "/etc/zingod/indexer-client.pem"
# key_path = "/etc/zingod/indexer-client-key.pem"
# Name on the validator's certificate (unset = the indexer_address host)
# server_name = "zakura-1.internal"
# [[trusted_validators]]
# jsonrpc_address = "127.0.0.1:18242"
# SendTransaction: a trusted validator's checkadmission answers the wallet (refused = that
# refusal, nothing sent); then one relay per attempt, through a random trusted validator, or
# the peers with [p2p] on (defaults shown)
[submission]
# Seconds an accepted transaction gets to be seen beyond the nodes relayed to before relaying again
propagation_threshold_secs = 15
# Relays per transaction at most; without a check, the wallet is answered at the first acceptance
max_attempts = 4
# Zingod's own peers on the Zcash p2p network (zebra-network), off by default: headers and blocks
# (checked by proof of work and merkle root: the tip keeps up while every trusted validator is
# down; finality too, once peers in more than 5 netgroups carry the chain), each peer's mempool
# polled (a transaction > 2/3 of them list from > 5 netgroups is served after Zingod's consensus
# checks: needs index.compact_block), and SendTransaction relays (zebra's own, to a third of the
# peers per attempt; then one trusted validator if still unseen). Zingod serves no peer (listener
# on loopback, no services advertised).
# [p2p]
# enabled = true
# Each peer holds one inbound slot on a public node: keep it modest
# peer_target = 8
# Empty = the network's DNS seeders (regtest has none: list peers)
# initial_peers = []
# cache_dir = "<ZINGOD_DATA>/peers"
[serve]
# CompactTxStreamer gRPC over HTTP/2
grpc_listen_address = "127.0.0.1:8137"
# serve browser (gRPC-web) clients too: permissive CORS (*) on this listener, native gRPC
# untouched; refuse origins with `zingod origin`. off = no CORS, browsers refused
grpc_web = false
# Max transparent receives one address request walks (all its addresses); over = RESOURCE_EXHAUSTED
# max_address_rows = 100000
# zingod's own state, kept across restarts (`zingod hold` writes its marker here);
# default <data dir>/zingod/state
# state_dir = "/var/lib/zingod"
# held: /readyz fails with "held" while still serving this long, then every open gRPC connection
# gets a GOAWAY; >= the balancer's down detection + DNS TTL
hold_goaway_delay_secs = 0
# TLS: unset = "off" on a loopback grpc_listen_address, refused at boot on any other
# - { cert_path, key_path } = PEM files, re-read on change (renewal = no restart)
# - "off" = plaintext (a TLS-terminating proxy in front; warned at boot off loopback)
# - "self_signed" = pair generated under <state_dir>/tls at first boot (fingerprint logged; only a
# client that pins it connects)
# tls = { cert_path = "/etc/zingod/fullchain.pem", key_path = "/etc/zingod/key.pem" }
# Serving caps (defaults shown); every cap refuses rather than queues, except the read lanes
[grpc]
# excess accepts closed; + 1024 must fit the open-file limit or boot fails
max_connections = 4096
# per client: the peer, or the address a trusted proxy's PROXY header names
max_connections_per_ip = 128
# HTTP/2 SETTINGS_MAX_CONCURRENT_STREAMS: every stream on one connection, of any method (as
# lightwalletd's grpc-max-concurrent-streams)
max_streams_per_connection = 100
# work streams (every method but GetMempoolStream) across all connections; excess = UNAVAILABLE +
# retry hint
max_work_streams = 2048
# work streams admitted at once follow measured latency, max_work_streams the ceiling: a saturated
# node refuses new work within seconds (UNAVAILABLE + retry hint); false = always max_work_streams
adaptive_work_streams = true
# GetMempoolStream subscriptions (idle until the next block), counted apart from max_work_streams
max_subscriptions = 4096
# index reads in flight on the blocking pool, per lane; excess waits in its own lane
# - point: one block / tree state / root list; range: one GetBlockRange window; scan: one address
# history
max_point_reads = 64
max_range_reads = 32
max_scan_reads = 4
# no-progress bounds, never a cap on a stream's length (the client's grpc-timeout is its deadline)
# data its client does not read for this long closes its connection
unread_timeout_secs = 300
# a stream with nothing to send for this long ends UNAVAILABLE (GetMempoolStream exempt)
silent_timeout_secs = 300
# GOAWAY after this long (±10% per connection): the client reconnects and re-resolves DNS, so a
# load balancer or a replaced node moves long-held wallet connections; 0 = never
max_connection_age_secs = 1800
# in-flight streams of a GOAWAY'd connection get this long to finish (a whole GetBlockRange sync)
max_connection_age_grace_secs = 600
# GOAWAY a connection with no open stream for this long (frees its per-client slot); 0 = never
max_connection_idle_secs = 1800
# proxies that open every connection with a PROXY header (v1/v2)
# - https://zingolabs.github.io/zingod/operators/network.html#behind-a-proxy
trusted_proxies = []
# SIGTERM/SIGINT closes the listener at once; open connections get this long to finish (GOAWAY
# sent). Out of a load balancer's rotation first = `zingod hold`
# - https://zingolabs.github.io/zingod/operators/maintenance.html#operator-hold
shutdown_grace_secs = 10
# When /readyz reports `Overloaded: <cause>` (0 = that cause off; `zingod overload` overrides live)
[grpc.overload]
# share of each minute zingod's cgroup waited (Linux PSI)
cpu_pressure_percent = 25
io_pressure_percent = 25
# One block-fetch pipeline, shared by every index (defaults shown; all non-zero)
[sync]
# Blocks below the tip kept reorg-able (in memory, served, written once buried); >= 1000 off regtest
finalised_depth = 1000
# Block fetches (and folds) in flight ahead of the next one needed, over every trusted validator
concurrency = 32
# MiB of finalized blocks one index may lag the fetch before it throttles the pipeline
queue_mib = 256
# The disk engine's map tables (one engine per process: every index shares merge_slots)
# [sync.lsm_store]
# Same-tier segments merged into one (>= 2; higher = less rewriting, larger merges)
# fanout = 16
# Merges doing disk I/O at once (the rest wait, smallest first)
# merge_slots = 4
# MiB/s every merge together may read + write (commits are never paced)
# merge_mib_per_sec = 200
# One table per index: `enabled` (default true; off = its methods UNIMPLEMENTED) and `path`
# (default $XDG_DATA_HOME/zingod/indexes/<name>); an omitted table takes both defaults
[index.compact_block]
# GetBlock, GetBlockRange (GetLatestBlock, GetLightdInfo = the served tip, either way)
# - its fee index (each CompactTx.fee) lives beside it, in `utxo_set`
enabled = true
path = "<ZINGOD_DATA>/compact_block"
[index.block_hash]
# GetBlock / GetTreeState by hash (by height unaffected)
enabled = true
path = "<ZINGOD_DATA>/block_hash"
[index.tree_state]
# GetTreeState, GetLatestTreeState, GetSubtreeRoots
enabled = true
path = "<ZINGOD_DATA>/tree_state"
[index.transparent_address]
# GetAddressUtxos(+Stream), GetTaddressBalance(+Stream), GetTaddressTransactions
enabled = true
path = "<ZINGOD_DATA>/transparent_address"
[index.txid_location]
# GetTransaction / GetTaddressTransactions bytes with no validator answering: the block it names,
# fetched from p2p peers
enabled = true
path = "<ZINGOD_DATA>/txid_location"
# Fill empty indexes from a published snapshot before boot (aria2c on PATH)
# [snapshot]
# manifest = "https://snapshots.zingolabs.dev/zingod-snapshot-1.0.0/manifest.json"
# connections = 8 # total across every archive
First sync
On first start, each index syncs from genesis. On Mainnet or Testnet this takes a while. After a restart, each index resumes where it stopped.
Zingod serves wallets while it syncs, at the height every index has reached (heights).
Storage
Keep the indexes on local storage, preferably NVMe. Zingod reads them through mmap, so network storage performs poorly.
Every index file carries page checksums. zingod verify finds a
damaged index; delete its directory and Zingod resyncs it.
Index snapshot
A published snapshot fills empty indexes before Zingod starts, which skips most of the first
sync. It needs aria2c on PATH; the Docker image and the Nix package include it.
[snapshot]
manifest = "https://snapshots.zingolabs.dev/zingod-snapshot-1.0.0/manifest.json"
connections = 8 # total across all archives, at most 16
- Only enabled indexes whose directories are missing or empty are filled. Indexes holding data are left alone, and if none are empty nothing is downloaded.
- Free space is checked before downloading. If it is short, startup stops and names the directory and the space needed.
- Transient download errors are retried. A restart resumes each archive where it stopped.
- Startup stops on a manifest for another network, a hash mismatch, a permanent download error, or an empty index the manifest has no archive for.
While the snapshot installs, the state is provisioning, with progress in detail
(states).
The manifest format, for publishing your own snapshot:
{"network": "...", "height": 0,
"indexes": {"<index>": {"archive": "...", "sha256": "...", "archive_bytes": 0, "installed_bytes": 0}}}
All archives come from one node at one height. Each is a zstd-compressed tar of the index
directory’s contents, with no top-level directory. archive is resolved relative to the manifest
URL, and installed_bytes is its unpacked size.
Monitoring
States
Zingod reports a single state. The same value appears in /readyz, /status, the
zingod_state{state} metric and the State changed log line. The first matching row applies:
| State | Meaning | /readyz |
|---|---|---|
held | an operator ran zingod hold | 503 |
down | cannot serve: the server is stalled or overloaded, startup is waiting for a validator, or the tip is unverified and stale | 503 |
provisioning | starting up, or installing an index snapshot | 503 |
degraded | serving a recent tip that no trusted validator has confirmed (p2p peers supplied it) | 200 |
syncing | serving, but behind the best block; or no validator’s mempool is available yet | 503 |
serving | serving at the best block | 200 |
- Confirmed means a trusted validator has had the block on its best chain. One block past the last confirmation still counts, to cover the validator’s polling delay.
- Recent means the block’s timestamp is within 24 block intervals of the current time.
- A
degradednode stays in load-balancer rotation so wallets keep being served while every trusted validator is down. Alert on the state, not on/readyz. syncingnever refuses requests. During a reorg, a node briefly reportssyncingwith thetip unvouchedalarm raised.
detail adds context to the state (null until measured):
| State | detail |
|---|---|
provisioning | {phase, bytes, total_bytes, bytes_per_sec} for a snapshot install; else null |
syncing | {height, target, blocks_per_sec}: served height, best height, rate over ~60 s |
serving | {start, mbps}: gRPC egress over the last full 10-minute UTC window |
degraded, down, held | {reason, message} |
reason is one of no_validator_reachable, tip_unvouched, tip_stale, runtime_stalled,
overloaded, goaway_pending or goaway_sent.
Endpoints
| Listener | Path | Response |
|---|---|---|
| gRPC | /readyz | {"state", "detail"}, 200 or 503 |
| gRPC | /status | public status (below), always 200 |
| admin | /readyz | same as on the gRPC listener |
| admin | /livez | 503 after 30 seconds without a heartbeat from the server |
| admin | /metrics | Prometheus metrics |
The gRPC listener answers these paths over HTTP/1.1 or HTTP/2 from the moment the process starts,
before any gRPC method is available. Each connection may send 10 requests at once, refilling at
one per second; beyond that it gets 429.
/status is safe to expose publicly. It contains no validator addresses, peers, disk details or
mempool contents:
{"state": "serving", "detail": {"start": 1790000400, "mbps": 12.5},
"chain": {"network": "mainnet", "tip": {"height": 3515406, "hash": "0000…"}, "served": 3515400,
"validators": {"trusted": 2, "answering": 1}, "alarms": []},
"build": {"version": "1.0.0"},
"indexes": {"compact_block": ["GetBlock", "GetBlockRange", "GetBlockRangeNullifiers"], "utxo_set": []}}
| Field | Meaning |
|---|---|
chain.tip | the best block Zingod has verified |
chain.served | the height wallets are served at (null before startup completes) |
chain.validators | trusted validators configured, and how many are answering |
chain.alarms | one message per raised alarm |
indexes | each enabled index and the methods it serves |
Alarms
Alarms appear in /status, and are logged at WARN when raised and INFO when cleared. They do not
change what Zingod serves.
| Alarm | Raised when |
|---|---|
| stale | a validator’s tip is 24 or more blocks behind the height expected from the clock |
| end of service | a validator’s release reaches its end-of-support height within a week |
| finality paused | more than finalised_depth non-finalized blocks for 60 s, and finality not advancing |
| tip unvouched | the best block is more than one block past the last one a trusted validator confirmed |
Metrics
Every metric has HELP text. Histograms are Prometheus native histograms, available only in the
protobuf format: enable scrape_native_histograms in Prometheus or Alloy. The text format carries
every counter and gauge, but no histograms.
Disk space and disk I/O are not exported; use node exporter or cAdvisor.
Logging
zingod start logs to stdout in logfmt. RUST_LOG sets the filter;
the default is zingod=info. An invalid filter stops startup.
ts=… level=info target=zingod::telemetry::progress message="Syncing blocks" height=3503023 target=3506659 bps=10 eta=5m49s component=ZingodNFS
- Alert on WARN and ERROR. INFO is progress and lifecycle.
componentnames the subsystem:Zingod,Admin,IndexSnapshot,Traffic,ChainView,ZingodSync,ZingodNFS,GrpcSnapshot,Grpc, or an index name.State changedis logged within a second of each state change, at ERROR fordown, WARN forheldanddegraded, otherwise INFO.Not readyrepeats every 60 seconds while the state isdown,degradedorheld.- While syncing, progress lines appear every 30 seconds.
Block fetch stalledwarns when syncing stops advancing.Serving the verified tipmarks the end of the sync.
The Serving line
Once a minute, the gRPC server logs a Serving summary: rps (streams completed per second) and
conns (open connections). It is logged at WARN if, during that minute, any request failed or
was refused, took more than a second to its first response (slow), stalled (stalled), or a
connection was refused at a limit (conns_refused). These fields appear only when non-zero.
INTERNAL,UNKNOWNandDATA_LOSScount as failures.UNAVAILABLEandRESOURCE_EXHAUSTEDcount as refusals. Other codes are client errors, counted inzingod_grpc_requests_total{code}.- The first failure of each minute is also logged as
Request failed(ERROR), and the first refusal not caused by capacity asRequest unavailable(WARN).
Maintenance
Operator hold
zingod hold takes a node out of rotation; zingod resume puts it back. A hold persists across
restarts, deploys and reboots.
Use a hold before any planned stop. Stopping without one closes the gRPC listener immediately.
zingod hold # returns once connections are drained, printing "safe to stop"
systemctl stop zingod
# ... maintenance ...
systemctl start zingod
zingod resume
holdwrites a marker file,held, in[serve] state_dir. It needs write access to that directory; there is no network endpoint.- Within a second, the running node reports
heldand/readyzreturns503. - After
[serve] hold_goaway_delay_secs, every gRPC connection receives an HTTP/2 GOAWAY, and in-flight streams getmax_connection_age_grace_secsto finish. Set the delay to at least your load balancer’s failure-detection time plus its DNS TTL. holdreturns once the GOAWAYs are sent.
A held node keeps running and answering health checks and metrics. After resume, a node that
fell behind reports syncing until it catches up. A stopped node is never reported as held,
so dashboards can tell planned maintenance from a crash.
Both commands are idempotent.
Overload
A node that is persistently short of capacity reports down with reason overloaded, so load
balancers move traffic away from it. Thresholds are set under [grpc.overload] (0 disables a
cause). A cause is raised after its signal stays above the threshold for the enter time, and
cleared after it stays below half the threshold for the exit time.
| Cause | Signal, per minute | Enter / exit |
|---|---|---|
CPU pressure, IO pressure | share of time Zingod’s cgroup waited (Linux PSI) | 2 / 3 min |
refusing work | at least 10% of requests refused, at 1/s or more (exit: < 2%) | 5 / 5 min |
Override the detector on a running node, without a restart:
zingod overload off # never report overloaded
zingod overload on # back to automatic
zingod overload set <key> <value> # override a threshold, e.g. cpu_pressure_percent 40
zingod overload unset <key> # back to the config's value
Overrides are stored in state_dir and persist across restarts. Zingod applies them within a
second and logs a warning while any is in effect.
systemd
[Service]
Type=notify
ExecStart=/usr/bin/zingod start --config /etc/zingod/zingod.toml
Restart=always
TimeoutStartSec=15min
- With
Type=notify,systemctl startreturns once Zingod is serving at the tip, or held. A deploy that cannot catch up shows as a failed unit. - While startup makes progress, Zingod extends the start timeout by 5 minutes at a time, so
TimeoutStartSeconly bounds a stalled start. TimeoutStopSecmust cover[grpc] shutdown_grace_secsplus the time to flush the indexes.NOTIFY_SOCKETmust be a filesystem path, not an abstract socket.
Stopping and restarting
On SIGINT or SIGTERM, Zingod closes the gRPC listener, sends GOAWAY to every connection, waits up
to [grpc] shutdown_grace_secs for in-flight streams, flushes the indexes, and exits 0.
Zingod exits rather than serve data it cannot trust, so run it under a service manager that restarts it. It exits with code 1 when:
- a validator entry is invalid, the gRPC address cannot be bound, or an index cannot be opened;
- an index directory belongs to another network or format, or is truncated or corrupt;
- an index no longer matches the chain (
Diverged); - any internal task stops.
For a corrupt or mismatched index, delete its directory and Zingod resyncs it. An unreachable validator never stops Zingod.
Restarts are fast. Opening an index checks only its file lengths and last pages, then Zingod
refetches the recent blocks above what was flushed (Fetching non-finalized blocks in the log).
zingod verify
zingod verify -c /etc/zingod/zingod.toml checks every page checksum of every enabled index. It
only reads, so it is safe to run next to a live node.
It prints a JSON report to stdout and a summary per index to stderr; --help lists the exit
codes. Per file, it reports lost, bad_sums and bad_pages. Bytes written by a live node after
its last commit are reported as orphaned_bytes, not as corruption.
If any file is corrupt or lost, delete that index’s directory and let Zingod resync it.
Network exposure
Zingod has two listeners:
- gRPC (
[serve] grpc_listen_address) serves wallets, plus/readyzand/status, which are safe to expose publicly. - Admin (
[metrics] listen_address, off by default) serves metrics and probes, with no authentication or encryption. Metrics include validator addresses. Keep it on loopback or a private network; Zingod warns at startup if it is bound to a public address.
Connections to validators are plain HTTP JSON-RPC.
Limits on connections, streams and reads are set under [grpc]; see the
reference. When a limit is reached, the request is refused, not
queued.
TLS
[serve] tls controls TLS on the gRPC listener:
| Value | Behaviour |
|---|---|
| unset | plaintext on a loopback address; any other address refuses to start |
{ cert_path, key_path } | TLS with the given PEM files |
"off" | plaintext on any address, for use behind a TLS-terminating proxy (warns at startup) |
"self_signed" | TLS with a generated certificate, for clients that pin it |
[serve.tls]
cert_path = "/etc/zingod/fullchain.pem" # chain, leaf first
key_path = "/etc/zingod/key.pem" # PKCS#8, PKCS#1 or SEC1
An invalid pair stops startup. Zingod checks the files every minute and picks up a renewed certificate without a restart; if the new pair fails to load, it keeps serving the old one.
With "self_signed", Zingod generates an ECDSA P-256 certificate with no expiry, stores it in
<state_dir>/tls, and logs its SHA-256 fingerprint at each start. It covers localhost, the
loopback addresses and the listen address. Wallets that verify against system roots, including
zingolib, reject it, so public servers need a CA-issued certificate. To reissue it, for example
after changing the listen address, delete cert.pem.
Browser clients (gRPC-web)
[serve] grpc_web = true serves browser wallets over gRPC-web on the gRPC listener, with CORS
allowing any origin. Native gRPC is unaffected.
Enabling it makes the server usable from any website. A popular site embedding it can send load proportional to its own traffic.
zingod origin manages per-origin policy. Changes apply within a second and persist until
changed:
| Command | Effect |
|---|---|
zingod origin block <domain> | refuse requests from the origin with PERMISSION_DENIED |
zingod origin allow <domain> | serve the origin even when a parent domain is blocked |
zingod origin reset <domain> | remove the origin’s policy |
zingod origin | list the policies in effect |
- Policies apply to the registrable domain and its subdomains. The most specific policy wins, so
allow app.example.comcan overrideblock example.com. - Every origin that is not blocked meets the same limits as any other client. Clients can set any
Originheader, so an origin never lifts a limit. zingod_grpc_origin_requests_total{verdict}counts browser requests by policy. To find an abusive origin, check your proxy or edge logs.
Over HTTP/1.1, browsers open one connection per in-flight stream, each counting against
max_connections_per_ip. Serve over TLS so browsers use HTTP/2.
Stream timeouts
Streams have no total time limit. Two settings close streams that stop making progress:
unread_timeout_secs: the client stopped reading. The connection is closed.silent_timeout_secs: the server has had nothing to send. The stream ends withUNAVAILABLE.GetMempoolStreamis exempt.
Behind a proxy
Behind a proxy, every connection comes from the proxy’s address, so max_connections_per_ip would
limit all clients together. To limit per client, list the proxy in trusted_proxies and have it
send a PROXY protocol header (v1
or v2):
[serve]
tls = "off" # the proxy terminates TLS
[grpc]
trusted_proxies = ["10.0.0.5/32"]
Connections from a trusted proxy must start with the header, or they are closed. Other addresses connect without one.
nginx must use the stream module, since grpc_pass in the http module cannot send the
header:
stream {
server {
listen 443 ssl;
ssl_certificate /etc/ssl/zingod.pem;
ssl_certificate_key /etc/ssl/zingod.key;
ssl_alpn h2;
proxy_pass 127.0.0.1:8137;
proxy_protocol on;
}
}
For HAProxy, use server zingod 127.0.0.1:8137 send-proxy-v2 in a mode tcp backend.
If your proxy cannot send the header, leave trusted_proxies empty and raise
max_connections_per_ip to max_connections.
Open files
Each connection uses a file descriptor. At startup, Zingod raises its soft RLIMIT_NOFILE to the
hard limit and refuses to start if max_connections plus 1024 does not fit. Raise the hard limit
(LimitNOFILE= in systemd, --ulimit nofile= in Docker) or lower max_connections.
Docker
The image zingodevops/zingod runs zingod start as user zingod (UID and GID 1000), listens
for gRPC on 0.0.0.0:8137, and has a built-in health check. It includes aria2c for
index snapshots. To start a container, follow the
quick start.
Other subcommands use the same config and data. Run them in the running container
(docker exec zingod zingod hold) or in a one-off container with the same mounts
(zingod verify).
Paths
| Path | Contents | Mount |
|---|---|---|
/etc/zingod/zingod.toml | the config file | file, read-only |
/etc/zingod/tls/ | TLS certificates and keys, by convention | directory, read-only |
/var/lib/zingod/ | indexes, state (including the hold marker), peer cache | volume, writable by UID 1000 |
A named volume gets the right ownership automatically. A bind mount must be writable by UID 1000:
mkdir -p ./data && chown 1000:1000 ./data. Under rootless Podman, --userns=keep-id maps UID
1000 to your user.
Configuration
The image reads zingod.toml like any install, and ZINGOD_CONFIG_* environment variables
override it (configuration). The file is required, because
[[trusted_validators]] cannot be expressed in environment variables.
docker run ... -e ZINGOD_CONFIG_SYNC__CONCURRENCY=64 -e RUST_LOG=info,zingod_grpc=debug zingodevops/zingod
The image differs from a host install in one default: it listens on 0.0.0.0:8137. Because that
is not a loopback address, [serve] tls must be set (TLS).
To expose metrics, set ZINGOD_CONFIG_METRICS__LISTEN_ADDRESS=0.0.0.0:9999 and publish that port
only to your scraper.
TLS certificates
Mount the certificate directory, not the individual files. Zingod reloads renewed certificates, but a single-file bind mount keeps pointing at the old file after renewal.
docker run ... \
-v /etc/letsencrypt/live/zingod.example.com:/etc/zingod/tls:ro \
-e ZINGOD_CONFIG_SERVE__TLS__CERT_PATH=/etc/zingod/tls/fullchain.pem \
-e ZINGOD_CONFIG_SERVE__TLS__KEY_PATH=/etc/zingod/tls/privkey.pem \
zingodevops/zingod
The key must be readable by UID 1000.
Health check
The built-in check requests /readyz on 127.0.0.1:8137, over HTTPS without certificate
verification, or over plaintext when tls = "off". It passes in the serving and degraded
states, so a new node is unhealthy until it reaches the tip.
If the gRPC listener uses another port, override the check:
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:9137/readyz"]
Compose
services:
zingod:
image: zingodevops/zingod:latest
ports:
- "127.0.0.1:8137:8137"
environment:
ZINGOD_CONFIG_SERVE__TLS: "off"
volumes:
- ./zingod.toml:/etc/zingod/zingod.toml:ro # jsonrpc_address = "zebra:8232"
- zingod-data:/var/lib/zingod
depends_on:
- zebra
zebra:
image: zfnd/zebra:latest
volumes:
- zebra-data:/home/zebra/.cache/zebra
# set [rpc] listen_addr so zingod can reach it
volumes:
zingod-data:
zebra-data:
jsonrpc_address accepts a hostname, so a Compose service name works.
Building the image
docker build -t zingod .
docker build -t zingod --build-arg CARGO_PROFILE=profiling . # with symbols and frame pointers
The build produces a static musl binary on Alpine. Base images are pinned by digest.
Migrating from Zaino
Zingod 1.0 replaces Zaino 0.10. It is a new daemon, not an upgrade: it reads nothing from a Zaino install, so set it up fresh and delete Zaino’s data once Zingod is serving.
| Zaino 0.10 | Zingod 1.0 |
|---|---|
repo zingolabs/zaino, tags zainod-X.Y.Z | repo zingolabs/zingod, tags vX.Y.Z |
binary zainod | binary zingod |
image zingodevops/zainod, /app/config, /app/data | image zingodevops/zingod, /etc/zingod/zingod.toml, /var/lib/zingod |
$XDG_CONFIG_HOME/zaino/zainod.toml, ZAINO_* variables | $XDG_CONFIG_HOME/zingod/zingod.toml, ZINGOD_CONFIG_* variables |
| one LMDB database | one directory per index |
cargo features (prometheus, no_tls_*, …) | none; every feature is always built |
Config
Start from zingod generate-config; unknown keys are rejected, so a Zaino config will not load.
Carry these values over:
| Zaino 0.10 | Zingod 1.0 |
|---|---|
network = "Mainnet" | network = "mainnet" (lowercase) |
[validator_settings] validator_jsonrpc_listen_address, validator_cookie_path, validator_user, validator_password | one [[trusted_validators]] table per validator: jsonrpc_address, cookie_path, user, password |
grpc_settings.listen_address, [grpc_settings.tls] | serve.grpc_listen_address, serve.tls (TLS) |
metrics_endpoint | metrics.listen_address |
backend, zebra_db_path, ephemeral_finalised_state, json_server_settings, storage, mempool, donation_address | removed |
Data
Zingod builds its indexes on first start, or installs them from an index snapshot. Plan for the first sync.
APIs
- Zingod serves no JSON-RPC. Zaino’s JSON-RPC server and port 8237 are gone; send node RPC to the validator.
zaino.index.v1.IndexedTipService,GetBlockNullifiersandPingreturnUNIMPLEMENTED. PollGetLatestBlockinstead, and useGetBlockRangewithpoolTypes.GetBlockRangeNullifiersandGetTaddressTxidsstill work but are deprecated.- See methods and status codes for other behaviour wallets may notice.
Metrics
All zaino_* and zainod_* metrics are gone; process_* remains. Closest replacements:
| Zaino | Zingod |
|---|---|
zaino_chain_tip_height | zingod_best_tip |
zaino_sync_fetched_height | zingod_fetch_height |
zaino_sync_finalized_height | zingod_index_finalized_height{index} |
zaino_grpc_request_duration_seconds | zingod_grpc_duration_seconds{method} |
zaino_rpc_outbound_* | zingod_validator_rpc_* |
zaino_mempool_transactions | zingod_chainview_mempool_transactions{state} |
Histograms are native histograms (metrics).
Logs
Logs are logfmt on stdout, filtered by RUST_LOG (default zingod=info). ZAINOLOG_* is
ignored.
Architecture
How zingod turns a Zcash chain into the lightwalletd gRPC service, and why it is built this way.
Crate APIs live in each crate’s usage.md: what answers each method in
zingod-grpc,
storage in
zingod-persistence.
1. Who owns what
- The validator (zebrad, zakura) owns consensus: valid blocks, valid transactions, the mempool.
- The wallet owns keys.
- Zingod owns every answer derived from the chain.
Data a validator needs for consensus stays there and is not copied; everything derived is a Zingod index and never falls back to a validator. So:
- Block, tree-state and transparent-address methods come from an index or not at all. A disabled
index answers
UNIMPLEMENTEDnaming it. GetTransactiongoes to a validator: raw transaction bytes are consensus data it already stores (why). With no validator answering, the txid-location index names the transaction’s block, fetched from p2p peers and checked against the hash our indexes hold (which does not cover v5 proofs and signatures).GetTaddressTransactionsworks the same way: the index names the txids,GetTransaction’s path supplies the bytes.- Zingod does not validate transactions. That would be a second mempool verifier (proofs, nullifiers, scripts, conflicts) tracking every network upgrade. Validators decide; Zingod checks only what the bytes, the next block and its own UTXO set decide (§5).
The principle behind the rest: verify what Zingod can, trust validators for as little as possible, take everything else from as many peers as possible.
2. The map
trusted validators (1–N, configured) peers ([p2p] on; default off)
JSON-RPC or gRPC, + zebrad push streams zebra-network peer set
│ │
└──────────────┬───────────────────────────┘
▼
zingod-traffic TrafficBalancer every request: lanes, hedge, blame
│
┌───────────────────────────┼─────────────────────────────────────┐
▼ ▼ ▼
HeaderSync (chainview) ObservationMerge + PeerMempool ChainView::validate_transaction
→ HeaderChain → mempool view → broadcast_transaction
(SendTransaction)
│ watch<HeaderChainSnapshot> │
├──────────────────┐ │
▼ ▼ │
FinalizedFollower Nfs (RAM) │
(zingod-sync) tip overlay │
│ FinalizedStream │ NfsPublication
▼ ▼ ▼
index writers ──▶ zingod-grpc-snapshot Publisher ──▶ one Arc<GrpcSnapshot> per request
(disk) durable views → zingod-grpc, /status, /readyz, /metrics
Two paths fold blocks, and their windows overlap:
- Finalized path (disk): blocks at or below the finalized tip, folded and committed by each index writer. Writers never see a reorg.
- Tip path (RAM, the NFS): every block above the indexes’ durable tips, finalized blocks not yet durable included, folded in memory and served at once. A block leaves once every index has committed it. Reorgs live here alone.
Both run the same fold per index. While the NFS folds, a block above the lowest durable tip is
folded on both paths: in RAM by the NFS, and by its writer once finalized. When the NFS stands down
(first sync, an index enabled late: §7) the finalized path folds alone.
3. Sources and what each is trusted for
A trusted validator is one the operator configured ([[trusted_validators]]). A peer is
any node on the p2p network: discovered, unauthenticated, free to multiply. Nothing a peer says
counts until Zingod has checked it.
| Need | Trusted validators | Peers | Check |
|---|---|---|---|
| Headers | ✓ (getblockheader runs) | ✓ (FindHeaders), after the anchor | trusted: link + work; peer: every rule |
| Anchor (first sync) | ✓ only source | headers below our durable tip (restart) | each hash = the one our indexes hold |
| Blocks with a header | ✓ | ✓ (by hash) | hash + merkle root + coinbase height |
| Blocks below the headers | ✓ only source (by height) | — | coinbase height (body trusted) |
| Mempool | ✓ (listing + fees) | ✓ (quorum of listings) | txid recomputed; §5 |
| Mined tx by id | ✓ only source | — | txid recomputed |
| Finality | ✓ (vouch) | ✓ (> 5 netgroups, no difficulty drop) | §4 |
| Submission | ✓ check (checkadmission) | ✓ relays (zebra’s, ~⅓ of peers) | §6 |
Why peers never decide the tip. A peer count proves nothing an attacker cannot buy: anyone can run ninety. Proof of work costs real mining however many peers present it. So peers are safe sources of headers and of anything checkable against them, and weak sources of claims. Two claims are taken from peers anyway, so Zingod keeps serving with every trusted validator down: finality and mempool admission. Both count only outbound peers (Zingod chose them) spread over more than 5 netgroups (/16 IPv4, /32 IPv6; one operator ≈ one netgroup).
4. The best chain
The Validated Header tree (zingod-header-chain, memory only) holds every valid branch above the
finalized tip; the best tip is the leaf with the most cumulative work. HeaderSync
(zingod-chainview) is its only writer and publishes an immutable HeaderChainSnapshot on a watch that
the NFS, the finalized path and the GrpcSnapshot publisher read.
finalized tip best tip (most work)
───●──●──●──●──●──●──●──●──●──●──● A ◀─
└──●──●──● B kept: may still win; overtaking A = a reorg
- Anchor. No history from genesis. With no chain, or a trusted validator so far ahead its anchor
lies above our fetch ceiling, the tree anchors at that validator’s claim − max(depth, 113) (113 =
the difficulty rule’s context). With no trusted validator answering (a restart mid-outage,
[p2p]on), the indexes’ own chain anchors instead: the 113 blocks below their durable tip, each peer header matched to the hash the indexes hold. A first sync has no such chain: only a trusted validator anchors it. - Trusted headers arrive in runs and enter on their parent link and nBits work alone: the validator already ran the rules.
- Peer headers (
[p2p]on, once anchored) run every consensus rule first (zingod_header_chain::validate, the rules). A header from the future is deferred, not blamed; one failing a rule gets its sender blamed. - Bounds. Side branches are capped in tips and nodes (past either, the lowest-work side leaf
goes), and only a window of finalized headers is kept
(limits).
depth=[sync] finalised_depth(default and floor).
Finality
A header proves work, not that its block’s transactions are valid. A finalized block is never reorged (the indexes hold it on disk), so finality is where Zingod stops being able to change its mind.
Every header node records its witnesses: who once had that block on its best chain, and so every ancestor too, for good. Each block carries all of its own accounting; nothing about it is kept anywhere else.
- Vouched. A trusted validator witnessed it (zebra commits only valid blocks). Vouches come
from the last header of each trusted run, each validator’s claimed tip, and its
getblockhashanswer at our best. Peer headers never vouch. - Peer-quorate (
[p2p]on). Live outbound peers in > 5 netgroups witnessed it by announcing it or a descendant. - finalized = min(highest settled, best − depth), where settled = vouched, or peer-quorate while no unvouched best-chain block has under half the work of the last vouched block. An attacker who eclipses us can feed a private chain, but its difficulty retargets to their hashrate, which shows up as that collapse. Both peer checks fail safe: a thin peer set or a real hashrate drop only delays finality.
- Neither holding: finality waits, the
finality pausedalarm rises (§11), serving continues from RAM. A trusted validator’s return vouches and finality resumes.
A reorg, step by step
Peers are first to every reorg: their headers enter on the rules alone, while a trusted validator
downloads and fully verifies the new branch before it switches, and Zingod learns of that on its
next poll (§9). Meanwhile the served tip drops to the fork point (§7) and synced closes (§8).
network ─── N ──┬── N+1 branch A, Zingod + the validator on it
└── N+1' ── N+2 branch B, more work, wins
──────────────────────────────────────────────────────────────────────────────────▶ time
t0 │ L2: validator verifies B │ L3 │ L4: fetch N+1', N+2 │
│ │ │ │ │
peers ─────────────┤ announce B headers │
│ │
best ─── N+1 ──────┤ N+2 (every header rule: PoW, difficulty, MTP, Equihash) │
│ │
served ── N+1 ─────┤ N ◀── fork point, vouched: what wallets get ───────────────────▶ N+2
│ │
validator ── N+1 ──┼────────────────────────────▶ N+2 │
│ │ │ │
vouched ─── N+1 ───┤ N ◀── last shared block ─────┼────▶ N+2 (poll, or push hint) │
| latency | what | bound |
|---|---|---|
| L1 | peers announce, headers validated, best switches | milliseconds |
| L2 | the validator downloads and verifies the branch, switches | its verification time |
| L3 | Zingod sees the validator’s new tip (the vouch) | one poll; ~0 with the push stream (§9) |
| L4 | Zingod fetches the branch’s blocks by hash and folds them | block fetch |
Through the window Zingod serves the fork point, a vouched block behind an unvouched best: a
currency gap, not a trust gap, and the state rule reads it so
(states). The tip unvouched alarm (§11) carries the
unvouched best. L2 is the validator’s job and no change on Zingod’s side removes it.
5. The mempool view
Every transaction the view knows is a sighting: its bytes (fetched once, txid recomputed), its
fee, a bitset of trusted validators listing it, a quorum flag and an ours flag.
Servable = listed by a trusted validator (not Down or catching up), admitted by a peer
quorum, or ours (accepted from our own submission). One trusted listing is enough: each trusted
validator is trusted.
Peer quorum ([p2p] on). Each live outbound peer is polled for its whole mempool, and each
listing counts while fresh (intervals). A
transaction is admitted when:
- more than 2/3 of the counted listings hold its exact id (txid + auth digest), from > 5 netgroups;
- its bytes decode, its expiry and (v5+) branch id fit the block above the verified best, it is not coinbase;
- every transparent prevout resolves in the served UTXO set or a held transaction’s outputs, and the resulting fee is not negative.
It stays while more than 1/3 still list it. No fee floor (zebrad’s and zakura’s differ), no proofs,
signatures or scripts: each lister is a full node that ran those. A consensus refusal is remembered
(REFUSED_TTL); an unknown prevout is retried next round. With honest peers two conflicting
transactions are never both admitted (one node’s mempool holds at most one, and > 2/3 + > 1/3
must overlap); a peer listing both can get both admitted and kept (mempool_model_tests.rs).
Cost to an attacker: > 2/3 of our outbound peers across 6 netgroups, for a served transaction that
never mines.
Transaction inv announcements are not used: nodes announce each transaction to a third of their
peers, once, so silence means nothing.
- Leaving. A transaction goes when nothing lists it and no quorum keeps it. A new tip clears
nothing; an unlisted
oursgoes at the next move of the best block. - Gate.
GetMempoolTx/GetMempoolStreamanswer only while aLivetrusted validator holds the verified best, or the polled peers span > 5 netgroups. OtherwiseUNAVAILABLE, never an unverified mempool.
6. Submission
SendTransaction asks one trusted validator for a verdict without a broadcast, answers the
wallet from it, then relays through the peers and watches the transaction spread. Relaying
every wallet transaction through the same validators would tell any observer which transactions
came through Zingod, and let one validator censor every wallet behind it. The relay follows
Dandelion++’s originator with its fail-safe timer.
precheck (nothing contacted): decodes (-22); expiry ≥ next block, v5+ branch = next block's (-25)
check: checkadmission on a random Live trusted validator holding the verified best
refused → wallet gets that refusal at once, nothing relayed
admissible → wallet answered Ok(txid) at once; transaction becomes `ours` (servable)
unsupported / unanswered → next holder; none → unchecked (below)
attempt: peers — zebra's own relay, PushTransaction to ⌈ready/3⌉ random ready peers
(no peers: a random untried trusted validator, sendrawtransaction)
unseen past propagation_threshold → next attempt, up to max_attempts
then, still unspread: one trusted sendrawtransaction (the fail-safe)
unchecked: Ok at the first acceptance (trusted Ok | any trusted listing | peer quorum);
never accepted → that trusted push is the verdict; else UNAVAILABLE
checkadmission(Zingod’s zebra fork) runssendrawtransaction’s checks, the fee policy and mempool conflicts included, without admitting or relaying. It goes to a holder of the verified best only (a validator behind it refuses what spends a fresh output). A validator without it (zcashd, stock zebrad, zakurad) answers-32601: never a failure, the next holder is asked. A refusal’s wire shape:SendTransaction.- A peer relay gets no reply and zebra names no recipient. Spread = a trusted validator never pushed to lists it, or a peer quorum admits it (past the ~⅓ one relay reaches). The job ends at spread, when nothing outside its trusted entries can be watched, or when attempts run out.
- Relays ride the peer set’s existing connections: a zebrad peer drops a second connection from
our IP, so a fresh connection per push would never deliver. With
[p2p]off every attempt is a trustedsendrawtransaction. [submission]setspropagation_threshold_secsandmax_attempts(SubmitPolicy).
7. Indexing
Finalized path (zingod-sync)
FinalizedFollower streams lowest durable tip + 1 ..= finalized tip into one FinalizedStream<Block>:
- Heights the header chain has a header for: fetched by hash from any member (peers included), checked against the header (hash, merkle root, coinbase height).
- Below the header chain (bulk sync, a restart from an old index): by height, trusted only, batched, checked by coinbase height only. A trusted validator’s body is trusted.
- Each block must link to every durable tip it extends and to the block sent before it; otherwise the follower stops and the process exits. Nothing is skipped.
- Queues are byte-bounded and a send reserves room in every queue first, so the slowest writer paces the fetch and memory stays bounded. A dropped queue or failed commit panics.
- Utxo-set resolves each transaction’s fee and feeds compact-block through a
FeeSink. - A writer commits when its buffer fills, after a run ending at the finalized tip, and at shutdown.
Tip path (zingod-nfs)
- Root = the lowest durable tip across indexes.
- The NFS folds only while the header just above the root is held and best is within 2 · depth of the root. Otherwise (first sync, an index enabled late) the finalized path does the work and the root is served.
- Each best-chain block above the root is fetched by hash, checked against its header, and folded over the indexes’ durable views (utxo-set first: its fees feed compact-block).
- Served tip = the deepest folded best-chain node, else the root. A reorg moves it to the fork point at once, then forward as the new branch folds. Side branches stay until the header chain drops them, so switching back costs nothing.
- A node goes once every index has committed its height.
8. Serving: one GrpcSnapshot
zingod-grpc-snapshot’s publisher combines the NFS’s latest NfsPublication and the chain view’s latest
publish into one Arc<GrpcSnapshot>. Every gRPC request or stream, probe, scrape and state log loads
one and reads only it, so no answer mixes two moments.
- No height named → the served tip, which every enabled index holds. A height named → up to that index’s own durable tip.
syncedopens when the served tip is the best block and no answering trusted validator claims a higher tip (ChainViewSnapshot::claimed_past_best: one already shown off our chain does not count, one not polled since best moved does), and stays while it is on the best chain withindepthof it. At boot, header sync anchorsdepthbelow a validator’s claim, so it stays shut until the header chain reaches that claim and the NFS has fetched the window.- The daemon’s state (
serving,degraded,down, …) is judged from the same load: states.
GetMempoolStream
The protocol says “close the stream when a new block is mined”; Zingod makes that exact. A stream
is bound to the served tip’s hash: it opens with the servable mempool, sends each new transaction
once (every message height: 0, never one mined in a block the served tip extends), and ends OK
once the served tip moves, never on an empty mempool. It ends only after the GrpcSnapshot
already serves the new tip, so the GetLatestBlock a wallet sends next sees it. One append-only
log per served tip is shared by every subscriber, so per-transaction work never scales with
subscribers.
9. Talking to validators and peers
Every request goes through one TrafficBalancer (zingod-traffic); lanes, intervals, hedge
delays and bench times are in its guide.
- Members. Each trusted validator, in
prioritytiers (0 first); then the whole peer set as one member below every tier (zebra picks the peer). Peers answer headers and blocks only. - Failure. A round asks each member once, best tier first; slow answers are hedged to another member. Blocks are retried until answered; everything else ends unanswered after one round.
- Blame. The balancer cannot judge a value; the caller checks it and
reports a bad one, which benches that member. Absent is never blame. The peer set is never benched: zebra scores the lying peer instead. - Polling. Each trusted validator is polled on its own permit, so no request delays tip
following, and sooner on a zebrad push event. Consecutive failures make it
Down, probed until it answers. A failing validator never stops the process. - Load on a zebrad =
max_connections. There is no request-rate or byte-rate limit.
10. When things fail
| Down | Chain + indexes | Mempool, SendTransaction | Finality |
|---|---|---|---|
| one trusted validator | ✓ | ✓ | ✓ |
every trusted, [p2p] on | ✓ near the tip (peers); bulk sync below the headers stalls | ✓ quorum; GetTransaction from the peers’ block | on peers’ word, else paused |
every trusted, [p2p] off | tip stops | UNAVAILABLE | paused |
every trusted, restart, [p2p] on | ✓ anchored on the indexes’ own chain, then as above | as above | as above |
| every trusted, first sync | nothing (only a trusted validator anchors) | UNAVAILABLE | — |
| every peer | ✓ (trusted) | ✓ | ✓ |
11. Telemetry
Observation only: none of it changes a tip or what is served. State is read from one GrpcSnapshot at
scrape (zingod_* and zingod_chainview_* gauges, /status); events are emitted where they
happen. Alarms log once WARN when raised, once INFO when cleared
(the alarms; stale = STALE_TIP_BLOCKS).
12. Seams
A trait exists only at a seam: a boundary with a second implementation (a real one or a test double), or one that keeps a dependency out of the crates above it.
zingod_source::ChainDataSource: JSON-RPC or indexer gRPC; tests drive the real client against aMockValidator.zingod_chainview::PeerSource,zingod_traffic::PeerTransport: zingod’sZebraPeers, which keeps zebra-network out of both crates.zingod_chainview::UnspentOutputs: the chain view sits below the indexes.zingod_persistence::PersistenceEngineandfs::Fs: indexes see the port, never the engine;SimFstests crashes.
Not seams: the gRPC service, each index, and the validator set (TrafficBalancer is not a
ChainDataSource, so a pool is never mistaken for one validator).
What the clients actually call
What the two wallet sync engines ask for, how often, and where answering slightly wrong is worse than not answering. Read at the source; references name a function or file, not a line.
- librustzcash:
zcash_client_backend/src/sync.rs, the only RPC caller in that workspace. It backs Zashi and zallet. - pepper-sync:
zingolib/pepper-sync, all traffic throughclient/fetch.rs. It backs zingolib and Zingo mobile.
Where Zingod answers each method from: zingod-grpc; the wallet-facing contract: the book.
The call matrix
| RPC | librustzcash | pepper-sync |
|---|---|---|
GetLatestBlock | every loop iteration | once per session |
GetBlockRange | per batch, dominant | per scan range, dominant |
GetBlock | not called | reorg check and backfill |
GetTreeState | 1:1 with every GetBlockRange, at scan_range.start - 1 (download_chain_state) | birthday and fallbacks |
GetSubtreeRoots | three pools at startup, ironwood included | three pools per session, two passes each |
GetTransaction | not called | per scan target |
GetTaddressTransactions | not called | not called (it uses GetTaddressTxids) |
GetTaddressTxids (deprecated alias) | not called | per known t-address, and gap-limit discovery |
GetBlockRangeNullifiers (deprecated alias) | not called | per ScannedWithoutMapping range |
GetAddressUtxosStream | per account per iteration | dead code |
GetMempoolStream | not called | background task, retries forever |
SendTransaction | the embedder’s job | zingolib |
GetLightdInfo | never called | zingolib server selection only |
GetTreeState is 1:1 with GetBlockRange, the highest-volume method after block fetch itself:
a librustzcash wallet cannot complete a single sync batch without it.
Two deprecated methods, served until pepper-sync moves off them
GetBlockRangeNullifiers and GetTaddressTxids are deprecated aliases, marked TODO: REMOVE
at every site. pepper-sync calls both on every sync and treats any non-OK status as a fatal
SyncError, so without them no pepper-sync wallet syncs against Zingod. They go when pepper-sync
calls the replacements.
GetBlockRangeNullifiersis superseded byGetBlockRangewith shieldedpoolTypes, a superset. Its shape is not apoolTypessubset (it dropschainMetadataand sapling outputs, and keeps only each action’s nullifier), so Zingod re-projects the stored compact blocks rather than keep a second projection that could disagree.GetTaddressTxidsis misnamed and deprecated by the proto itself (“please use GetTaddressTransactions instead”).
Correctness landmines
Ironwood tree state must be served correctly or wallets silently mis-sync. Both clients map
an empty TreeState.{orchard,ironwood}_tree to an empty commitment tree with no error
(TreeState::orchard_tree / ironwood_tree in zcash_client_backend/src/proto.rs;
pepper-sync/src/witness.rs), indistinguishable from a pre-activation height; GetSubtreeRoots
is swallowed into Vec::new() the same way. So Zingod serves each pool from its activation on as
the real tree (000000 when empty), and "" only below activation
(tree-state guide).
The wire format is a CommitmentTree, not a frontier. TreeState.*_tree is the hex of
zcash_primitives::merkle_tree::write_commitment_tree; both clients parse it with
read_commitment_tree, then .to_frontier().
Neither client sends a block hash. Both pass hash: vec![] to GetTreeState.
GetSubtreeRoots costs one extra empty round trip. pepper-sync re-requests until a pass
returns zero roots, so a session makes six calls, not three.
GetBlockRange has no client-side size limit. pepper-sync requests a whole scan range in
one call, and scan ranges follow subtree boundaries (2^16 notes, not blocks): a mainnet wallet
sync asked for 158,853 blocks in one call. Any server-side maximum breaks zingolib for some
birthday, and because it retries a stream error unboundedly, it presents as a hang. So Zingod has
no range cap, and never answers short: a wallet given fewer blocks than it asked for reads that
as the end of the chain.
Do not implement GetAddressUtxos paging naively. startHeight is not a unique resume key
(lightwallet-protocol issue #33), so paging can stall. No current client pages.
pepper-sync refuses a server below protocol v0.5.0. It rejects any
lightwalletProtocolVersion below v0.5.0, an empty one included
(check_lightwallet_protocol_version in pepper-sync/src/client.rs). Zingod reports its vendored
protos’ release (zingod_proto::LIGHTWALLET_PROTOCOL_VERSION), so a new upstream release moves
the reported version with it.
Transparent: address queries and the private path
pepper-sync discovers transparent funds by sending each derived address to the server, one
per round trip, until gap_limit consecutive addresses come back empty (address_discovery in
pepper-sync/src/sync/transparent.rs), and re-pays it every session. The server learns every
address a wallet derives, which are used, and the order that links them to one wallet.
The private alternative: request the TRANSPARENT pool in BlockRange.poolTypes and match
scripts locally (lightwallet-protocol v0.4.0). Zingod serves it. CompactTxIn carries no prevout
value, so compact blocks alone cannot yield a transparent balance: the address RPCs stay even
once clients move to the private path. As lightwalletd does, a coinbase’s vin is omitted (a
client detects it by CompactTx.index == 0).
Never silently prune a requested
TRANSPARENT. If Zingod cannot serve it, return an error. Silent pruning costs a user their privacy with nothing in the logs.
The contract is the proto, not a ZIP
ZIP-307 covers Sapling only and ZIP-314 is Reserved with an empty body, so the .proto
comments and CHANGELOG in zcash/lightwallet-protocol are the normative source, and
LightdInfo.lightwalletProtocolVersion is how a server states what it supports.
Testing
Zingod has two kinds of test. The unit and model tests in packages/* run on the host with
cargo nextest run --workspace and need no validator, although they are not network-free
(zingod-grpc, for example, binds a loopback socket). Their blocks, headers and validators come
from one builder, MockChain. The live
tests in live-tests/ run a deployed zingod against a real zebrad on Kubernetes, driven by
ztest.
| Set | What it covers |
|---|---|
live-tests/clientless | zingod’s gRPC answers checked against the validator as oracle, with no wallet |
live-tests/e2e | a real wallet syncing and sending through zingod’s gRPC to a live validator |
live-tests/non-finalized-state | reorgs forced on a regtest zebrad, checked across every index (README) |
live-tests/sync | the long-running mainnet profiles zingod_index_construction and zingod_index_crash_consistency |
live-tests/ is a standalone Cargo workspace, and live-tests/sync/ a second one nested inside
it, each with its own lock. The rules for writing live tests: live-tests/CLAUDE.md.
Model tests in bulk
Every model suite runs at its in-code case count under cargo nextest run --workspace. To run
them harder, use the models nextest profile (.config/nextest.toml), which selects only the
model suites and lifts the default profile’s 60 s per-test kill, and set PROPTEST_CASES:
PROPTEST_CASES=5000 cargo nextest run --profile models
PROPTEST_MAX_SHRINK_ITERS shrinks a failure harder and PROPTEST_VERBOSE=1 prints each case.
Raising PROPTEST_CASES across the whole workspace also scales
transactions_decode_to_their_source (zingod-source), which is slow.
A new model suite joins the profile by its name: its test path contains model or
random_histories. Its oracle is written independently of the code under test, and moves the
model cannot apply are clamped rather than filtered out, so shrinking stays effective. A failing
case is saved and replays on every run: beside an integration test as
<test>.proptest-regressions, or under the crate’s proptest-regressions/. Commit it with the fix.
Heavy run
A change to zingod-persistence, zingod-header-chain, zingod-nfs, zingod-grpc-snapshot or
zingod-traffic is done only once this passes for that crate. Each round runs the crate’s whole
suite on fresh seeds, so the timing-sensitive tests (background merges, failed I/O, crash
states) repeat alongside the models. Then the crates built on it run once:
crate=zingod-persistence
end=$((SECONDS + 180)); round=0
while [ $SECONDS -lt $end ]; do
round=$((round + 1))
PROPTEST_CASES=1000 cargo nextest run -p "$crate" --all-features --profile heavy \
|| { echo "$crate: FAILED in round $round"; break; }
echo "$crate: round $round passed"
done
cargo nextest run --workspace -E "rdeps($crate)" # after the loop passes
--profile heavylifts the default profile’s 60 s kill (at 1000 cases a model test outlives it): a test still running at 300 s is reported slow, never killed.--all-featuresbuilds the crate with every optional feature on.- Lengthen the run by raising
180, never shorten it. Keep the test profile (no--release): overflow checks stay on. - A failure saves its shrunk case (above). Fix the bug and rerun the whole block.
Commands
cargo nextest run --workspace # from the repo root
cd live-tests
ztest run # clientless, e2e and non-finalized-state
ztest run -p clientless # one partition
ztest run --rerun latest -p clientless # the last run's failures
ztest run --no-cleanup -p clientless # keep each test's namespace (pods, logs) to inspect
cd sync # now in live-tests/sync
ztest sync start zingod_index_construction --watch
ztest sync start zingod_index_crash_consistency
ztest run MUST be launched from live-tests/ and ztest sync from live-tests/sync/: neither
finds its tests from any other directory. ztest run takes cargo nextest run arguments, but
its own flags such as --rerun and --cluster MUST come before them.
The ztest CLI
Both live workspaces depend on the published ztest crate (ztest = "0.2.0" in each
Cargo.toml). Install the matching CLI from crates.io:
cargo install --locked ztest_cli
The sync profiles build zebrad’s image from a zebra checkout beside this repository.
Cluster setup
Any reachable Kubernetes cluster works. For a local kind cluster:
kind create cluster --name ztest
ztest cluster add kind --kind ztest --set-default # profile "kind" on kind cluster "ztest"
ztest cluster setup # provisions ztest's resources; takes a few minutes
ztest cluster check # read-only readiness report
The sync profiles restore validator chain-state snapshots of 40 to 200 GB. A CSI driver with
copy-on-write VolumeSnapshot support makes that restore fast (without one the snapshot is
copied); the other live tests need neither. Pick the driver when you add the cluster profile:
ztest cluster add kind --kind ztest --storage-driver topolvm.io --set-default # LVM thin pools
ztest cluster add kind --kind ztest --storage-driver rook-ceph.cephfs.csi.ceph.com --set-default # shared cluster
Updating Zebra
Zingod’s validator path links no Zebra crate. zingod-source fetches each block as consensus
bytes (JSON-RPC getblock <hash> 0, or the indexer gRPC), decodes it with librustzcash, and
parses every other JSON-RPC result itself (packages/zingod-source/src/json.rs). The one Zebra
link is zingod-peers, which embeds zebra-network (and zebra-chain) as Zingod’s p2p layer,
pinned by git rev in the root Cargo.toml.
So keeping up with Zebra means testing against each new zebrad release, plus moving the
zebra-* rev when the p2p layer needs it. A spread between the latest zebrad and the one the
live suite tests is high-priority tech debt. When checking a JSON-RPC response’s shape, read the
tagged Zebra source for the node version you target, not the crates.io crates.
Moving to a new zebrad
- Take a baseline on
devwithcargo nextest run --workspaceand the live suite. - Bump the version in every
Validator::zebrad("…")underlive-tests/, and the zebrad version stated in the requirements. - Check out the matching tag in the
zebracheckout beside this repository: the sync profiles build zebrad’s image from it. - Run the live tests (testing.md). A changed response shape shows up as a parse
error in
zingod-source, which is where it is fixed.
Pinning unreleased dependencies
librustzcash’s crates (zcash_primitives, zcash_protocol, zcash_transparent,
zcash_address, zcash_encoding, equihash) and the crates under them (orchard,
sapling-crypto, incrementalmerkletree) track upstream closely. An unpublished change is
pinned through the root [patch.crates-io] (Cargo honours it only in the workspace root), by
git rev, or by path to a sibling checkout while a change spans several repositories. Patching
one of these crates usually forces patching the others, so every crate agrees on one version of
the types that cross between them; the zebra-* dependencies point at a zebra fork built on the
same forks for the same reason. Comment each patch with why it is not a published version and
when to drop it.