---

order: 3 title: Requirements for the Application

Requirements for the Application

• Requirements for the Application
  • Formal Requirements
    • Consensus Connection Requirements
    • Mempool Connection Requirements
  • Managing the Application state and related topics
    • Connection State
      • Concurrency
      • FinalizeBlock
      • Commit
      • Candidate States
    • States and ABCI Connections
      • Consensus Connection
      • Mempool Connection
        • Replay Protection
      • Info/Query Connection
      • Snapshot Connection
    • Transaction Results
      • Gas
      • Specifics of CheckTxResponse
      • Specifics of ExecTxResult
    • Updating the Validator Set
    • Consensus Parameters
      • List of Parameters
        • BlockParams.MaxBytes
        • BlockParams.MaxGas
        • EvidenceParams.MaxAgeDuration
        • EvidenceParams.MaxAgeNumBlocks
        • EvidenceParams.MaxBytes
        • FeatureParams.PbtsEnableHeight
        • FeatureParams.VoteExtensionsEnableHeight
        • ValidatorParams.PubKeyTypes
        • VersionParams.App
        • SynchronyParams.Precision
        • SynchronyParams.MessageDelay
      • Updating Consensus Parameters
        • InitChain
        • FinalizeBlock, PrepareProposal/ProcessProposal
    • Query
      • Query Proofs
      • Peer Filtering
      • Paths
    • Crash Recovery
    • State Sync
      • Taking Snapshots
      • Bootstrapping a Node
        • Snapshot Discovery
        • Snapshot Restoration
        • Snapshot Verification
        • Transition to Consensus
  • Application configuration required to switch to ABCI 2.0

Formal Requirements

Consensus Connection Requirements

This section specifies what CometBFT expects from the Application. It is structured as a set of formal requirements that can be used for testing and verification of the Application's logic.

Let p and q be two correct processes. Let r_p (resp. r_q) be a round of height h where p (resp. q) is the proposer. Let s_p,h-1 be p's Application's state committed for height h-1. Let v_p (resp. v_q) be the block that p's (resp. q's) CometBFT passes on to the Application via PrepareProposalRequest as proposer of round r_p (resp r_q), height h, also known as the raw proposal. Let u_p (resp. u_q) the possibly modified block p's (resp. q's) Application returns via PrepareProposalResponse to CometBFT, also known as the prepared proposal.

Process p's prepared proposal can differ in two different rounds where p is the proposer.

• Requirement 1 [PrepareProposal, timeliness]: If p's Application fully executes prepared blocks in PrepareProposal and the network is in a synchronous period while processes p and q are in r_p, then the value of TimeoutPropose at q must be such that q's propose timer does not time out (which would result in q prevoting nil in r_p).

Full execution of blocks at PrepareProposal time stands on CometBFT's critical path. Thus, Requirement 1 ensures the Application or operator will set a value for TimeoutPropose such that the time it takes to fully execute blocks in PrepareProposal does not interfere with CometBFT's propose timer. Note that the violation of Requirement 1 may lead to further rounds, but will not compromise liveness because even though TimeoutPropose is used as the initial value for proposal timeouts, CometBFT will be dynamically adjust these timeouts such that they will eventually be enough for completing PrepareProposal.

• Requirement 2 [PrepareProposal, tx-size]: When p's Application calls PrepareProposal, the total size in bytes of the transactions returned does not exceed PrepareProposalRequest.max_tx_bytes.

Busy blockchains might seek to gain full visibility into transactions in CometBFT's mempool, rather than having visibility only on a subset of those transactions that fit in a block. The application can do so by setting ConsensusParams.Block.MaxBytes to -1. This instructs CometBFT (a) to enforce the maximum possible value for MaxBytes (100 MB) at CometBFT level, and (b) to provide all transactions in the mempool when calling PrepareProposal. Under these settings, the aggregated size of all transactions may exceed PrepareProposalRequest.max_tx_bytes. Hence, Requirement 2 ensures that the size in bytes of the transaction list returned by the application will never cause the resulting block to go beyond its byte size limit.

• Requirement 3 [PrepareProposal, ProcessProposal, coherence]: For any two correct processes p and q, if q's CometBFT calls ProcessProposal on u_p, q's Application returns Accept in ProcessProposalResponse.

Requirement 3 makes sure that blocks proposed by correct processes always pass the correct receiving process's ProcessProposal check. On the other hand, if there is a deterministic bug in PrepareProposal or ProcessProposal (or in both), strictly speaking, this makes all processes that hit the bug byzantine. This is a problem in practice, as very often validators are running the Application from the same codebase, so potentially all would likely hit the bug at the same time. This would result in most (or all) processes prevoting nil, with the serious consequences on CometBFT's liveness that this entails. Due to its criticality, Requirement 3 is a target for extensive testing and automated verification.

• Requirement 4 [ProcessProposal, determinism-1]: ProcessProposal is a (deterministic) function of the current state and the block that is about to be applied. In other words, for any correct process p, and any arbitrary block u, if p's CometBFT calls ProcessProposal on u at height h, then p's Application's acceptance or rejection exclusively depends on u and s_p,h-1.

• Requirement 5 [ProcessProposal, determinism-2]: For any two correct processes p and q, and any arbitrary block u, if p's (resp. q's) CometBFT calls ProcessProposal on u at height h, then p's Application accepts u if and only if q's Application accepts u. Note that this requirement follows from Requirement 4 and the Agreement property of consensus.

Requirements 4 and 5 ensure that all correct processes will react in the same way to a proposed block, even if the proposer is Byzantine. However, ProcessProposal may contain a bug that renders the acceptance or rejection of the block non-deterministic, and therefore prevents processes hitting the bug from fulfilling Requirements 4 or 5 (effectively making those processes Byzantine). In such a scenario, CometBFT's liveness cannot be guaranteed. Again, this is a problem in practice if most validators are running the same software, as they are likely to hit the bug at the same point. There is currently no clear solution to help with this situation, so the Application designers/implementers must proceed very carefully with the logic/implementation of ProcessProposal. As a general rule ProcessProposal SHOULD always accept the block.

According to the Tendermint consensus algorithm, currently adopted in CometBFT, a correct process can broadcast at most one precommit message in round r, height h. Since, as stated in the Methods section, ExtendVote is only called when the consensus algorithm is about to broadcast a non-nil precommit message, a correct process can only produce one vote extension in round r, height h. Let e^r_p be the vote extension that the Application of a correct process p returns via ExtendVoteResponse in round r, height h. Let w^r_p be the proposed block that p's CometBFT passes to the Application via ExtendVoteRequest in round r, height h.

• Requirement 6 [ExtendVote, VerifyVoteExtension, coherence]: For any two different correct processes p and q, if q receives e^r_p from p in height h, q's Application returns Accept in VerifyVoteExtensionResponse.

Requirement 6 constrains the creation and handling of vote extensions in a similar way as Requirement 3 constrains the creation and handling of proposed blocks. Requirement 6 ensures that extensions created by correct processes always pass the VerifyVoteExtension checks performed by correct processes receiving those extensions. However, if there is a (deterministic) bug in ExtendVote or VerifyVoteExtension (or in both), we will face the same liveness issues as described for Requirement 5, as Precommit messages with invalid vote extensions will be discarded.

• Requirement 7 [VerifyVoteExtension, determinism-1]: VerifyVoteExtension is a (deterministic) function of the current state, the vote extension received, and the prepared proposal that the extension refers to. In other words, for any correct process p, and any arbitrary vote extension e, and any arbitrary block w, if p's (resp. q's) CometBFT calls VerifyVoteExtension on e and w at height h, then p's Application's acceptance or rejection exclusively depends on e, w and s_p,h-1.

• Requirement 8 [VerifyVoteExtension, determinism-2]: For any two correct processes p and q, and any arbitrary vote extension e, and any arbitrary block w, if p's (resp. q's) CometBFT calls VerifyVoteExtension on e and w at height h, then p's Application accepts e if and only if q's Application accepts e. Note that this requirement follows from Requirement 7 and the Agreement property of consensus.

Requirements 7 and 8 ensure that the validation of vote extensions will be deterministic at all correct processes. Requirements 7 and 8 protect against arbitrary vote extension data from Byzantine processes, in a similar way as Requirements 4 and 5 protect against arbitrary proposed blocks. Requirements 7 and 8 can be violated by a bug inducing non-determinism in VerifyVoteExtension. In this case liveness can be compromised. Extra care should be put in the implementation of ExtendVote and VerifyVoteExtension. As a general rule, VerifyVoteExtension SHOULD always accept the vote extension.

• Requirement 9 [all, no-side-effects]: p's calls to PrepareProposal, ProcessProposal, ExtendVote, and VerifyVoteExtension at height h do not modify s_p,h-1.

• Requirement 10 [ExtendVote, FinalizeBlock, non-dependency]: for any correct process p, and any vote extension e that p received at height h, the computation of s_p,h does not depend on e.

The call to correct process p's FinalizeBlock at height h, with block v_p,h passed as parameter, creates state s_p,h. Additionally, p's FinalizeBlock creates a set of transaction results T_p,h.

• Requirement 11 [FinalizeBlock, determinism-1]: For any correct process p, s_p,h exclusively depends on s_p,h-1 and v_p,h.

• Requirement 12 [FinalizeBlock, determinism-2]: For any correct process p, the contents of T_p,h exclusively depend on s_p,h-1 and v_p,h.

Note that Requirements 11 and 12, combined with the Agreement property of consensus ensure state machine replication, i.e., the Application state evolves consistently at all correct processes.

Also, notice that neither PrepareProposal nor ExtendVote have determinism-related requirements associated. Indeed, PrepareProposal is not required to be deterministic:

• u_p may depend on v_p and s_p,h-1, but may also depend on other values or operations.
• v_p = v_q ⇏ u_p = u_q.

Likewise, ExtendVote can also be non-deterministic:

• e^r_p may depend on w^r_p and s_p,h-1, but may also depend on other values or operations.
• w^r_p = w^r_q ⇏ e^r_p = e^r_q

Mempool Connection Requirements

Let CheckTxCodes_tx,p,h denote the set of result codes returned by p's Application, via CheckTxResponse, to successive calls to CheckTx occurring while the Application is at height h and having transaction tx as parameter. CheckTxCodes_tx,p,h is a set since p's Application may return different result codes during height h. If CheckTxCodes_tx,p,h is a singleton set, i.e. the Application always returned the same result code in CheckTxResponse while at height h, we define CheckTxCode_tx,p,h as the singleton value of CheckTxCodes_tx,p,h. If CheckTxCodes_tx,p,h is not a singleton set, CheckTxCode_tx,p,h is undefined. Let predicate OK(CheckTxCode_tx,p,h) denote whether CheckTxCode_tx,p,h is SUCCESS.

• Requirement 13 [CheckTx, eventual non-oscillation]: For any transaction tx, there exists a boolean value b, and a height h_stable such that, for any correct process p, CheckTxCode_tx,p,h is defined, and OK(CheckTxCode_tx,p,h) = b for any height h ≥ h_stable.

Requirement 13 ensures that a transaction will eventually stop oscillating between CheckTx success and failure if it stays in p's mempool for long enough. This condition on the Application's behavior allows the mempool to ensure that a transaction will leave the mempool of all full nodes, either because it is expunged everywhere due to failing CheckTx calls, or because it stays valid long enough to be gossipped, proposed and decided. Although Requirement 13 defines a global h_stable, application developers can consider such stabilization height as local to process p (h_p,stable), without loss for generality. In contrast, the value of b MUST be the same across all processes.

Managing the Application state and related topics

Connection State

CometBFT maintains four concurrent ABCI connections, namely Consensus Connection, Mempool Connection, Info/Query Connection, and Snapshot Connection. It is common for an application to maintain a distinct copy of the state for each connection, which are synchronized upon Commit calls.

Concurrency

In principle, each of the four ABCI connections operates concurrently with one another. This means applications need to ensure access to state is thread safe. Both the default in-process ABCI client and the default Go ABCI server use a global lock to guard the handling of events across all connections, so they are not concurrent at all. This means whether your app is compiled in-process with CometBFT using the NewLocalClient, or run out-of-process using the SocketServer, ABCI messages from all connections are received in sequence, one at a time.

The existence of this global mutex means Go application developers can get thread safety for application state by routing all reads and writes through the ABCI system. Thus it may be unsafe to expose application state directly to an RPC interface, and unless explicit measures are taken, all queries should be routed through the ABCI Query method.

FinalizeBlock

When the consensus algorithm decides on a block, CometBFT uses FinalizeBlock to send the decided block's data to the Application, which uses it to transition its state, but MUST NOT persist it; persisting MUST be done during Commit.

The Application must remember the latest height from which it has run a successful Commit so that it can tell CometBFT where to pick up from when it recovers from a crash. See information on the Handshake here.

Commit

The Application should persist its state during Commit, before returning from it.

Before invoking Commit, CometBFT locks the mempool and flushes the mempool connection. This ensures that no new messages will be received on the mempool connection during this processing step, providing an opportunity to safely update all four connection states to the latest committed state at the same time.

CometBFT unlocks the mempool after it has finished updating for the new block, which occurs asynchronously from Commit. See Mempool Update for more information on what the update task does.

WARNING: if the ABCI app logic processing the Commit message sends a /broadcast_tx_sync or /broadcast_tx and waits for the response before proceeding, it will deadlock. Executing broadcast_tx calls involves acquiring the mempool lock that CometBFT holds during the Commit call. Synchronous mempool-related calls must be avoided as part of the sequential logic of the Commit function.

Candidate States

CometBFT calls PrepareProposal when it is about to send a proposed block to the network. Likewise, CometBFT calls ProcessProposal upon reception of a proposed block from the network. The proposed block's data that is disclosed to the Application by these two methods is the following:

• the transaction list
• the LastCommit referring to the previous block
• the block header's hash (except in PrepareProposal, where it is not known yet)
• list of validators that misbehaved
• the block's timestamp
• NextValidatorsHash
• Proposer address

The Application may decide to immediately execute the given block (i.e., upon PrepareProposal or ProcessProposal). There are two main reasons why the Application may want to do this:

• Avoiding invalid transactions in blocks. In order to be sure that the block does not contain any invalid transaction, there may be no way other than fully executing the transactions in the block as though it was the decided block.
• Quick FinalizeBlock execution. Upon reception of the decided block via FinalizeBlock, if that same block was executed upon PrepareProposal or ProcessProposal and the resulting state was kept in memory, the Application can simply apply that state (faster) to the main state, rather than reexecuting the decided block (slower).

Excerpt (19967 of 59921 characters). Read the whole page on CometBFT specification ↗