Solution for running active-active validator nodes

Many validators want to run two active-validator nodes for production. We have ran validators without any problems for 1+ month. So I want to share to you guys. • Servers: two validators + one tmkms server. we did’t use sentries in this test. but you can add it for production. 2 Steps: 2.1 Install HSM server: visit tmkms for more information, Make sure that your version of tmkms is higher than >= v0.6.3, otherwise you will get double signing. 2.2 setup tmkms.toml: ``` [[chain]] id = "kava-testnet-2000" key_format = { type = "bech32", account_key_prefix = "kavapub", consensus_key_prefix = "kavavalconspub" } [[validator]] addr = "tcp://validator-1-ip:26658" chain_id = "kava-testnet-2000" secret_key = "/data/test.key" [[validator]] addr = "tcp://validator-2-ip:26658" chain_id = "kava-testnet-2000" secret_key = "/data/test.key" [[providers.yubihsm]] adapter = { type = "usb" } auth = { key = 4, password = "kms-validator-password-1y58g2...." } keys = [{chain_ids = ["kava-testnet-2000"], key = 11}] ``` 2.3 start tmkms server. then tmkms server will try to connect to validator nodes. 2.4 edit config.toml of each validator: ``` # TCP or UNIX socket address…

Excerpt (1199 of 2631 characters). Read the whole post on the forum ↗

@ping, Thanks for sharing!

does the two validators need to share the same secret key?

A quick note on this: while this functionality is available in the latest releases of KMS, we’re still worried about validators running this configuration in a steady state. See this blog post on the topic:

Postmortem: 2019-08-07 Tendermint KMS-related Cosmos Hub Validator Incident •...

Last night and early this morning we encountered a series of small outages in iqlusion’s Cosmos Hub validator related to a recently released version of Tendermint KMS: v0.6.1. We are the primary contributors to Tendermint KMS and generally try to......

Notable excerpt:

For this reason we recommend validators don’t run this sort of configuration in perpetuity, but use the functionality to failover between validators. Based on this incident, and others we’ll describe below, we in fact recommend you only run in this configuration on testnets for now because we think this operational mode needs a lot more testing to be safe .

I don’t think secret key must be same.

B-Harvest is researching on using multiple consensus key for one validator operator. (Upgrading tendermint/cosmos-sdk)

The concept is that only one of the consensus key can propose blocks, but any of the listed key can vote on consensus.

Possible advantage of this solution is

1. achieve active/active setup without kms
2. achieve active active with actual decreased latency(validator server nearest to the proposer will vote), so result in shorter global block time, possibly under 2 seconds, not losing any geological resilience of the network.

You’re talking about some sort of load balancing on blockchain, right ?
We would be interested in the research, as we are using cosmos sdk to register a big amount of transactions in a short period of time
How do you deal with staking ? Does every consensus key gets the same voting power ?

Thanks for asking. From our draft design, one validator operator possess multiple consensus keys. • It is kind of very restricted “load balancing” + “co-location” of validator operation because more number of consensus keys implies more block header cost(size of header and computational cost). Therefore practically it will only allow several consensus keys(for practically 3, America/Europe/Asia) per validator operator. • I guess the packet traffic performance will be greatly increased to allow the network to handle more transaction in given time, because most validator servers participating in a round of block will be located in same continent.(usually having much better speed if servers are closely located) • on staking perspective, there is no difference. delegators are delegating tokens to “validator operator”, not “each consensus key”. • not all consensus key of a validator participate in consensus. only the one which arrives earliest(from the eyes of the proposer) will be valid to participate in consensus. rest of the consensus keys will be ignored or cannot reach because of short timeout. For example, • validator A has proposing validator in SF, voting…

Excerpt (1198 of 1837 characters). Read the whole post on the forum ↗