Performance of Tendermint in Case of Large Number of Nodes

Hi all, could anyone share some information regarding this question: How many nodes can participate in a Tendermint PBFT network before the transaction rate is reduced – and, what is the network traffic that is produced in these networks? I know that the results will depend on the application, but any insights would be useful. Thanks a lot.

Hi, Thanks for reaching out. 300 Year 0: 100 Year 1: 113 Year 2: 127 Year 3: 144 Year 4: 163 Year 5: 184 Year 6: 208 Year 7: 235 Year 8: 265 Year 9: 300 Year 10: 300 On the grand scale globally, it is improved by a new paradigm in edge and core routing, with exceptional scalability, carrier-class reliability, environmentally conscious design : • aggregation services routers, optics fibers evolution across internet backbones improvements • low latency in the vacuum if data go via optical inter-satellite links and phased array • Moore law, Amdahl’s law, quantum computing mainstream usage sooner or later. More details Tendermint Upgrade - Commit Size Reduction Proposal Ideas Tendermint blocks contain a Commit structure that is a list of Votes, where each Vote has about 80 bytes of redundant data. With 100 validators, that means each block on the Cosmos Hub has ~8kB of unnecessary data. That’s ~100MB/day, and over 30GB/year. There is a proposal in ADR-025 to deduplicate this data, saving all our hard drives from it. It also saves some bandwidth, and will be especially beneficial to light clients and IBC, both of which depend on…

Excerpt (1194 of 1320 characters). Read the whole post on the forum ↗

Hi,

thanks a lot for your answer. I would be very thankful if you can share some insights on how you tested it and what would be the procedure to perform such tests? Also some insights into network traffic would be great.

Thanks again.

Sure

Yelong Overview

Data visualization & monitoring with support for Graphite, InfluxDB, Prometheus, Elasticsearch and many more databases

Cosmos Network Dashboard dashboard for Grafana

Should provide an overview over a node in the cosmos network using Prometheus

Kysenpool produces a fantastic job with the outpost for network stats, alongside other validators, monitoring the ecosystem.

Hi again,

I just want to make sure I understood the numbers good. So today Tendermint can handle 300 nodes without a drop in the transaction rate, right? Could you please clarify what are the years 0-10 in the above answer? Many thanks.

Hi, I have done some testing on Tendermint with testnets ( https://github.com/informalsystems/testnets ) using AWS EC2. I got the block creation time a bit too big (~10mins), where the mempool size is almost all time at its full (10K), but there are some time slots where is changes its size. Here I attach my config and the graphs. I just wonder why the block creation time is that big and what is going on? Btw, I have 100 validators network, deployed on AWS t3.medium instances and I did a load test with 1000txs/s for the duration of 5 mins. Thanks a lot for your time. tendermint-total-txs 1919×900 Configs: ``` # This is a TOML config file. # For more information, see https://github.com/toml-lang/toml ##### main base config options ##### # TCP or UNIX socket address of the ABCI application, # or the name of an ABCI application compiled in with the Tendermint binary proxy_app = "kvstore" # A custom human readable name for this node moniker = "" # If this node is many blocks behind the tip of the chain, FastSync # allows them to catchup quickly by downloading blocks in parallel # and verifying their commits fast_sync = true # Database backend: goleveldb | cleveldb |…

Excerpt (1199 of 10488 characters). Read the whole post on the forum ↗

I think because of a lot of gossip overhead among 100 nodes, t3.medium might not be able to handle 1000tx/s within several seconds.

practically, the validator nodes usually have much higher resource than t3.medium, and it usually has
only several very trusted internal peers to minimize gossip overheads.

so, i expect those are the reasons. From the number 10min(=600s), I expect only 1~2 nodes are succeeding for block creation in several seconds, rest of the attempts are failed because of lack of time I guess.

therefore, i suggest you to limit

• max_num_inbound_peers = 10
• max_num_outbound_peers = 10

and try the test again.
with this config, you might need sophisticated peering control to make all 100 nodes connected indirectly. (creating 5 clusters with whole connections and connecting 5 clusters can be a strategy.)

I think I have tried with:

• max_num_inbound_peers = 40
• max_num_outbound_peers = 30
  but the problem I had is that the 100 validators somehow do not get connected at all.

from my experience, i made a small script to write each config.toml with predefined persistent_peers.

if your nodes are i=1 to 100 then,
persistent_peers_of_node_i = node_(i+k), node_(i+10k) (where k=1 to 9)
of course you can deduct 100 if (i+k) or (i+10k) exceeds 100

then all nodes will be well connected with less peers (18 each)
theoretically, the maximum distance between any two random nodes is 2.

but Tendermint p2p is not very efficient in peering at first.
it is due to exponential backoff and weird dialing behavior.
this(https://github.com/tendermint/tendermint/blob/master/docs/architecture/adr-050-improved-trusted-peering.md) will let you have more stable connection among trusted peers.

Thanks a lot. So you basically suggest to use this tree: https://github.com/tendermint/tendermint/commit/701e9cac4d91474bfc04082e56a583bf77524f18 and to define 2 new parameters in the config.toml file: unconditional_peer_ids, and persistent_peers_max_dial_period. Right? Btw. these unconditional_peer_ids can be specified from the file config/addrbook.json file, but the file is empty. I use testnets: https://github.com/informalsystems/testnets
So basically I can add like 15-20 validators addresses to the unconditional_peer_ids field, or what would you suggest? Also what would you suggest for the value for the second parameter: persistent_peers_max_dial_period? Many thanks in advance!

It is included after Tendermint v0.33, so you can use any version after v0.33

those 2 new parameters are not for genesis.json, but for config.toml in each node.
and, for unconditional_peer_ids, you should not put “validator keys” but should put “node id”.
node id can be seen by “gaiacli tendermint show-node-id” from each node.
this can be seen “after” the initiation of gaiad.

for persistent_peers_max_dial_period, 60 is reasonable i guess.
it means you dial every disconnected persistent peers for each minute, which will not be any burden.

I have tried this for getting the node id:

[ec2-user@ec2-3-81-200-67 gaia]$ gaiacli status -n http://ec2-3-84-78-212.compute-1.amazonaws.com:26657
{“node_info”:{“protocol_version”:{“p2p”:“7”,“block”:“10”,“app”:“0”},“id”:“31aaac3e4f7ecd2dfeee16089ba2dcd2c9824e8b”,“listen_addr”:“tcp://0.0.0.0:26656”,“network”:“testnet_abcd”,“version”:“0.31.7”,“channels”:“4020212223303800”,“moniker”:“ec2-3-84-78-212.compute-1.amazonaws.com”,“other”:{“tx_index”:“off”,“rpc_address”:“tcp://0.0.0.0:26657”}},“sync_info”:{“latest_block_hash”:“ABEEF5CBEB169334B754C20CF436BA38776C73FD6E628C64A96C0BF4D297DE9F”,“latest_app_hash”:“”,“latest_block_height”:“1”,“latest_block_time”:“2020-04-17T09:59:44.415854Z”,“catching_up”:false},“validator_info”:{“address”:“B221B42E63FD825E711B115EDD8B36E99BA939A1”,“pub_key”:{“type”:“tendermint/PubKeyEd25519”,“value”:“94wpwymSS0Ht56I2o206Kyc7CdagEiQTkPqu8NT7cS4=”},“voting_power”:“1000”}}

The id should be:

"id":"31aaac3e4f7ecd2dfeee16089ba2dcd2c9824e8b"

Right? Many thanks again!

Hi again, I have tried to test with tendermint` v0.33.3-13eff7f7` as you suggested and my config file is the following: ``` proxy_app = "noop" moniker = "ec2-3-81-200-67.compute-1.amazonaws.com" fast_sync = false db_backend = "memdb" db_dir = "data" log_level = "*:error" log_format = "plain" genesis_file = "config/genesis.json" priv_validator_key_file = "config/priv_validator_key.json" priv_validator_state_file = "data/priv_validator_state.json" priv_validator_laddr = "" node_key_file = "config/node_key.json" abci = "socket" prof_laddr = "" filter_peers = false [rpc] laddr = "tcp://0.0.0.0:26657" cors_allowed_origins = [] cors_allowed_methods = [ "HEAD", "GET", "POST",] cors_allowed_headers = [ "Origin", "Accept", "Content-Type", "X-Requested-With", "X-Server-Time",] grpc_laddr = "" grpc_max_open_connections = 900 unsafe = false max_open_connections = 900 max_subscription_clients = 100 max_subscriptions_per_client = 5 timeout_broadcast_tx_commit = "10s" max_body_bytes = 1000000 max_header_bytes = 1048576 tls_cert_file = "" tls_key_file = "" [p2p] unconditional_peer_ids =…

Excerpt (1093 of 3839 characters). Read the whole post on the forum ↗

Hi,

I have managed to run the v 0.33 version with my setup. However, if only unconditional_peer_ids field is defined, without persistent_peers, the connection to the unconditional_peer_ids is not established from some reason. Also I put max_number_of_inbound_peers and max_number_of_outbound_peers equals 10, so the nodes made additional connections beside the 10 connections I specified in the unconditional_peer_ids (i.e. in the persistent_peers) so that the maximal number of connections per node was 30 = 10 (persistent_peers or unconditional_peers) + 10 inbound connections + 10 outbound connections. I also tries to specify the max_number_of_inbound_peers and max_number_of_outbound_peers equals 0, so that each node only makes the connections to the persistent_peers, i.e. unconditional_peers but that did not work, and the nodes finished with 0 connections. Is there a way to tell node to only connect to the persistent peers? Also do you have any other suggestion how I can further improve the performance of the network? Many thanks! Btw. the block time interval now looks like this (for 100 validators and 1000tx/s for the period of 3 mins):

graph3974×388 39.4 KB

if only unconditional_peer_ids field is defined, without persistent_peers, the connection to the unconditional_peer_ids is not established from some reason. this is an expected behavior, because, if you see “unconditional_peer_ids”, it only store node_id. So, your node does not have IP address to peer the node. So, the unconditional_peer_ids is only supposed to be used when calculating maximum inbound/outbound peers, but not supposed to be used for making new connection. Also I put max_number_of_inbound_peers and max_number_of_outbound_peers equals 10, so the nodes made additional connections beside the 10 connections I specified in the unconditional_peer_ids (i.e. in the persistent_peers) so that the maximal number of connections per node was 30 = 10 (persistent_peers or unconditional_peers) + 10 inbound connections + 10 outbound connections. this is also an expected result. If you want to only allow 10 specified peers, then you should set • maximum inbound connection = 0 • maximum outbound connection = 0 this is because unconditional peers can be connected regardless of maximums. Then I expect you will get max 10 peers, which are members of the 10 specified…

Excerpt (1195 of 1987 characters). Read the whole post on the forum ↗

try put 10 specified peers in persistent_peers. the nodes need them to be listed in persistent_peers so that they are allowed to dial the peers.

unconditional_peer_ids only means that the node will “allow” the peer to connect even if maximum connection is reached, but it does not generate connection itself.

I have done some testing on Tendermint with testnets github (.) com/informalsystems/testnets using AWS EC2 I got the block creation time a bit too big (~10mins), where the mempool size is almost all time at its full (10K), but there are some time slots where is changes its size. Here I attach my config and the graphs. I just wonder why the block creation time is that big and what is going on? Btw, I have 100 validators network, deployed on AWS t3.medium instances and I did a load test with 1000txs/s for the duration of 5 mins. Thanks a lot for your time. Configs: ``` # This is a TOML config file. # For more information, see https://github.com/toml-lang/toml ##### main base config options ##### # TCP or UNIX socket address of the ABCI application, # or the name of an ABCI application compiled in with the Tendermint binary proxy_app = "kvstore" # A custom human readable name for this node moniker = "" # If this node is many blocks behind the tip of the chain, FastSync # allows them to catchup quickly by downloading blocks in parallel # and verifying their commits fast_sync = true # Database backend: goleveldb | cleveldb | boltdb # * goleveldb…

Excerpt (1180 of 10483 characters). Read the whole post on the forum ↗