Cloud Platform peering issues

I’ve having ongoing issues with nodes hosted on google cloud. They work, but they do not collect more than a handful of peers. Peer connection drop frequently. The only thing I can think of that makes GCP different from the simple cloud providers is the external <-> internal address mapping. On the GCP side I have `external_address` set to the external IP, and `laddr` set to the internal IP. I isolated a typical peer session in the logs from both ends, one on GCP and one on OVH, a cloud provider that terminates real IP numbers directly on the instance. This same snippet repeats in the logs constantly. I’m not certain that this is a tendermint issue, but it seems likely to me. Is anyone else seeing an issue like this? instance-1 hosted on GCP (external <-> internal IP) ``` Jul 18 15:19:30 instance-1 sh[2422]: I[07-18|15:19:30.869] Will dial address module=p2p [email protected]:26656 Jul 18 15:19:30 instance-1 sh[2422]: I[07-18|15:19:30.870] Dialing peer module=p2p [email protected]:26656 Jul 18 15:19:30 instance-1 sh[2422]:…

Excerpt (1194 of 6206 characters). Read the whole post on the forum ↗

I also see the “use of closed network connection” errors on our sentries on GCP, occurring every few minutes.

Two things to note:

When connecting to a peer running with --p2p.seed_mode, the peer will disconnect immediately after sending some peers and we’ll see this as a EOF. We should do a better job of sending an error message so this is less mysterious.

When connecting to a peer that already has the maximum number of connections, the peer will disconnect and we’ll see EOF.

Could explain some of what’s happening here.

I’m having the same issue. When I restart the validator, I end up with a lot of MConnection flush failed … use of closed network connection which persists until I restart the sentry nodes. Sentry 1 has a public IP Sentry 2 and the validator are behind NAT ``` Sentry 1 grpc_max_open_connections = 900 max_open_connections = 900 max_subscription_clients = 100 max_subscriptions_per_client = 5 persistent_peers = "[email protected]:26656,[email protected]:26656" addr_book_strict = false max_num_inbound_peers = 10 max_num_outbound_peers = 5 pex = true seed_mode = false private_peer_ids = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" max_open_connections = 3 Sentry 2 grpc_max_open_connections = 900 max_open_connections = 900 max_subscription_clients = 100 max_subscriptions_per_client = 5 persistent_peers = "[email protected]:26656,[email protected]:26656" addr_book_strict = false max_num_inbound_peers = 10 max_num_outbound_peers = 5 pex = true seed_mode = false private_peer_ids = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" max_open_connections = 3…

Excerpt (1196 of 1624 characters). Read the whole post on the forum ↗