Sentry Node Architecture Overview

Overview The Sentry Node Architecture (referred to as SNA in this document) is an infrastructure example for DDoS mitigation on Gaia / Cosmos Hub network validator nodes. Disclaimer It is important to understand that this is only one example of solving DDoS mitigation for validator nodes. For diversity in the network, validators are encouraged to implement their own solutions. Each validator is responsible for their own solution. This example might be missing crucial security features that need to be implemented for production use. Problem description On the Cosmos Hub, a validator node can be attacked using the Distributed Denial of Service method. The validator node has a fixed IP address and it opens a RESTful API port facing the Internet. Proposed solution To mitigate the issue, multiple distributed nodes (sentry nodes) are deployed in cloud environments. With the possibility of easy scaling, it is harder to make an impact on the validator node. New sentry nodes can be brought up during a DDoS attack and using the gossip network they can be integrated into the transaction flow. Network layout image1.png 772×797 43.4 KB The solution provided here is…

Excerpt (1196 of 10214 characters). Read the whole post on the forum ↗

To give credit where it is due, please thank @Greg for his great work on this doc! There will be additions and more info added here!

Thank for this post.
And just one note. In persistent_peers for sentrys. No need write the validator data. Or other sentrys. Just add other full-nodes and don’t your validator
Like .

Capture _2018-07-12-00-56-05.png1080×2004 345 KB

Great overview! Thanks Jack and Greg!

Just wanted to comment on the Sentry Node Configuration section where the text “They should also make sure they don’t gossip the validator node id and IP address, hence the private_peer_ids should contain the validator node’s ID.” is not supported by the table which shows that private_peer_ids is omitted (which would be true for the validator configuration above).

From my understanding, the body text is correct and the table should be updated to include the validator ID in the private_peer_ids field.

Cheers

This passage is excellent! It gives us more knowledge on security concerns and the architecture.

A little confusion on the pex=false and provate_peer_ids.

As discussed before, if pex=false is set on the validator node, it is not necessary to put the node id of the validator node in the private_peer_ids of the sentry nodes as the validator node will not gossip anyway. Or I have gotten this wrong?

Schema is hard to read in dark theme

Thanks for this very helpful information. I plan to read it a second time, to process it more thoroughly. Until then , I’m wondering -

1 - About the benefit of allowing sentry nodes to communicate directly to each other, i.e. w/the optional private connections?

2 - Is there still a concept of public and private sentry nodes? I can’t remember if I saw this mentioned in Riot somewhere or not.

Jack is fixing it right now. Thanks for pointing it out.

I have the draw.io files, we can create a light version. (The PNGs are transparent.) Will do later.

pex=false on the validator node means that the validator node will not gossip to anyone. pex=true on the sentry node means that the sentry node will gossip to everyone. This means that the persistent_peers list on the sentry node will be gossiped together with every other node detail that the sentry node receives through other gossips.
In effect, the sentry node will give away the nodeid & private IP address of the validator node. Unless of course if the sentry node is explicitly told not to do that by listing the nodeID in the private_peer_ids.

As Zaki pointed out to me in an internal discussion, it’s only the private IP of the validator node. My security standards tell me to not give away any crumb of information unless necessary but it’s not the end of the world, if private_peer_ids are not set.

Interesting idea. I was under the assumption, that a new sentry node can automatically talk to the validator node, if persistent_peers is set up - which is true, but it might not help the validator at all. (The sentry might deplete the validators resources while trying to sync up and the validator will not try to connect to the sentry node.) I’ll do some more digging and if I don’t see any problems with this, I’ll update the doc.

One thing I don’t like is that a sentry node only becomes useful when the validator node is updated (Sentry node added to persistent_peers on the validator.) Currently, this is only possible during a maintenance schedule of the validator, since it requires the validator to be restarted. (So, I wouldn’t automate it just yet.) I have to check if sending a NOHUP signal would reload the config, I’m not sure.

The other thing that comes to mind is that there is a limitation, how many peers the validator node is going to connect to. (Current default is 50.) It’s something to be aware of when someone sets up too many sentry nodes.

I was not going to address this because I think these are the kind of questions the community should discuss among the members. But alas, here’re my two cents: • This is something to discuss. I see the benefit of connecting to trusted nodes - especially in a hostile environment. A scenario I can think of is trying to sync up during a DDoS attack and sync-up is hindered because of malicious nodes timing out on you. (Or your public Internet connection is already saturated.) It’s definitely not necessary for the core SNA setup but you might find it useful when creating your threat model. • Based on the configuration, you can set up a sentry node to be private. (The task is left for the reader as an excercise. ) You can use it to have “warmed up” nodes that you want to add soon-but-not-just-yet to your defense system or other use-cases, like, making snapshots of the private node so you can use that as your template for sentry nodes (see the issue with slow syncing when you bring up a node and automation ideas). This is again up to you and you have to find your own use-case. The core SNA didn’t discuss it. Maybe it’s worth creating extensions to this document that deals with…

Excerpt (1196 of 1223 characters). Read the whole post on the forum ↗

hi, there. thanks. **when you said. ** One thing I don’t like is that a sentry node only becomes useful when the validator node is updated (Sentry node added to persistent_peers on the validator.) the sentry is useful all time, because is connected almost to more 40 peers all time, and the validator is only connected to the sentrys example 4. i am missing some translation maybe. help me to understand. and in the other hand when you said: “Currently, this is only possible during a maintenance schedule of the validator, since it requires the validator to be restarted. (So, I wouldn’t automate it just yet.)” is not automate yet, but when validator lost peers form peersisten_peer list, need to connect automate again for my undertand and for logical to the sentrys for the peersitent_peer: list. now is one issue open about this, but is in Terdermint. for that now sentrys not sure is going to work in the next release gaia-7000 for this Gaia-6002 Postmortem Validation Just for my understands. This issue is open and now in Tendermint; so, is not resolved yet. for the gaia-7000 for example. And is related to the update from Terdermint and not for Cosmos.…

Excerpt (1199 of 1403 characters). Read the whole post on the forum ↗

But is different the setup for one sentry work private_peer, and other peersitent_peer.

sentry can work private, or public. (different setup each mode.)

for what i understand.

in Private mode have to used private_peer: only id, no need id+ip+port .

@kwunyeung i have some notes from the test, i am wrong now?

Thanks for the reply. This is what I expected to happen but it’s not. I was confused because in gaia-6002, when I put the nodeID of the validator node in the private_peer_ids, one can still see the info of the validator node under n_peers with the private IP of the node and is_outbound equals false. This made me think of setting up private sentry which won’t gossip and only let the validator node to connect to, and it connects to the public sentry in private network. Then the public sentry will gossip info about the private sentry but never get the info of the validator node.

Maybe the behavior is different in gaia-7000. Let’s see how it happens with the setup.

Thanks!

Greg: One thing I don’t like is that a sentry node only becomes useful when the validator node is updated (Sentry node added to persistent_peers on the validator.) What @Greg wanna explain was that when there is a new sentry, the node address has to be updated on `persistent_peers` of the validator node. Otherwise, the new sentry node won’t be useful as the validator node is not talking to the new sentry node. Greg: Currently, this is only possible during a maintenance schedule of the validator, since it requires the validator to be restarted. He meant the validator node has to be restarted as the config.toml file was updated with the new sentry node address. @Greg there is a way to avoid restarting the validator node which is to add the a new peer using the `/dial_peers` endpoint. @jack has mentioned about this in the following thread. Sentry nodes - What they are! - How they work! - Why they exist! Validation @katernoir This doc has some more information about the /dial_peers endpoint. You need to enable it by setting [rpc]unsafe = true in your config.toml. Also for enabling autoscaling you will need to be taking regular…

Excerpt (1190 of 1476 characters). Read the whole post on the forum ↗

Question, let’s say I’m not using a private network to connect a sentry node to a validator, should -

addr_book_strict = false

In this case too?

If there are no private addresses you can leave addr_book_strict as true.

when you reference private addresses, do you mean addresses you don’t want gossiped or private networking addresses?

@Greg or @jack Assuming a sentry and validator are running, how can one confirm the validator is running through the sentry and not directly through to the network, i.e. how can one confirm the validator is running “behind the sentry”?

What logging are you seeing in the validator? Is it signing blocks? Thats one way to definitely tell. You should also see your validator in the dump_consensus_state results:

http://<node_ip>:26657/dump_consensus_state

Yes, it’s signing blocks and I see it in the dump_consensus_state results. However, How do I know it’s connecting through the sentry rather than simply connecting directly to the network?

You can see what peers your node is connected to by using the net_info route:

http://<node_ip>:26657/net_info

jack:

http://<node_ip>:26657/net_info

Thanks. So I see my validator connected to ~38 peers. If it was connected to my sentry, I would only see the sentry correct?

chris-chainflow:

Thanks. So I see my validator connected to ~38 peers. If it was connected to my sentry, I would only see the sentry correct?

That’s correct. You can get a quick list of all the peers your validator (or any node) is connected to with:
curl -s http://localhost:26657/net_info |grep moniker

In order to get your validator to only connect to your sentries, you need to do a few things. On the validator:
pex needs to be false.
no seeds
only your sentries in persistent peers
Since you already have a bunch of peers, and you want to get rid of them, delete your address book.

On your sentries
Do not put your validator in persistent peers
Do put your validator’s ID (just ID, not ID@IP:port) in private peers
pex needs to be true

I think you will still have a problem, because at least 32 nodes know about your validator, and they will try to reconnect. Firewall rules to prohibit incoming connections will help, so that your validator decides who it connects to, and/or move it to a new IP number that won’t be in anyone’s address book.

I’m pretty new to Cosmos and still understanding Validators, so forgive me if this is a stupid question.

Is it possible to run multiple Validator servers behind your Sentry Nodes? Either by load balancing them, or just having your Sentry Node talked to one of them directly.

@gkrizek:HA validators is something I’m thinking about, too. You can’t just set up HAProxy in front of 2 validators and Round-Robin between them because of the double-signing issue. Let’s say delegators have staked their tokens with your validator but you are running two carbon copies behind an LB…

The carbon copies will also have the same ~/.gaiad/config/priv_validator.json which will cause you to double-sign blocks and get slashed and unbonded.

What I am thinking of doing is running a full node alongside my live validator; the full node will keep a copy of all the blocks but when the validator dies, I will overwrite the full node’s priv_validator.json with the one from the validator (and perhaps re-run gaiacli stake create-validator ...)

Fully automated failover will require scripting and some way to trigger the script when the validator dies. Any suggestions?

Glad I’m not the only one! That’s exactly what I was afraid would be the problem, double-signing. That’s not a bad idea to have a stand-by node to flip to for a Validator. Although I would still much prefer to run multiple Validators at once, if that’s not possible then this would do. If you didn’t do something like this, how would you preform maintenance? You get slashed for downtime, so how are you supposed to update your Validator?

Yeah you could definitely script the whole failover. How would depend on what software you are using, but regardless of that, you should have a service running monitoring on your whole Validator infrastructure. You should be able to check if that Validator is running by hitting the /health endpoint. (There might be a better way to check if it’s running than that, but it would work). If it doesn’t respond or responds in a bad way, then trigger your failover script. There are tons of ways to actually script that out, either with bash scripts and HA proxy or with some Cloud solutions.

Hello. Are you guys also using LoadBalancers ? If so, how do you guys inject newly spun up instances [id@ip:port] in config.toml? Each new node should have a new ID and maybe even a new IP in some cases. Cheers !

I’m not sure what kind of setup @archjun is running, but it seems totally possible to run your Validator behind a Load Balancer with a master/slave type setup. To speak specifically to your question, @jack posted an answer to this:

This doc has some more information about the /dial_peers endpoint. You need to enable it by setting [rpc]unsafe = true in your config.toml .

When your Sentrys are spun up you can use the /dial_peers rpc endpoint to add them to your validator. This can easily be automated.

Edit: Fixed link

But I don’t quite understand how load balancing is working on a p2p connection.

I think there are some important distinctions to make here. First, what I was referring to was a type of HA setup for the Validator only. (nothing to do with Sentry Nodes). We were discussing how you would run more than 1 Validator in order to make it High Available. We determined that it theoretically should be possible to run a Validator and a Full Node side by side behind a type of Load Balancer or Proxy. You could have some kind of health check running on the validator and if it fails, it turns the Full Node into a Validator (with the same key) and routes traffic to it. This isn’t really “load balancing” because there is only 1 server receiving traffic at all times. But a load balancer could be used to achieve it. I hope that makes sense. Another thing to note with this setup, is you would need a way to manage state between the two Validators. Like when a new Sentry comes online, you could use the `/dial_peers` endpoint to add it to the Validator. But you would have to make sure the secondary Validator knows about the new Sentry as well in case of failover. For the Sentry Nodes specifically, that’s something I’m try to dig into now. I’m not familiar enough with them and…

Excerpt (1198 of 1891 characters). Read the whole post on the forum ↗

I’ve been thinking that validators should consider adding another type of node to their Architecture- Relay Nodes.

Here is the definition of a Relay node. A Relay Node is a full node that only make connections to sentry nodes of other validators that the operator of the Relay Node. it runs with pex disabled. The firewall on the Relay node blocks all connections from ip address other than what is on a white list.

The Relay node operator will white list the ip addresses of other validators sentries on the relay node firewall and the validators will add the Relay node’s id & ip address to persistent peers.

The presence of a small number of relay nodes could help ensure that consensus operates at maximum efficiency.

I think these relay nodes from different validators can connect to each other too. Assuming the validator node only connect the relay nodes, the validator node depends the network performance of the relay nodes. Actually the relay node you are defining is my original interpretation of “private sentry”.

I do have some questions, and hopefully I can find some answers:

1. with the Sentry node architecture, the validators IP is hidden. How will the other validators know each other’s IP address to perform consensus, or is this not required and we solely rely on broadcasting.
2. If the validators do not know each other’s IP address, wouldn’t there be a performance hit? (additional relay time)
3. How would the network measure a validators up time now that they are hidden.

Thank you

This is an interesting idea. So would Relay Nodes be an alternative to Sentry Nodes? If I’m understanding correctly, these seem the same as Sentry Nodes, but they only allow connections from other trusted Sentry Nodes.

If this is instead a proposal for Relay Nodes to be an addition to Sentry Nodes, then it seems like maybe an over complication. But maybe I need to evaluate it more.

I’d extend this idea by suggesting that private sentry / relay nodes can communicate over private links, rather than public internet. VPC networks within GCP and AWS can peer within a cloud platform, and connect between platforms using VPN links.

1.) Yes, Validator IP Addresses are hidden. They shouldn’t even have a public IP at all. The rest of the network will know about it because its votes and blocks are broadcasted via the Sentry Nodes.

2.) Yes, theoretically there would be a small performance hit when using Sentry Nodes. I think it’s just expected to create an infrastructure for your Validator that is highly optimized and as quick as possible. As far as I know, the performance of a Validator doesn’t matter. As long as it’s adhering to the rules of no double-sign, no downtime, and, always vote, it will stay in the network. Maybe @zaki could chime in here about performance requirements of Validators.

3.) It’s not like the network does a ping to the Validator to see if it’s up. It’s checked by it’s participation in the network. If it’s private, you should still see it participating in the network.

I’ve started to look into Sentry Nodes more in depth and I think my original vision for them was a little off. I was originally imaging something like SentryA, SentryB, and SentryC where each of those actually go to a Load Balancer. Then have a group of servers behind the load balancer with the same `external_addr` and `~/.gaiad/config/node_key.json`. These servers would scale based on # of requests or something to handle DDoS attacks. (I’m still exploring the possibility of this even being an option.) But the more I’ve thought about it, shouldn’t Sentry Nodes be transient? Setup something like a lifecycle for Sentry Nodes where they only live for like 6 or 12 hours. Then automatically replace the servers with fresh ones (and fresh IPs). These could too scale based on traffic in the event of a DDoS Attack. And if one server is getting hammered, just remove it and create a new one. This would be much more of a moving target rather than strictly handling the load horizontally. Of course, this method also requires a lot more tooling for handling this dynamic of an infrastructure. Sentry Node availability is just as important as Validator availability, right? If all your Sentrys…

Excerpt (1197 of 1244 characters). Read the whole post on the forum ↗

That’s an interesting idea. I’m also thinking the private sentry/relay nodes should not always connecting to the same set of sentry nodes in the persistent peers. As the relay nodes won’t gossip and they rely on the public sentry nodes to connect to the network, it the small numbers of public sentry nodes disconnect, the validator node can’t be synced and push votes. It will be interesting if the relay nodes would switch to connect to different known health sentry nodes from time to time. The list of sentry nodes should be managed by the validators themselves. gkrizek: Sentry Node availability is just as important as Validator availability, right? If all your Sentrys go down, your Validator does as well, correct? This is exactly how I experienced in 7001. The sync speed was slow. Even all my connected sentries are healthy and can sync up-to-date, the validator node was always out-of-sync evening `catching up` was false. It had to wait `public network > sentry > relay > validator` The validator node had to wait until relay to be synced, the relay waited until sentry to be synced. That made the validator node always missed votes. If we need the validator…

Excerpt (1195 of 1887 characters). Read the whole post on the forum ↗

Where/how do we find the “ID” for a node on which gaiad isn’t running yet?

“gaiacli status” won’t work if gaiad isn’t running. I don’t want to run gaiad first, because then the validator would be visible. Ideally, there’s a way to find “ID” on a node where gaiad isn’t running.

I think @kwunyeung might have pointed me to this earlier in Riot…

You can run gaiad tendermint show_node_id. It reads the node_key.json to generate the node ID.

For (yes, shameless plug) https://validator.network , we have developed a small script to enable sentry discovery. The script serves two purposes: • Avoids the hazard of requiring sentries to “dial in” to the validator(s), but instead let the validator discover sentries and only establish outbound connections. • Enables “local peer” discovery between sentries Prerequisites: • Requires unsafe RPC (unsafe = true in config.toml) • As a consequence, the RPC should be proxied by nginx or similar to ensure only /status is exposed • Sentry RPC must be behind a load balancer which will distribute traffic among instances (round robin) So the basic idea is that anyone (be it a 3rd party, sentry or validator) requesting /status via the load balancer will receive a random status containing node-id, ip and port. Do this enough times, periodically, and one will eventually learn about all sentry nodes. A local node (sentry or validator) can then feed this information into the local gaiad instance using the /dial_peers RPC. Like so: ``` #!/bin/bash -e while true do STATUS=$(curl -s https://gaia.validator.network/status) PEER=$(echo ${STATUS} | jq -r…

Excerpt (1182 of 1718 characters). Read the whole post on the forum ↗

Hello everyone. I posted this to Cosmos Discord too, but in the interest of time and greater exposure, am posting it here too. I want to get this right, and need expert feedback on sentry architecture. Promise to write a medium post on this once I am done :). Here is what I’ve designed as my sentry-validator architecture based on numerous posts I have seen. I am a bit puzzled why nobody has suggested a VPN for both sentries and validators as I mention here (unless it amounts to crazy VPN costs). Would greatly appreciate any input as I am in the process of automating this. ONE big note: I am placing the validators in the cloud – not in a data center with dedicated hardware. Please tell me your most critical thoughts. Also, I don’t see any specific mention of a “sentry” P2P option and assume that Sentry is a result of settings and context. • We need a VPN on which all the sentries and validators have an IP address. This VPN itself is inaccessible to the public Internet, with respect to the addresses on the VPN. The VPN IP addresses are therefore not Internet accessible. • Each sentry is assumed to be PAIRED with one or more validators each. • Validators: ONLY have a…

Excerpt (1193 of 2356 characters). Read the whole post on the forum ↗

Connecting the sentries and validator node via VPN have been mentioned many times in different posts. It was also mentioned in the first post from @jack in this thread. Sentry, Relay, Validator are how we call the nodes depends how you set them up in the infrastructure, just like proxy and load balancer. It’s more on the functionality and how you connect the nodes. If you are looking for setting up VPN between nodes, you may consider WireGuard or Tinc.

Currently, dVPN Node Hosting is available on the Sentinel Tendermint Testnet.
"Node on docker and share unused bandwidth "
Protocols supported:

• OpenVPN
• WireGuard is coming soon
  https://medium.com/@kidinamoto/setup-cosmos-validator-relay-network-6b6e63661100
  https://docs.sentinel.co/dVPN/Nodes/tendermint/
  Query the forum/web before post/ask question

What’s the benefit of the VPN? I don’t see how it makes the setup more robust or secure.

then how do you setup you sentries environment without vpn?

What would you need the VPN for? The communication between the nodes is already encrypted and authenticated.

Thanks to everyone who has contributed to this thread Coming to this thread three years after it started – I’m preparing for www.Regen.Network mainnet launch, and various people have referenced this thread as a good source of info. So, I’m curious if any of this info has changed – maybe there are other threads that are more appropriate for some of these questions – I will look for other threads later. jack: solving DDoS mitigation for validator nodes Now that various cloud / datacenter providers offer DDoS protection (for example: OVH, Vultr): Are any validators using those DDoS solutions? Are they a decent option for protecting a validator? When this thread started in 2018, SNA was a concept. Now, how has it worked in practice? Now that various networks have been in production for a while, what attacks have actually happened? Where can we find more info about real attacks, what impact they had, and how validators responded? How has SNA affected network latency? jack: For diversity in the network, validators are encouraged to implement their own solutions. What other solutions have been created and put into practice? The only other one I…

Excerpt (1198 of 4474 characters). Read the whole post on the forum ↗

Just published some documentation about sentry node config in Spanish (focused on the config/toml and app.toml files) – ahora publicamos documentación parcial para la configuración de esta arquitectura de nodos guardianes en castellano:

Arquitectura · Wiki · Jonathan Finlay / regen

Planificación del proyecto

Seguiremos mejorando esa documentación con el pasar del tiempo.

patogit:

Is there still no viable cloud KMS? I’m not sure what to look for. A quick search for cloud KMS yields services from Amazon, Google, Alibaba and Tencent.

Amazon and Google HSM confirmed to still not support the algorithm that Tendermint uses.

There is a fork of Tendermint KMS in development focused on signing in TEE (Trusted Execution Environments)** : currently, Intel(R) SGX and AWS Nitro Enclaves are supported. This would allow an alternative to the YubiHSM2 hardware HSM. See details at:

1. GitHub - crypto-com/tmkms-light: TEE-based Key Management System for Tendermint validators.
2. https://github.com/tomtau/tmkms/blob/feature/nitro-enclave/README.nitro.md

Zerotier fan here

But the last time I used it properly was gaia-5001.

Currently considering what it looks like if a whole chain were to adopt i2p.

There would probably be increased latency but you would definitely gain some protection because real world IP addresses wouldn’t be known.

I think there is a case to be made for chains that disappear and i2p seems to fit the bill.

Currently, my thinking on “the hot setup” is:

• Sentries in akash
• Blocks signed with tmkms

One thing I like very much about tmkms is that it should allow for single board computers at the edge of the network to play a bigger role in validation.

I noted some other conversations about cloud-based key management systems with some concern. I think that keys really do not belong in the cloud, even if they’re living in some kind of an enclave like nitro.

SGX shouldn’t be trusted, there have been too many incidents where total compromises are demonstrated.

why i can’t post new post !

Hello! I think about that an entire network could use i2p. Have you progressed on this research, where could I find out more? Thank you!

Specifically into I2P, not really. However I have looked into several other transport protocols and have tested nebula, which is a product that grew out of the team at slack.

The only difference between I2P and nebula is that I2P possibly provides privacy for the actual IP address of your machine.

Currently I’m not working too much on this but if you do have any questions about building tendermint networks I’m very happy to discuss that with you.

Twitter.com/gadikian

Hey one other thing that I wanted to mention – I think it would be really fun to launch an i2p cosmos testnet. Would you like to try?

I think this could reinvigorate some key security practices that I’d love to see strengthened.

The Sentry Node Architecture provides a proactive approach to DDoS mitigation for Cosmos Hub validator nodes. By leveraging distributed sentry nodes and a gossip network, the solution aims to ensure the continued operation of the validator node even under attack. Validators are urged to customize and enhance the architecture according to their security needs, recognizing their individual responsibility for the robustness of their DDoS mitigation solutions.

is the i2p testnet stlll a thing?

@catdotfish @lexa

Why is this post still open?

Been way more than 14 days

I think every time someone answers, it reopens for another 14 days. oops. btw. it was a serious question. lets organize something similar