Backing up Validator Server (Physical Data Center)

Redundancy of Validator Server (Physical Datacenter) Security and maintenance have been mentioned multiple times among validators, since it plays a critical role in safe operation of validator nodes within the network. While considering multiple security structures, such as the article “Sentry Node Architecture”, our team has come to an issue of making a backup for Validator server as one of crucial issue. Documentation of the Cosmos Network recommends that validators keep their Validator nodes located within a local datacenter, while operating Sentry Nodes in cloud environment such as AWS or GCP. But even in a well-managed datacenter, there can be several unexpected issues that will bring the validator node down: • Power of data center goes down • Hardware malfunction • Numerous reasons that can possibly affect the healthy operation of a validator node. Thus, we have come to a new idea to prevent the above problems through setting the validator node by the procedure below: • Connect NFS to two servers • Set up validator in the connected NFS • Create two identical accounts in each of the two servers • Run validator on one of the newly created…

Excerpt (1196 of 2069 characters). Read the whole post on the forum ↗

Some notes on this approach: • Your NFS server will be a single point of failure, and running a highly available NFS cluster (or anything that touches storage, really) is a science in itself. You now have two separate interdependent HA clusters to care about instead of just one (the validator and NFS). • A highly available enterprise SAN is very expensive and there’s still a chance of failure. • NFS is very latency-sensitive, so you can’t distribute it across multiple data centers. Same goes for a SAN - there are mechanisms for cross-data center mirroring, but they’re asynchronous (and therefore useless). • You will need a bullet-proof failover mechanism like pacemaker with an odd number of nodes to ensure that there’s always ever at most one validator process running, otherwise, you will end up double signing. Pacemaker and friends aren’t designed for cross data center operation, either, and finnicky to operate. • Failover will be rather slow and you will miss blocks. • By sharing the disk storage, you effectively have a single failure domain: there are a number of failure scenarios that you can’t recover from, like corrupted files, a filled-up disk or…

Excerpt (1191 of 2113 characters). Read the whole post on the forum ↗

thanks for your kind answer

i guess its important to understand that ‘slashing’ or any type of punishment is greater for double signing and actions that are ‘arbitrary’, compared to missing few blocks once in a while.

Can use DRBD for replication dual node, but important to manage the risk of double signature !!!