[Sane Defaults] Universal use of remote signers
The community consensus is quite clear that using a remote signer is the correct way to configure one’s validator. Why doesn’t Notional use a remote signer? human error I personally call validators after they get slashed. In the case of this most recent slashon the hub, one of them is a dog and so I haven’t called them. The other is @serejandmyself and I haven’t called them because they have produced very detailed documentation on the incident. Both were victims of human error – and not using a remote signer. Using a remote signer goes outside the default flow of comet. In the default flow of comet, there is a signing key in the filesystem, eg: ``` cd ~/.gaia/config config % ls -a -l total 239248 drwxr-xr-x 9 faddat staff 288 Jun 18 04:33 . drwxr-xr-x 4 faddat staff 128 Jun 16 15:22 .. -rw-r--r-- 1 faddat staff 1058374 Jun 16 15:30 addrbook.json -rw-r--r-- 1 faddat staff 9446 Jun 16 15:22 app.toml -rw------- 1 faddat staff 742 Jun 18 04:33 client.toml -rw-r--r-- 1 faddat staff 18605 Jun 16 15:22 config.toml -rw-r--r-- 1 faddat staff 121386449 Jun 16 15:22 genesis.json -rw------- 1 faddat staff 148 Jun 16…
Excerpt (1195 of 4961 characters). Read the whole post on the forum ↗
Hi Jacob, jacobgadikian: A number of double signing incidents have actually come from users in the process of adopting a remote signer, forgetting to take the key out of the file system, and then and ending up double signing. The remote signer migration When you adopt a remote signer, you set the priv_validator_laddr ``` # TCP or UNIX socket address for Tendermint to listen on for # connections from an external PrivValidator process priv_validator_laddr = "tcp://a.b.c.d:1234" ``` Once this setting is set, the node stop looking at the local priv_validator_state but will interact with the remote signer to get the signatures to broadcast. This is on this point the validators fails. Because they do not report the current validator node state to the remote signer. Migrating to remote signer is something to do when you are not drunk/tired/distracted. I also suspect the validators who double sign on migration to not have made the migration on a testnet first to exercise the process. The node data reset In the case of the Pumpmos and Citizen Cosmos, the error come from a reset of the node data. ie the command `<bin> tendermint unsafe-reset-all` even…
Excerpt (1199 of 3119 characters). Read the whole post on the forum ↗
David_Crosnest:But when a validator is not confident enough to make an action, he must avoid doing it.
Exactly my point.
You’ve described a complex and error-prone process, that people can do wrong. I’d love to make it impossible for people to do it wrong.
David_Crosnest:I also suspect the validators who double sign on migration to not have made the migration on a testnet first to exercise the process.
My point again. So, we’ve tested remote signers, and my concern is human error.
Without too terribly much work, we can fully eliminate the possiblity of human error in this process, so that:
- migrations are completely safe
- everyone uses remote signers in production
Sounds like a lot of overhead and changes for simply asking validators to NOT double sign. The system works amazing as-is.
Just need validators to operate their machines as if they have hundreds of thousands (millions) at stake…which they do.
It’s actually not a ton of overhead though. I think that the changes are relatively minor. We already have a test mode that keeps the key and ram and adding a flag is not hard. The real goal here is to put an end to conversations where we encourage validators to do things that are outside of the default flow. There’s also this insight that I’ve gained from seeing even really excellent teams screw up various processes, those interviews that I have had with validators after getting slashed I’ve been a real eye-opener. I wish that I had been keeping formal notes, and we’ll do so in the future, but going from memory, I would say that we have had no malicious double signs except for the case on chronos where where an actual attacker had seized control of a validators system, and the attacker deliberately chose to double sign because the team that was being attacked kept trying to take control of their node again. I would also say that approximately half of the teams I’ve contacted post/ have failed in the configuration of a remote signer. Contrary to what @David_Crosnest is claiming above, these have at times been very experienced teams who know the stack extremely well, where…
Excerpt (1198 of 1765 characters). Read the whole post on the forum ↗
jacobgadikian: I would say that we have had no malicious double signs except for the case on chronos where where an actual attacker had seized control of a validators system, and the attacker deliberately chose to double sign because the team that was being attacked kept trying to take control of their node again. If the server is compromised, and the remote signer is running next to the node, the result will be the same. And we cannot go against the opinion of @Golden-Ratio-Staking Golden-Ratio-Staking: Just need validators to operate their machines as if they have hundreds of thousands (millions) at stake…which they do. When you have the responsability of milions of USD, the saving of peoples that trust you to not loose it, you must learn every days and make many test before to do it live. jacobgadikian: I would also say that approximately half of the teams I’ve contacted post/ have failed in the configuration of a remote signer. If they fail in the configuration on a testnet, that’s fine, it is supposed to happen. and for this point you are able to learn from your mistake and make multiple successful try before to make it for…
Excerpt (1198 of 1339 characters). Read the whole post on the forum ↗
David_Crosnest:When you have the responsability of milions of USD, the saving of peoples that trust you to not loose it, you must learn every days and make many test before to do it live.
There isn’t one single way to accept and serve out responsibility @David_Crosnest.