[ANN] Tendermint KMS v0.2: Validator Signing Support

Tendermint Key Management System (KMS) , a.k.a. `tmkms` , is a signature service backed by Hardware Security Modules (HSMs), namely YubiHSM2 devices from Yubico, and soon, the Ledger Nano S . It’s intended to be run alongside Cosmos Validators, ideally on separate physical hosts, providing defense-in-depth for online validator signing keys as well as a central signing service that can be used when operating multiple validators in several Cosmos Zones. This release (and the previous, unannounced v0.1 release) contain initial support for using `tmkms` as a `priv_validator` backend, providing full end-to-end support for storing consensus keys in a YubiHSM2 device and using them on the upcoming gaia-9002 and Game of Stakes testnets. Note that the code is presently alpha quality and this will be the first time it is (potentially) usable on a live testnet. Expect crashes, bugs, and protocol changes for the time being. Installation For detailed install instructions, please see: • Project README: https://github.com/tendermint/kms#installation • tmkms v0.0.1 release notes Short list: • Install Rust: https://rustup.rs/ • Install tmkms: `cargo install tmkms` • Linux:…

Excerpt (1198 of 2962 characters). Read the whole post on the forum ↗

I’m having trouble building v0.2.0. Apologies if this resulting form newb errors, rust is new to me.

error[E0658]: `crate` in paths is experimental (see issue #45477)
   --> /home/mattharrop/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.13.5/src/test.rs:463:9
    |
463 |     use crate::{error, polyfill, private, rand};
    |         ^^^^^

error: aborting due to 71 previous errors

For more information about this error, try `rustc --explain E0658`.
error: Could not compile `ring`.
warning: build failed, waiting for other jobs to finish...
error: failed to compile `tmkms v0.2.0`, intermediate artifacts can be found at `/tmp/cargo-installetzhiM`

Caused by:
  build failed

I tried replacing my rust install with the nightly build, no difference.

Any ideas?

Thanks,
matt

Looks like one of the dependencies was recently updated to require rust 1.30. However, nightly should also work, if it’s new enough.

What does rustc --version say?

I was on 1.29.2
This is the version that was installed with curl https://sh.rustup.rs -sSf | sh, today.

I have updated to 1.30.1 using rustup update stable and problem solved.

Thanks for the tip!

Hello. How does one configure secret_key = "path/to/secret_connection.key" and
path = "path/to/signing.key" ?
I also get this error while doing a test

tmkms yubihsm keys test 1

yubihsm: unrecognized command `test`

aurel:

Hello. How does one configure secret_key = "path/to/secret_connection.key"

This will get autocreated when it doesn’t exist (we should probably add a full tmkms init command ala gaiad though).

Regarding this whole section of the config:

[[providers.softsign]]
id = "gaia-8000"
path = "path/to/signing.key"

This is a testing-only software backed signer in the event you want to test tmkms but don’t have a YubiHSM2. If you do, you can simply delete this whole section of the config.

If you do actually want to generate a software-backed key though, the command is tmkms generate path/to/signing.key

aurel:

I also get this error while doing a test

The command to run a test is: tmkms yubihsm test 1 (you had an extra keys there)

Thanks! Everything works well.

This will get autocreated when it doesn’t exist

@iqlusion I am setting this to test with a local testnet. Where the secret key will be created?

Thanks!

The secret key will be created at the path you specify, provided the parent directories exist.

Ar… I see! Thank you! That’s very clever.

Got it working, thanks for the instructions!

Just want to be sure that it is intentional that the secret_connection file contains all zeroes?

xxd /opt/tmkms/config/secret_connection.key
00000000: 0000 0000 0000 0000 0000 0000 0000 0000 …
00000010: 0000 0000 0000 0000 0000 0000 0000 0000 …

BR,
Martin

mdyring:

Just want to be sure that it is intentional that the secret_connection file contains all zeroes?

That is definitely NOT intentional, and is a high-severity bug. I have opened an issue here:

Yes I get the same.

BTW, what capabilities are required for tmkms? I want to create an authkey which only have enough capabilities for gaiad.

mdyring:

xxd /opt/tmkms/config/secret_connection.key
00000000: 0000 0000 0000 0000 0000 0000 0000 0000 …
00000010: 0000 0000 0000 0000 0000 0000 0000 0000 …

I have reproduced this problem on Linux. The problem does not seem to occur on macOS. I am continuing to investigate.

kwunyeung:

BTW, what capabilities are required for tmkms? I want to create an authkey which only have enough capabilities for gaiad.

The only capability presently needed is asymmetric_sign_eddsa, however this may change as tmkms evolves.

I plan on writing a (customizable) provisioning tool that can generate a multi-level account hierarchy including administrative, operational, auditing, and application roles. When that happens, it will consider the full set of capabilities needed for each role.

I’ve put out a small point release, v0.2.1, which should hopefully address all known bugs, including one for another feature in the original release which was buggy so I didn’t announce it: priv_validator.json import support If you’ve already registered a key in the genesis.json for an upcoming testnet, such as gaia-9002, you can import it into a YubiHSM2 using the new `tmkms yubihsm keys import` command: ``` $ tmkms yubihsm keys import --path ./priv_validator.json 9002 ``` This will import the Ed25519 key from the `./priv_validator.json` file into slot 9002 of the configured YubiHSM2 (you will need to have `tmkms.toml` configured in advance). Secret Connection key mini-post-mortem A quick aside: the format for SecretConnection keys (i.e. `secret_key` under the `[[validator]]` section) is now Base64. This means any previously generated keys will not load and are unusable, but that’s ok because any keys generated by a release build were all zeroes! If you see an error like this, please delete the offending key to ensure it’s regenerated: ``` config error: error loading SecretConnection key from tmkms.key: invalid encoding ``` The root cause of the issue was a bug in…

Excerpt (1199 of 1778 characters). Read the whole post on the forum ↗

Quick status report on attempting to use tmkms on gaia-9002:

We had to squash a few bugs around things like key encodings, but we managed to have tmkms live for a little bit.

We encountered a bug which appears to be a blocker for using tmkms on a testnet for the time being:

https://github.com/tendermint/kms/issues/130

It looks like it shouldn’t be difficult to resolve though. Hopefully we can have another point release out soon, along with instructions for how to enroll a tmkms-backed validator into a testnet.

I’ve released tmkms v0.2.2 which, to my knowledge, addresses all remaining known bugs.

We’ve been using it live on gaia-9002 and have so far it has produced over 30,000 signatures.

Tony, what should be the settings for both gaiad and tmkms if both of them are running on the same machine?

There are a couple options for localhost usage. When configuring tmkms.toml:

[[validator]]
addr = [...] 

You can use either TCP or a Unix domain socket to talk to gaiad running on the same host:

• Pick a port (e.g. 12345) and use tcp://127.0.0.1:12345 or tcp://localhost:12345. This will use local TCP/IP loopback
• Use a Unix domain socket: unix:///path/to/socket

When configuring gaiad’s ~/.gaiad/config/config.toml, make sure to use the same value for priv_validator_laddr

Thank you! They connected to each other correctly now.

When I configure the line

priv_validator_laddr = "tcp://localhost:26666"

in config.toml, gaiad starts up with this error:
I[7126-12-07|20:42:34.593] Starting ABCI with Tendermint module=main E[7126-12-07|20:42:37.737] OnStart module=privval err="accept tcp 127.0.0.1:26666: i/o timeout" ERROR: Error with private validator socket client: failed to start: accept tcp 127.0.0.1:26666: i/o timeout

I’m on an ubuntu 18.04 host. gaiad 0.27.1

Any ideas?

EDIT: Problem solved, but brings up a question. The issue was starting things in the wrong order. Starting the KMS first solves the problem. Question – is it intended that gaiad exits if the KMS isn’t online?

I tried stopping KMS after gaiad and KMS are connected. Gaiad will blame not seeing the remote signing service. If I start KMS again, gaiad won’t connect it back.

mattharrop: EDIT: Problem solved, but brings up a question. The issue was starting things in the wrong order. Starting the KMS first solves the problem. Question – is it intended that gaiad exits if the KMS isn’t online? If you run both processes under supervisors (e.g. systemd), the ordering doesn’t matter as KMS will continue trying to connect, and gaiad will keep restarting waiting for a KMS connection. That said, the timeouts right now on both sides are pretty bad and not presently adjustable. I think KMS should try to reconnect more frequently, and gaiad should wait longer for a KMS connection (and ideally start asynchronously while it waits for KMS to connect) kwunyeung: I tried stopping KMS after gaiad and KMS are connected. Gaiad will blame not seeing the remote signing service. If I start KMS again, gaiad won’t connect it back. I suspect it was this bug, which should be fixed on the `develop` branch: github.com/tendermint/tendermint Close and retry a RemoteSigner on err `tendermint:develop` ← `zmanian:zaki/retry_tcp_validator` opened 02:31AM - 18 Nov 18 UTC zmanian +5 -0…

Excerpt (1194 of 1310 characters). Read the whole post on the forum ↗

Hello, i have a question:

Tendermint’s height increased with time, then i did three operations:
1、generate softsign.key, and corresponding amendments to tmkms.toml of tmkms.
2、use tests/support/secret_connection.key as secret_key in tmkms.toml of tmkms.
3、change priv_validator_laddr of tendermint and then they connect each other success.

Then i discovered the height of tendermint stop increasing, i do not know why？

Is that all right? Do i have anything else to do?

I’m having trouble figuring out how to configure gaiad to use the KMS signing keys.

The KMS is connecting to my gaiad process. Using the gaiad tendermint show-validator command does not show the key that the KMS is serving up, it shows the key from the local priv_validator.json file. I’ve tried commenting out the priv_validator_file = line. I’ve tried deleting priv_validator.json. Not sure what else to try.

The KMS log indicates 01:02:58 [INFO] [genki-2000@tcp://127.0.0.1:12345] connected to validator successfully

Matt, do you have errors after deleting the line or the file?

There is no error when I delete the file, gaiad recreates a new one on startup.

I have it working now - I declared the validator using the public key from the kms. It works.

Matt, is the height of tendermint still growing after connecting successed with kms?

My kms log also indicates [test-chain-CJ8kTB@tcp://127.0.0.1:26659] connected to validator successfully, but i don’t know if signing service is working correct or not, because the height of tendermint stopped growing.

mattharrop: I have it working now - I declared the validator using the public key from the kms. It works. Great news! This is the only thing that worked for me too. I think there are parts of gaiad that still hardcoded to `priv_validator.json`. To use `tmkms` on a testnet, I needed to take the Bech32 key that `tmkms` printed out (via `tmkms yubihsm keys list`) and pass that as the argument to `gaiad tx stake create-validator`: ``` $ gaiacli tx stake create-validator --amount=10000STAKE --pubkey=cosmosvalconspub1zcjduepqfgjuveq2raetnjt4xwpffm63kmguxv2chdhvhf5lhslmtgeunh8qmf7exk --moniker="iqlusion.io" --chain-id=gaia-9002 [...] ``` For this to work, however, you’ll need v0.2.2+ of `tmkms` (v0.2.3 includes some timeout fixes which you’ll probably want). Earlier versions printed out incorrect `cosmosvalconspub` addresses. yulidai: My kms log also indicates [test-chain-CJ8kTB@tcp://127.0.0.1:26659] connected to validator successfully , but i don’t know if signing service is working correct or not, because the height of tendermint stopped growing. You’ll need to create a validator with a `pubkey` that matches one of the keys stored in `tmkms` for it…

Excerpt (1197 of 1350 characters). Read the whole post on the forum ↗

Thank you for replying

1、The content of signing.key in tmkms is

GNx76GuJt8rWkruCYZKieE7P4eVYymPD4MkNS1OuWLc=

2、Then execute in tendermint

tendermint unsafe_reset_all

3、Editing the priv_validator.json in tendermint like below:

{
  ...

  "pub_key": {
    "type": "tendermint/PubKeyEd25519",
    "value": "GNx76GuJt8rWkruCYZKieE7P4eVYymPD4MkNS1OuWLc="
  }
}

4、Executing in tendermint

tendermint node --proxy_app=kvstore

But the height of tendermint still not growing. And it’s strangely enough when execute

tendermint show_validator

Will get the result:

{
  "type":"tendermint/PubKeyEd25519",
  "value":"gn5KbGzulPkvN2uExWqb9FR2M2ThlupP5zFh3Bxw6t4="
}

Is there a cache or did I correct it?

yulidai:

Is there a cache or did I correct it?

Unfortunately, tendermint show_validator doesn’t query the kms for the public key but returns what it finds in $HOME/.tendermint/config/priv_validator.json (this is the default path). If you edited that file, you should see the same key as you’ve put in there.

Yes, i means i have edited $HOME/.tendermint/config/priv_validator.json but the result of tendermint show_validator still not change.

But if execute tendermint init，the result of tendermint show_validator will changed immediately.

But, when i change the value of pub key in the priv_validator.json, it would not changed.

I am very confused

Edit: if i remove priv_validator.json and restart tendermint, it will recreate priv_validator.json like matt said. And the result of tendermint show_validator will changed immediately too.

Seems the only thing that didn’t work was that i edited it. Strange

Looks like the pub-key in tendermint show_validator is derived from the private key in there and is not read from the pub_key field. That is confusing indeed.

If you use KMS, you should read the pub key from the HSM but not using gaiad tendermint show-validator.

Right, but i run show-validator just for testing. Because no effect after modify the pubkey’s value of priv_validator.json in tendermint

Whether use this HSM or not, you have to change the pubkey of priv_validator.json in tendermint, right?

mattharrop: I have it working now - I declared the validator using the public key from the kms. It works. How do you declared the validator using the public key from the kms ? iqlusion: You’ll need to create a validator with a `pubkey` that matches one of the keys stored in `tmkms` for it to begin signing transactions using the command above. How to create a validator with this pubkey? I just use tendermint instead gaia for testing, so what i did is modify the pubkey of priv_validator.json , but the value is EgkEmDLdrPuhbQGWSKoHc9fLnvGbmpyDlQwHplbavHE= Because the value like cosmosvalconspub1zcjduepqe9jexppzpyvg86unlu8d39xqa3xwwtn6allu5zdsp33kqc8tdnus7rmf78 is invalid, the error info after restart tendermint is: Error reading PrivValidator from /root/.tendermint/config/priv_validator.json: illegal base64 data at input byte 80 then i encrypt it with Base64 encoding, the error info is: Error reading PrivValidator from /root/.tendermint/config/priv_validator.json: decodeReflectJSONArray: byte-length mismatch, got 83 want 32 Maybe my understanding is wrong? What should i do for using pubkey like…

Excerpt (1164 of 1281 characters). Read the whole post on the forum ↗

yulidai: How do you declared the validator using the public key from the kms ? There are a couple methods to print out the public keys which are enrolled in `tmkms`: • `tmkms yubihsm keys list` will list all accessible, valid Ed25519 `cosmosvalconspub` keys, but only for the YubiHSM. Note that the YubiHSM will be one of the only two supported validator key storage methods for the Cosmos mainnet launch, the other being the forthcoming Ledger app. The `softsign` API is for testing-only, and will not be supported for production validators. If you have already purchased a YubiHSM, I would suggest using it instead of attempting to use `softsign`. • Running `tmkms` in verbose mode with `tmkms start -v` will print all public keys for all keys configured in the `tmkms` keyring including `softsign` keys. This is probably the easiest way to figure out which public key you should be using. For testnet purposes including Game of Stakes, regardless of how the private key is stored in `tmkms`, you will need to use one of the two methods above to determine the `cosmosvalconspub` address for your validator’s signing key. Pass that public key as the `--pubkey` parameter when you…

Excerpt (1198 of 1511 characters). Read the whole post on the forum ↗

Thanks for your patience. Now the main problem with me is how to create validator with designated pubkey , or, how to using the command you gived in local single-node. ``` $ cd $HOME $ gaiad init --chain-id testing --moniker testing $ gaiacli keys add validator $ gaiad add-genesis-account $(gaiacli keys show validator -a) 1000STAKE,1000validatorToken $ gaiad gentx --name validator $ gaiad collect-gentxs ``` Then error “ERROR: ABCIQuery: Post http://localhost:26657 : dial tcp 127.0.0.1:26657: connect: connection refused” occured when execute $ gaiacli tx stake create-validator --pubkey=cosmosvalconspub1zcjduepqe9jexppzpyvg86unlu8d39xqa3xwwtn6allu5zdsp33kqc8tdnus7rmf78 […] But if execute “gaiad start” first, error “ERROR: Msg 0 failed: {“codespace”:“STAKE”,“code”:101,“message”:“validator already exist for this operator address, must use new validator operator address”}” occured when execute $ gaiacli tx stake create-validator --pubkey=cosmosvalconspub1zcjduepqe9jexppzpyvg86unlu8d39xqa3xwwtn6allu5zdsp33kqc8tdnus7rmf78 […] Which of the above commands create validator by using my operator address? What is the right time to create my validator with designated…

Excerpt (1192 of 1318 characters). Read the whole post on the forum ↗

Solved the problem now! with these command:

$ gaiad init --chain-id testing --moniker testing
$ gaiad add-genesis-account $(gaiacli keys show validator -a) 1000STAKE,1000validatorToken
$ gaiad gentx \
$   --amount 100STAKE \
$   --commission-rate="0.10" \
$   --commission-max-rate="0.20" \
$   --commission-max-change-rate="0.01" \
$   --pubkey cosmosvalconspub1zcjduepqe9jexppzpyvg86unlu8d39xqa3xwwtn6allu5zdsp33kqc8tdnus7rmf78 \
$   --name validator
$ gaiad collect-gentxs
$ gaiad start

then modify priv_validator_laddr of config.toml for connecting kms, and signing service works good!

Thanks!!!

Edit:
So i think the tendermint/tmkms is only suit for gaia, not for tendermint with kvstore, is it right?

yulidai:

So i think the tendermint/tmkms is only suit for gaia, not for tendermint with kvstore, is it right?

The long-term plan for tmkms is for it to be generally useful for any Tendermint application, not just Cosmos, which is why it’s called “Tendermint KMS” and located under the tendermint GitHub organization.

However, for practical reasons we’re focusing on Cosmos right now, which is why the description says “Key Management System for Cosmos Validators”. It’s a bit confusing, I understand.

If you’re interested in using tmkms in a non-Cosmos Tendermint application, that’s a great thing to open a GitHub issue about.