Syncing stopped; panic: Unkown pub key type
So earlier today I got a message from my monitor idicating that the syncing stopped for a given sentry. Normally I SSH into the machine restart gaiad and its working again. This time that didn’t work. When checking the logs It seems to got stuck at block 302207, which eventually ends up in a panic causing gaiad to stop. Log looks like this: `Sep 13 15:50:01 doghouse2 systemd[1]: Started Cosmos Gaia Sentry 1. Sep 13 15:50:02 doghouse2 gaiad[10304]: I[09-13|13:50:02.011] Starting ABCI with Tendermint module=main Sep 13 15:50:02 doghouse2 gaiad[10304]: I[09-13|13:50:02.170] Starting multiAppConn module=proxy impl=multiAppConn Sep 13 15:50:02 doghouse2 gaiad[10304]: I[09-13|13:50:02.170] Starting localClient module=abci-client connection=query impl=localClient Sep 13 15:50:02 doghouse2 gaiad[10304]: I[09-13|13:50:02.170] Starting localClient module=abci-client connection=mempool impl=localClient Sep 13 15:50:02 doghouse2 gaiad[10304]: I[09-13|13:50:02.170] Starting localClient module=abci-client connection=consensus impl=localClient Sep 13 15:50:02 doghouse2 gaiad[10304]: I[09-13|13:50:02.170] ABCI Handshake module=consensus appHeight=302206…
Excerpt (1159 of 9904 characters). Read the whole post on the forum ↗
So, I did a unsafe_reset_all and it started syncing from 0. Yet, it again stopped syncing at block 302207 with a panic: Unknown pubkey type. Only fix I can imagine now is cloning a data folder of a working sentry and replace it with the data folder that results into this error. Still its not a good fix, but im not sure what else I can do.
Are you sure you are running the right version? This would explain why your sentry node isn’t in consensus with the other nodes.
One cause might be if dep ensure wasn’t run
The stuck sentry is running on gaiad 0.24.2-0bf061b7. What do you mean exactly with “One cause might be if dep ensure wasn’t run”. How can I run dep ensure?
run make get_vendor_deps in the SDK repo. That will handle making sure all of the SDK’s dependencies are installed with the correct version.
yes I did that, but with no result. It’s still stuck
I saw this bug on 7005. this bug is caused by Evidence. Here is block #302207 . Block{ Header{ ChainID: gaia-8001 Height: 302207 Time: 2018-09-13 07:44:05.967930651 +0000 UTC NumTxs: 0 TotalTxs: 27429 LastBlockID: 964EDFF97CB4FF77F5BC35C001EC1BAD4FE96F03:1:CC3617D7459D LastCommit: 728F50FC4153CA9F356818B93E080E7C09126E69 Data: Validators: 47A9259082080A05ED133D2C9603FE99DCD5DB6B App: 72F5882F762B55DFB119E015BA31A7F1B8C07A85 Consensus: D6B74BB35BDFFD8392340F2A379173548AE188FE Results: Evidence: 74E8D63C6C36F3409AD5CA3ECB24F7D4CCCA3608 } #E7F24667DCE11E9D78BA6558E8055DE32C123317 Data{ }# EvidenceData{ Evidence:VoteA: Vote{27:570B6A755ED3 302207/00/1(Prevote) 104C214A0ED3 73FAAB3BD709 @ 2018-09-13T07:44:03.398900853Z}; VoteB: Vote{27:570B6A755ED3 302207/00/1(Prevote) 882090C3B8CF 20B96D112F7B @ 2018-09-13T07:44:09.828057795Z} } #74E8D63C6C36F3409AD5CA3ECB24F7D4CCCA3608 Commit{ BlockID: 964EDFF97CB4FF77F5BC35C001EC1BAD4FE96F03:1:CC3617D7459D Precommits: Vote{0:01F78669F951 302206/00/2(Precommit) 964EDFF97CB4 DB075A707F1C @ 2018-09-13T07:43:57.763348976Z} Evidence data: EvidenceData{ Evidence:VoteA: Vote{27:570B6A755ED3 302207/00/1(Prevote)…
Excerpt (1194 of 1559 characters). Read the whole post on the forum ↗
Here is what i saw: github.com/cosmos/cosmos-sdk Error decoding message at block #30798 on gaiad-7005 opened 02:03AM - 15 Aug 18 UTC closed 04:27PM - 18 Sep 18 UTC liangping <!-- < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < < ☺ v … ✰ Thanks for opening an issue! ✰ v Before smashing the submit button please review the template. v Please also ensure that this is not a duplicate issue :) ☺ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --> ## Summary of Bug One of my full-node stop synced at #30798. then i can see many error messages like: `E[08-15|01:44:53.958] Error decoding message module=blockchain src="Peer{MConn{54.193.28.244:26656} 27cfb50a6c593a76b1fd02824150e5d8f0e14af8 **out**}" chId=64 msg="[bcBlockResponseMessage 30800]" err="unmarshal to blockchain.BlockchainMessage failed after 210 bytes (Unregistered interface types.Evidence):…
Excerpt (969 of 3275 characters). Read the whole post on the forum ↗
Thx for the help. Created backup like Melea described and successfully transferred it to the troubled sentry. Now everything works perfectly again
Fix underway.
Issue: Evidence Deserialization
It looks like we still have an unfixed deserialization bug. cosmos/cosmos-sdk#2031
Hej, thanks for bringing this up and reporting the issue, a hotfix against Tendermint master is underway and will be released later today: https://github.com/tendermint/tendermint/pull/2424
Hi guys, This bug should have been fixed in v0.24.1 of the SDK. The panic pasted above starts off with: ``` panic: Unknown pubkey type Sep 13 15:50:02 doghouse2 gaiad[10304]: goroutine 1 [running]: Sep 13 15:50:02 doghouse2 gaiad[10304]: github.com/cosmos/cosmos-sdk/x/slashing.BeginBlocker(0x10a9b80, 0xc42175e5a0, 0xc4217534c0, 0x8, 0xc421758f40, 0x14, 0x20, 0xc422139400, 0x9, Sep 13 15:50:02 doghouse2 gaiad[10304]: /home/nckrtl/go/src/github.com/cosmos/cosmos-sdk/x/slashing/tick.go:34 +0x6ec Sep 13 15:50:02 doghouse2 gaiad[10304]: github.com/cosmos/cosmos-sdk/cmd/gaia/app.(*GaiaApp).BeginBlocker(0xc420a14000, 0x10a9b80, 0xc42175e5a0, 0xc4217534c0, 0x8, 0xc421758f40, 0x1 Sep 13 15:50:02 doghouse2 gaiad[10304]: /home/nckrtl/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/app/app.go:139 ``` Note the stack trace says the panic is called at github.com/cosmos/cosmos-sdk/x/slashing/tick.go:34 . So if we look there on v0.24.2, we don’t see a panic: https://github.com/cosmos/cosmos-sdk/blob/v0.24.2/x/slashing/tick.go#L34 But at v0.24.0, we do: https://github.com/cosmos/cosmos-sdk/blob/v0.24.0/x/slashing/tick.go#L34 Indeed, this was the bug that was fixed in v0.24.1. Is it…
Excerpt (1198 of 1259 characters). Read the whole post on the forum ↗
Im sure I applied the hotfix asap as it became available. Gaiad version shows 0.24.2-0bf061b7. Perhaps I didn’t run other steps properly like make get_vendor_deps etc. Hard to check as its now running just fine with the backup restored from another sentry. I