Gaia-7003 Postmortem

We’ll investigate the gaia-7003 halt by traveling back in (virtual) time: starting at the fault and tracing back the causal antecedents until we find the root issue. Starting with the halt stack trace , we see that at block 28132, the Gaia state machine incorrectly attempted to unbond a validator (`974A7590D8BC1B9D323F7A9770D5388D324DA0E7`) which was already unbonded. This isn’t itself the bug - it’s just a sanity check, we architect the code defensively to panic any time we hit an unexpected state, which is the case here. Bonding and unbonding happen in the `UpdateBondedValidators` sequence , so that’s a likely place to start debugging. Using @jaekwon ’s gaiareplay debugging branch, we log the validator iteration sequence and start searching for any irregularities - if you want to follow along, you can run it yourself or download the output logs . Starting from the end of the log, we first see `974A7590D8BC1B9D323F7A9770D5388D324DA0E7` set as the cliff validator in block 27480 (line 200139) - but when we look at the validator set updates two lines below, we see that `974A7590D8BC1B9D323F7A9770D5388D324DA0E7` was unbonded! (compare the pubkeys) That’s not supposed to happen;…

Excerpt (1197 of 5444 characters). Read the whole post on the forum ↗