Cosmos Hub v17.1 Chain Halt - Post-mortem

Overview On June 5th, 2024, at 19:21 (UTC), the Cosmos Hub chain halted, and users could not successfully execute any transactions. The incident occurred slightly after the scheduled v17 software upgrade took place. The upgrade triggered a bug when a validator leaves the active set of validators and another validator takes its place. The Informal Systems Cosmos Hub team was first informed about the incident through the Informal Staking team. The Informal Systems Cosmos Hub team, in coordination with Hypha and Binary Builders, provided the fix, and the chain resumed on June 6th, 2024, at 0:02 (UTC). Timeline Event | Block Height | Time (UTC) | Chain upgrade started | 20739800 | 16:58 (on June 5th, 2024) | Chain upgrade completed | 20739802 | 17:15 | Chain halts | 20740970 | 19:21 | Informed (over Slack) by the Informal Staking team that the chain has halted | chain is halted | 19:46 | Hypha (over Slack) confirms the error in their mainnet node as well | chain is halted | 19:47 | Hypha, the Informal Systems Cosmos Hub team, and Binary Builders meet on Zoom to fix the issue | chain is halted | 20:06 |…

Excerpt (1191 of 8945 characters). Read the whole post on the forum ↗

This was an incredibly rapid and effective post-mortem process – huge props to everyone involved from Informal especially in how quickly we got to the bottom of this behaviour from a technical standpoint. we should have inactive validators in future testnet scenarios. In Hypha’s internal debrief, we talked about why we didn’t encounter this issue on testnet even though we’ve been running v17 for weeks, including during ISLE. We try to keep most of the params on the testnet close to Hub params, including `maxValidators`. Right now our `provider` chain’s `maxValidators` param is 175 despite having only 45 validators actually active and securing the network. There’s plenty of room in the active set, so validators performing normal operations like bonding, unbonding, unjailing, etc don’t actually put the chain in a state where we have 176 bonded validators. Another thing we discussed is that our testnets have to meet two (sometimes contradictory) goals: training and realism . We want validators to know what to do and be capable of doing it perfectly, but we also want to experience the chaos that we’re sure to see on mainnet. For both ISLE and the Testnet Incentives…

Excerpt (1194 of 1910 characters). Read the whole post on the forum ↗

The QA team gets paid way too much money for this to happen. This clearly wasn’t some kind of weird edge case bug because it transpired immediately after the upgrade - in other words it should have been caught in the testnet. From what I see in lexa’s post, it is clear that the testnet is only a partial copy of the production environment. This is the reason Gavin has Kusama which is a pre-production environment. Pre-production environment are different than testing environments because they are not environment in which new functionality is tested but an environment which mirrors production conditions. Pre-prod (or UAT - user acceptance testing) environments are there exactly to catch bugs like the one that transpired here, for this type of situation. You are not testing new functionality, you are testing compatibility of the new code with the production environment. If there is anything to fix here, you need to create a UAT/pre-prod environment of some sort and then add a couple of days of UAT testing to your schedule before initiating a prod release. At one of the companies, the UAT environment was such a good mirror of production that we utilized it as an active backup. If…

Excerpt (1199 of 1906 characters). Read the whole post on the forum ↗

Thank you for the thorough technical explanation and timeline, especially on the heels of running this incident! The retrospective itself is exemplary work, and all of the teams involved have done a fantastic job to investigate, resolve, and account for the issue that triggered the chain halt.

First of all, thanks for the detailed post-mortem overview, it seems really thorough and I agree with all of what was said here. A few thoughts from a validator point of view: • IMO the solution to basically ignore the error if it happens seems like a workaround rather than a proper solution, and I also voiced it in Discord, but that has already been outlined in the post-mortem, appreciate that a lot • sometimes I had a feeling that the validator set felt like they have to wait for the team to do stuff without knowing the exact timing. I recall the GitLab case, when they accidentally dropped the production database: they made a public post-mortem with all of the details and what are they doing to prevent this from happening in the future (also mentioned in @lexa 's post), but one cool thing they did was having a livestream when they restored the data and basically answered the questions in chat. Not sure of that, but maybe having a Zoom call or something with devs who are trying to fix that and validators who can provide support, as some of us are also programmers and can help (but also so they’d be aware of what’s happening and were more informed and were able to understand…

Excerpt (1199 of 3964 characters). Read the whole post on the forum ↗

Thank you for providing an excellent post-mortem. This issue, initially identified on the Cosmos Hub chain, is something that could potentially occur on other chains utilizing the Cosmos SDK. We believe it’s crucial to thoroughly investigate potential corner cases within the cosmos-sdk and devise appropriate solutions.
What are your thoughts on establishing a contribution group to collectively conduct source code reviews and add test cases? We can definitely take this opportunity to enhance our processes.

• Please note that this is my personal opinion and does not reflect the views of DSRV.

Exemplary execution. Thank you much for the rapid and careful response and the clarity about the problem and solution.