Cosmos Hub 3 Upgrade Post-Mortem

Cosmos Hub 3 Upgrade Post-Mortem In preparation for the upgrade, validators halted the cosmoshub-2 chain at 11:39 am UTC. Around twenty-five minutes into the migration, multiple validators uncovered an issue with the derived genesis file that was caused by a bug in the migration command. This bug ensured that cosmoshub-3 would fail to launch with the instructions provided in the upgrade proposal . Validators then followed the proposal’s recovery plan to relaunch the cosmoshub-2 chain, which resumed operating at 1:54 pm UTC. Total downtime for this failed upgrade was 2 hours and 15 minutes. This bug would seem to have been caught during the gaia-13004 (v0.34.7) to gaia-13005 (v1.0.0-rc3), but there were some additional changes to the migration logic that were not tested during the gaia-13005 to gaia-13006 (v2.0.0) upgrade. A fix to the migration issue has been contributed by Kwun Yeung and the SDK team is in the process of putting together a release (v2.0.2) of gaia (stay tuned here ). This failure to upgrade exposed flaws in our testing and upgrade procedures as well as some internal processes. In the course of discussions with validators over the last week the following…

Excerpt (1197 of 3292 characters). Read the whole post on the forum ↗

Currently, Cosmos is the most advanced blockchain technology and that article is a real proof, not only because of the IBC, but because the whole system is flexible in a constant evolutionary state.