Use governance for emergency upgrades
Planned software upgrades are coordinated through a software upgrade proposal with a 2 week voting period. Discussions for an expedited (1 week voting period) software upgrade proposal process are underway here . Changing the process for regular software upgrade proposals is beyond the scope of this post. Current state of the Hub’s coordinated/emergency upgrades To begin, let’s think about our current process for all software upgrades. Why do some of them use governance and some not? • gaia versions use semantic versioning to refer to new versions. • In the binary’s code, the version name is a parameter whichis identified by only the MAJOR name (e.g., v16, not v16.1 or v16.4). The binary for v16.0.0 and the binary for v16.1.0 both used ‘v16’ as the version name parameter. • For those who remember v7-Theta and v8-rho, we had issues because the names are case-sensitive, which introduced opportunity for human error. After this, we changed to just the numeral format. • Examples of where this constant is identified in the gaia code: v10 , v16 • Example of v15.2 , where you can see the version name is still v15 • It’s a Hub norm that governance is used only for…
Excerpt (1199 of 3016 characters). Read the whole post on the forum ↗
Top-level post is intended to collect relevant info! Here are my personal opinions: Validator attention The Hub has a very responsive validator set that can be counted on in emergencies. I wouldn’t want to abuse that responsiveness by introducing a lot more rapid governance processes, however I prefer (and I think validators would prefer) rapid governance → on-chain data for emergency upgrades over rapid off-chain coordination via email and Discord. Below, I highlight the risk of exposing vulnerabilities and wonder about setting up an empty upgrade proposal (pointing at an empty repo) and then adding the binary 24h in advance of the upgrade height. If this is possible (and acceptable, from a software dev and social norm perspective) then I think the governance doesn’t actually have to be ‘rapid’ after all. We could probably use the same voting period as a normal software upgrade (whether that’s 1 or 2 weeks). Risk of exposing vulnerabilities Imo, the biggest risk in using governance for emergency upgrades is when we are fixing a major security vulnerability. In this case, we need to publish the fix and have it go through governance before doing the upgrade. The worst…
Excerpt (1197 of 2112 characters). Read the whole post on the forum ↗
As a validator, I heavily support this idea, here’s my overview: • What would using governance for emergency upgrades improve ? Currently, having an off-chain upgrade (that’s done via all validators setting a halt-height on their nodes and upgrading their binaries at the same time) has a few downsides compared to using governance upgrades: • There’s completely no info on chain about such an upgrade, the only way you can know about this is by the team reaching out to you via Discord, email or whatever. As of me, I prefer relying on the automated monitoring that should get me alerted about something I should keep in mind instead of Discord messages/emails: it’s really easy to get distracted and miss a really important notification about the emergency chain upgrade, while if it’s done by a monitoring system, it’s way more reliable. For example, this is one of the reasons I’ve build https://github.com/QuokkaStake/cosmos-node-exporter: it also provides you the information about the current upgrade plan for the proposal that has passed but not yet applied (and if Cosmovisor has the appropriate binary in the correct folder for it) and the estimation time till the block it’s…
Excerpt (1195 of 4093 characters). Read the whole post on the forum ↗
@lexa Thanks for starting the conversation and for sharing your views. I’d also welcome Marius’ thoughts on shifting from naming versions using only the major version to using the full semantic name. I believe Neutron does this ( example ) and thus uses governance for all upgrades, but it also introduces the risk of human error in failing to keep the release and upgrade name in sync ( example from Neutron). This shouldn’t be a problem. We used only the major version until now just because we didn’t need to use the minor. We never had a software upgrade proposal to bump minor versions of Gaia. Imo, the biggest risk in using governance for emergency upgrades is when we are fixing a major security vulnerability. In my view, emergency upgrades for both major and critical security vulnerabilities should not go through governance at all. Even sharing publicly that there is a critical vulnerability on the Hub would help attackers to exploit it. would it be possible to put an upgrade on chain and point it at a repo where the fix will be published 24h in advance of upgrade height, the way we currently do for coordinated emergency upgrades? Like we set up the upgrade…
Excerpt (1195 of 1580 characters). Read the whole post on the forum ↗