Security advisory ICS-2024-002 and Gaia v19.2.0 patch retrospective

A security advisory has been made public here with a description of the bug and severity assessment performed by the ICS team and me. Please review this information as context for the patch process, timeline, and feedback received throughout this upgrade. Patch process Based on our severity assessment, Informal and Hypha decided on the following process: • Soft patch in a 6 hour window that covers most timezones (9:00 UTC - 15:00 UTC) • Have >⅓ of the Hub’s active set apply a privately-released security patch to mitigate the impact of an attack. • Once >⅓ have applied the patch, the chain will halt due to consensus errors if any of the four vulnerable message types are sent. • To recover from a chain halt, the remaining ⅔ of the active set will need to apply the same patch ASAP. • Coordinated upgrade on Sep-05 at 15:00 UTC • The Hub’s coordinated upgrade process for publicly released binaries is reliable and operators know how to do it. • If the chain halts, ⅓+ of the set has already upgraded and is running the canonical version of the chain. The dev team is primed to rollback a problematic transaction and instruct validators on next steps. Soft…

Excerpt (1194 of 10047 characters). Read the whole post on the forum ↗

Appreciate the detailed response. One question: so I worked with a big tech company, and we had a culture of incidents discussion, and one of the biggest aspects was so-called “action items”, e.g. “the list of things we’re doing so we can be sure that this would technically not be possible to happen again”. Does the team have any answer on how to prevent such bugs in the future? Asking because considering the bug description, it could’ve been really catastrophic if someone would be able to actually abuse it. Communication-wise I don’t have anything bad to say, other than it would’ve been better if the announcement said that we are safe to update at any time, but better to do it ASAP. Agree with the rest of what’s written in the initial post. lexa: Other chains use a Google sheet where validators can mark themselves as having completed the upgrade and see the voting power tally up. This is feedback we received after the last emergency upgrade. A bit off-topic but I also wonder if it’s possible to somehow build it into cosmos-sdk, so there’d be a way to say for sure which validator is using which version, so we won’t need Google spreadsheets or something…

Excerpt (1194 of 1687 characters). Read the whole post on the forum ↗

In review of the security issue, the procedure taken was acceptable.

The only source of confusion was the uncommon instruction of a halt-height when 33% already upgraded.

A halt-height makes sense if a halt until the estimated block time is accepted in case of an early attack.

Eliminating the halt risk would require all remaining validators to upgrade immediately.

freak12techno: A bit off-topic but I also wonder if it’s possible to somehow build it into cosmos-sdk, so there’d be a way to say for sure which validator is using which version, so we won’t need Google spreadsheets or something similar. Not off-topic at all, and this would make it much easier to keep track of upgrades. I’ve heard several technical ways of doing this discussed in the past, such as broadcasting the version in every signature (or maybe once per epoch nowadays?) but I honestly don’t remember why those ideas were dismissed. I recall someone suggesting that it is a security issue if someone is able to detect that the chain is split between versions, but I’m not an expert in this. I’ll ask around. The Google Sheet is not my preferred method, for the reasons you describe. It’s an open link and prone to human error. In the absence of a technical solution, the Google Form worked well enough…it’s a lot harder to accidentally type the wrong validator name than it is to hit the wrong checkbox. Here’s the form, for reference: https://forms.gle/pqpoREbYpCtbXmo1A Does the team have any answer on how to prevent such bugs in the future? We’re still in…

Excerpt (1190 of 2164 characters). Read the whole post on the forum ↗

Thank you for the question and the engagement.

One approach that’s seems worth exploring is using vote extensions.

This is the general idea:
It seems possible to write a module that would utilize vote extensions to signal that an upgrade was completed by a validator.

Upon starting the patched binary, the validator node would add a short string specifying which version of the software it is currently running. In this the nodes would have written v19.2.0.

We could use this information to establish the current rate of adoption of the new patched binary.

Limitations:
This would also signal to potential adversaries that the nodes are upgrading because the data is accessible to everyone. Mitigating this is possible but quickly turns a simple module into a more involved piece of code.

We are open to suggestions if you have other ideas.

EDIT: I will make another post to answer other questions.

MSalopek: It seems possible to write a module that would utilize vote extensions to signal that an upgrade was completed by a validator. Upon starting the patched binary, the validator node would add a short string specifying which version of the software it is currently running. In this the nodes would have written `v19.2.0`. Actually seems like a nice solution. I’d love to see an ADR on that. Thanks for elaborating! lexa: We’re still in conversation about it, and I know that’s not a very satisfying answer (either to give or to receive but it’s only been 4 days). As a suggestion, from my experience from working with enterprise-level companies, a nice way would be to have a public meeting to debrief such an incident. Here’s how we did it: I worked in a company that has like 20 different services within it, and every time there was a severe incident, there was a public meeting where the representatives of the service that had an incident were making a report, including the following: the incident timeline, how it was fixed, what went good, what went bad, and (the most important) the action items (e.g. “what can be done so this problem would not ever…

Excerpt (1199 of 1716 characters). Read the whole post on the forum ↗