[Downfall.fail] Shared Tenancy Considered Harmful

Downfall note and call to action The first thing I need to say is that Robin from our team is going to have a more detailed write up on this topic. I will be submitting this forum post and Robin’s forum post to the ICF and to AIB in the hopes that both organizations will delegate only to validators who own their own machines from here forward. it is entirely unsafe to operate a validator on any shared tenancy machine (AWS, gce, and any cloud virtual machine) This has probably been true since the rowhammer attacks, but it seems that downfall makes clear what many of us have long suspected. Everything about cloud scale virtualization and shared tenancy computing completely inappropriate for the blockchain space. You can find more information at Downfall Attacks Downfall Downfall attacks targets a critical weakness found in billions of modern processors used in personal and cloud computers. advice for large delegators who is a large delegator? • Liquid staking protocols • Funds • Family offices • Individual large holders of atoms In a proof of state network it is folks like yourselves who secure and govern the network.…

Excerpt (1199 of 4337 characters). Read the whole post on the forum ↗

I’m Jacob Gadikian, CEO at Notional and this post is my attestation that we use no shared tenancy machines for validation and are therefore not at risk from the downfall attack.

Delegators who would like more information about our systems are encouraged to reach out on Twitter to myself directly at twitter.com/gadikian

The majority of our systems are hosted at our office in Hanoi Vietnam, and and we have a small number of systems hosted at hetzner in Germany and Finland exclusively on single tenant machines.

All of our cosmos hub systems are hosted in Hanoi on equipment that is in our physical custody.

I am founder of Architect Nodes and happy to see this discussion on Cosmos forum. Architect Nodes is primarily using self owned Bare Metal servers in our possession for validator nodes. We also use rented single tenancy Bare Metal servers for redundancy with different providers that are using remote signers with yubihsm in our possession.

Security is important and we should encourage safer practices when it comes to running validator nodes

I am Tricky, Co-Founder of Cosmos Spaces. The typical set up for a chain that we validate consist of Horcrux clusters running on a mix of self hosted and dedicated bare-metal servers. The machines are spread across different locations in order to guarantee the most redundancy and security as possible.

We’re always working to ensure we have a top-tier operation for our delegators.

Tricky | Cosmos Spaces:

The typical set up for a chain that we validate consist of Horcrux clusters running on a mix of self hosted and dedicated bare-metal servers.

So let me just clarify one thing, every single machine that you have cryptographic material on is exclusively single tenancy?

Hello, everyone. Ryan (aka Phunky) here from Lucky Friday Labs, LLC. I have verified with our tech team that we only use “single tenancy” on our nodes. All of our validators (and RPCs, etc) are run on fully owned bare metal servers and are under lock and key in SOC2 compliant Tier 4 and Tier 5 data centers. The only two members of our team who have physical access to the server racks locked within these cages are gentlemen with over 30 years of data center experience, one of whom used to work for the US Department of Defense and the other used to do cybersecurity work for the FBI. Rest assured, we take security very seriously and are incredibly cautious in our set up and maintenance of all nodes. We have recently signed lease agreements for more SOC2 data center space in Phoenix, AZ and in Amsterdam, and we hope to have both of these live with new servers by the end of Q4. This will put us on four continents, I believe, and we have aims to be on every continent at some point in 2024. P.S. - We are currently in the process of our own internal SOC2 audit for our team and its practices, and once we achieve this certification we will begin the process of applying for the more…

Excerpt (1198 of 1233 characters). Read the whole post on the forum ↗

How does this relate to validators running multiple Cosmos chains on one bare metal server? Are the same risks there, but in the context of one chain messing with others?

Hey I’m sorry – it doesn’t relate to that.

The risk is specifically to teams that are running on virtual machines that are shared with other organizations purchased from a cloud service.

Having multiple chains on a single bare metal machine is in my opinion equally safe as it is today… Except for what you just mentioned, one chain messing with the others.

My opinion here is that all of this really just underscores the need for validator teams to be vigilant.

Code review remains important, and this attack is basically terrifying. Personally I haven’t believed in the security of the cloud since rowhammer, and that was rather a long time ago now.

Hi,

I’m Keefer, founder of Tessellated. We use single tenancy bare metal servers to run our nodes, and our signing machines are located on hardware we own, backed by YubiHSM2s.

Awesome! I’ll add you to the list

I’m quite curious if a server can be compromised if it’s a self-hosted cloud. Like, in our case, we use Proxmox internally to split bigger services into a smaller containers/virtual machines, but these servers are only used by us and nobody else. I assume we should be safe, as we don’t have other people using the same underlying host as we do, but please let me know if I am wrong here.

We use 3 kinds of servers:

1. Proxmox instances hosted at home and split into containers - used for all validators
2. Hetzner virtual machines - for some public nodes
3. Hetzner dedicated servers - for other public nodes

Additionally, we do not store keys on validators themselves, we use TMKMS + YubiHSM2 for signing blocks on all validators, and it’s not hosted on VM at all, it’s a separate server.

I believe that this means that you’re okay but I am beginning to think that we should ask someone like in a formal manner.

This is no normal security exploit, it happens at the hardware level and so first of all it’s very hard to fix and secondly I think that it can imply things that probably haven’t been realized yet.

If you are the only user of those virtual machines that you described, and you own them then probably okay. Except they maybe can read stuff from each others ram, so that is super not ideal…

My biggest concern here is that in such setup, if one server is compromised, the others are as well. But to be honest, if someone can run an arbitrary code on your server, you have way bigger issues, and downfall is not the biggest issue here.

If you are the only user of those virtual machines that you described, and you own them then probably okay.

That’s pretty much it for us. Also I guess for us having a separate layer for HSM is also nice, as we do not store keys on servers, so basically there’s not much to steal.

Meanwhile, reading the news I realized that the patched Linux kernel is either in the works, or already have been released, so I think it’s worth it for anybody (doesn’t matter if it’s affected or not) to check whether your distro has released a patch and update, if yes. At least that’s my plan, we have a monitoring system that’ll alert us if there are upgrades we have not installed yet, and I highly recommend everybody to do the same, it really helps making your setup more secure.