Our understanding of the Cosmos Hub mempool issues
EDIT: @Rarma used his own homebrew BananaKing, not one he got from Jacob Gadikian Around 11/2/2023, @Rarma submitted a large volume of transactions to the Cosmos Hub using a shell script. I’ll refer to this as Someone Posting A lot of Messages, aka S.P.A.M. The transactions submitted were of the “BananaKing” type, IBC transfers with a long random string inserted into the memo field. This random string bulked up the transaction size. Due to gas pricing factors that I will get into further down, it also didn’t cost a huge amount of gas (only 2,000,000 gas units per tx). However, there doesn’t seem to be anything particularly malicious about “BananaKing” transactions. Many legitimate IBC transfer transactions could have large memo fields. A good example is a complicated workflow using Packet Forward Middleware and IBC hooks. The script ran for a short time, and got a very large number of transactions into the network. Over the next few days, a number of validators and full nodes struggled, with missed blocks and network saturation, and one of the S.P.A.M transactions got in every few minutes- even though the script stopped running days ago. I found this very interesting.…
Excerpt (1196 of 10043 characters). Read the whole post on the forum ↗
To underscore, there are architectural and design problems in the Cosmos fee system and mempool. These are well known and being worked on through a variety of efforts. Validators are expected to respond to S.P.A.M by adjusting their fee and mempool settings. The Comet team has been working all year on ways to reduce bandwidth usage and have some patches that should help (as I summarized here). Ultimately Cosmos needs a more sustainable fee system and mechanisms for building blocks and gossiping txs in app-specific ways.
Appreciate you highlighting the props we submitted to mitigate against this. Afaik Jacob has not given the script to anyone. The method Jacob has refined through weeks of research has far more severe consequences than what Rarma submitted onchain and most certainly not something we ever want to see tentatively tested on mainnet. There are a few ways to skin this cat as we have highlighted in the report we sent to relevant teams. 100% share you thoughts on a fee market.
One of the points I am trying to make is that most of what is good about the current mempool design. • Txs rarely get stuck in subgraphs that never reach a block proposer • Latency from random part of the node graph to proposer is generally <1 second. As you start harden the mempool against attacks, these properties diminish and basically the current design just becomes less and less usable. These scenarios seem to primarily consider situations where innocent nodes are recruited via RPC to participate in the attack because of the tx fee policies on their RPC. The scenario that worries me a lot is a fleet of bots that connect to the p2p layer rather than the rpc layer to inject txs into the networks and also try to trick node into connecting to them. The way to mitigate this is going to be tracking source ip addresses and rate limiting connection churn. These properties would have be correctly tuned. But if we eliminate the in node mempool concept, we have a lot more degrees of freedom in designing a secure and performant system that offers back pressure in terms of rising fees, doesn’t require every node to handle bursts of expensive to verify messages and ensure…
Excerpt (1197 of 1263 characters). Read the whole post on the forum ↗
jtremback: Many legitimate IBC transfer transactions could have large memo fields. Jehan, how about recieve addresses? Can legitimate IBC transactions have multi megabyte receive address fields? What has informal systems been doing since September 21 2023? Do you and @ebuchman agree with @zmilosevic when he says: Jacob, the issue you are talking about is a mempool/protocol fee design issue, so it is not security issue, it is a complex design issue that will take some time to be properly designed and implemented. And it is not comet only issue: it involved things to be designed and implemented at the level of the whole stack and then every application will need to also implement what makes sense to them. We have heard you when you surfaced it for the first time; at this point in time, there are workarounds that involve validators adjusting fees or someone proposing global feel to be non zero. I personally don’t believe that having non zero global feel is a solution for anything as validators can already adjust it locally and the benefit is that we can be more adaptive. Changing global params take weeks. You repeating the same thing every day will not change that…
Excerpt (1197 of 5903 characters). Read the whole post on the forum ↗
zaki_iqlusion:As you start harden the mempool against attacks, these properties diminish and basically the current design just becomes less and less usable.
That is absolutely correct.
Of course in order to know that, informal systems would have needed to spend less time making personal attacks against myself, such as claims that I’m unprofessional, and “a hysterical child”, and that my incident report was nothing more than a series of personal attacks against informal systems team members. If any informal systems team member is aware of any personal attack made by myself or anyone at notional, they should please let us know here in public.
But the honest truth is that they did not. Furthermore, and formal systems was in possession of numerous video and statistical data, as well as the full code to the attack, since the 25th of September 2023.
** PLEASE NOTE THAT THIS IS FACTUAL INFORMATION NOT A PERSONAL ATTACK**
I’m mostly looking at it from the perspective of trying to make sure that the network doesn’t struggle under a high volume of transactions. Limiting mempool gossip traffic by limiting the size of the mempool seems like a common sense step that I’m surprised nobody has suggested before.
Once you do this, the mempool could still be filled with crap but the nodes will be running fine.
With the addition of a fee market, it seems that any sustained attack will start to hit a limit as its transactions get in and are charged for gas. What’s the scenario that you’re imagining?
We studied small mempools in Istanbul. I’ve also studied them in 2020 times. We found it doesn’t really help. @joe-bowman will be familiar with the Istanbul experiments.
It helps if the only attack vector is via RPC methods on publicly available nodes.
Once you introduce " attack nodes" that stream large amounts of txs over p2p, those attack nodes are still able to introduce unstable subgraphs.
The only mitigations that I can think of rate limiting both amount of data a peer can send to you and the frequency with which a new peer can connect.
But there are lots of scenarios where for things like
- Node catch up
- A network self healing after a large/ high compute block that putting these limits in will slow recovery.
If you remember during the early eras of Osmosis, the entire p2p network would collapse and be rebuilt after the osmosis epoch.
If you start limiting connection rates and churn, you loose the the self healing function of the current design.
zaki_iqlusion:If you start limiting connection rates and churn, you loose the the self healing function of the current design.
This. But as you are aware, informal is 2.5 months behind and focused on blaming the reporter and banning them from the comet repo.
Proof:
@zaki_iqlusion quote=“jtremback, post:8, topic:12040”]
Once you do this, the mempool could still be filled with crap but the nodes will be running fine.
[/quote]
You should base nothing on the kangaroo attack. Please run the attack yourself. I have asked you this many many times, since the 25th of sept. please do it.
That is how you will understand it.
I do not know what @Rarma did. I know what I can do. These aren’t the same thing.
Please note this is not a personal attack (I need to include this in all informal comms now)
block gossip
Hi @jtremback @ebuchman I’m surprised that this analysis does not even touch on the issues with block gossip. Do you have an understanding of the issues with black gossip? In your opinion, do any such issues exist?
timeline
@jtremback @ebuchman @Jessysaurusrex
The timeline of this issue is incredibly important. Do you see any inaccuracies here? Please respond.
@zaki_iqlusion Is the timeline that I have presented accurate in your opinion?
Hey @jtremback , thanks for adressing this–and hopefully allowing open discussions/suggestions to take place to make CometBFT + Cosmos SDK more robust. Wanted to pitch in our 2uatoms here. Keplr team has had a lot of experience running node endpoints, and have frequently ran into issues where nodes would miss blocks at times when S.P.A.M. happens. While we haven’t done a full-on detailed investigation into this (as much as we’d love to, we don’t have sufficient resources on this side rn). We hypothesize that the blocks dropping issue is unrelated to the issues Jacob raised, nor block size, nor gas pricing, etc, but something more fundamental. In the words of my co-founder (since he’s the technical one not me–and pardon if i mess up the technical nuance in translation), he mentioned that it seems likely that because ABCI is not multithreaded and goes through running the checkTx() sequentially, checkTx() takes up all the process when processing a massive amount of transactions. This leads to issues where important ABCIs such as beginBlock(), endBlock(), deliverTx(), commit(), (or even potentially other p2p comms) is stalled until the backlog of massive checkTx() has…
Excerpt (1190 of 2420 characters). Read the whole post on the forum ↗
jtremback: EDIT: @Rarma used his own homebrew BananaKing, not one he got from Jacob Gadikian If he had used the attack methods that I have developed the hub would have stopped. I did not give them to @Rarma . How sad that it took a real attack on mainnet to get informals attention, when i have been taking down testnet for months. This isn’t stewardship, it borders on sabotage and is without a doubt gross negligence. No one should blame the most responsive person at informal though ( @jtremback ) Blame should rightly be placed on the executives who have been denying the entire thing while falsely claiming to be being harassed and simultaneously describing myself as a hysterical child and kindergartner (actual harassment) So, specifically I am naming @ebuchman and @zmilosevic . Furthermore, blame should be placed on the security contractor who allowed a universal chain halt to be published, @Jessysaurusrex . I would love to have a purely technical conversation, but unfortunately, those conversations have been deliberately stopped by the above named executives. @crainbf sir, I hope you take appropriate action. All of cosmos owes @Rarma some thanks.…
Excerpt (1195 of 1951 characters). Read the whole post on the forum ↗
The title refers to mempool issues in the cosmos hub. This is untrue. These are global and universal issues for every chain that uses comet.
dogemos:This leads to issues where important ABCIs such as beginBlock(), endBlock(), deliverTx(), commit(), (or even potentially other p2p comms) is stalled until the backlog of massive checkTx() has completed.
Two things related to this:
- Did you have pruning enabled in those nodes? Pruning can affect the amount of time a node spends holding the ABCI lock (in commit if I recall correctly) and delay block production. Injective disabled pruning for this reason in some of their nodes.
- More importantly: In Comet we added a fine-grained locking that will be shipped in v1. This allows higher levels of parallelism across ABCI calls. dYdX already using fine-grained locks, and I suspect Osmosis also. We can investigate backporting this to older version of Comet – and the SDK would need to absorb these changes – but if there’s interest this is a good way forward.
Forgot to say also, as someone mentioned earlier here, NewMetric team have been investigating parallelism across ABCI queries and we’ve been chatting with them to learn from their experience. Seems like the direction with fine-grained locking is appropriate. One problem left that we’re not sure yet how to approach (but this is getting off-topic) is that CheckTx is considered intrusive in the way it requires lock holding at any moment during block lifecyle (whenever a tx arrives). This problem overlaps with mempool architecture, which is where ADR 110 Remote mempool comes into play, following recommendations from community.
IMHO there being a lock itself is a problem, no matter how efficient and practical the state tree is via pruning. Technically and at least logically, CheckTx() doesn’t need to be locked because it’s really just doing 2 things - dry-running the transaction to see if the transaction makes sense, and putting into CList.
Dry-run to check validity of a transaction doesn’t need lock; it is checked against a previous state root + merge-sorted in-memory CacheKV which resides in the app side. Therefore it could technically be an Async call where response is eventually routed back to Mempool.
One thing we get by locking is serializability between Txns in the order they were received. But AFAIK this isn’t the goal of CometBFT, neither does it make sense – the transaction order received at any validator’s mempool is never guaranteed to be ordered in the first place.
Given the lock could be lifted in Mempool related ABCI functions, another interesting lock point is mutex in CList. I believe this can be removed too; we could just use a simple lockless RingBuffer with in-place tombstones for Reap. We are doing that internally for our product (along with experimental lockless CheckTx…) this seems to work great, as long as the RingBuffer is big enough to handle incoming Txns.
The scenario that @dogemos mentioned (which I’m too familiar with 
) is very true, and often renders as the single point of failure for liveness goal. This is also true for any queries as most of them are routed through QueryABCI.
Fine-grained Lock is great, but I think most of the control should just be moved to Application for methods that aren’t related to consensus-critical serializability. This IMO leaves BeginBlocker, DeliverTx, EndBlocker, Commit and Proposal related methods.
btw: ADR110 rocks
This post frames a block size increase as strictly positive. In order to enable voters on proposal#845 to make an informed decision, can this post please also document the downsides associated with a block size increase?
Hey, @rootulp – to my knowledge there aren’t any downsides to a block size increase, until the block size reaches ~5mb, at which point there could be issues propagating the block to peers due to issues with block gossip.
That said, there’s a bigger issue at play, which came up during the testing that Notional did on the replicated security testnet:
- 200kb and 21mb perfomed about the same, for different reasons
- with 200kb blocks it is much easier to fill the mempool and cause it to freak out.
- with 21mb blocks it is much easier to get the block gossip mechanism to freak out
1mb was much better than 200kb
2mb was a bit better than 1mb
jtremback:The transactions submitted were of the “BananaKing” type, IBC transfers with a long random string inserted into the memo field. This random string bulked up the transaction size. Due to gas pricing factors that I will get into further down, it also didn’t cost a huge amount of gas (only 2,000,000 gas units per tx).
Banana Kings use the receiver address field.
Hey yea, I’m not completely 100% on the block size increase myself. That being said, it is a very standard size in the blockchain world, and I am ideologically a “big blocker”. I also have a feeling (kind of the whole point of this thread) that block size and mempool size must be tuned together. Once our tests complete, we’ll be able to make a stronger recommendation on the block size change.
Was this a Banana Queen?
Hey, great to have someone with a ton of real world RPC experience in here!
In this analysis, I’m mostly looking at mempool gossip performance as a possible culprit. I think that improving mempool performance could be a win by itself. During a S.P.A.M. event, we’ve observed a lot of nodes suffering, not just the RPC used.
That being said, the multithreading suggestion is a good one. I’m actually not sure where that needs to happen- Comet or Cosmos-SDK. Maybe both. This highlights another tricky part of these performance tuning questions: they can involve a lot of libraries maintained by different teams.
The fact that there are stateful checks in the p2p hot loop via checktx are somewhat inherently a liveness vulnerability.
@kjessec and @dogemos are right to emphasize the risk from sequential stateful checks .
Once we start building under the adr 110 model, these stateful checks can be scaled horizontally in a way they can’t be right now.
I would like first to thank everyone for contributing their understanding to this very important topic and offering some ideas how things can be improved. Let me try to offer my perspective: • I agree with @zaki_iqlusion that the nice properties of the existing system is that it is very robust and that messages get propagated through the network pretty fast. In fact, a part of the problem is that the current design is probably too robust. We have actually been doing some research back in 2020, as it was clear even back then that this is probably the most challenging and least mature part of the tendermint/comet. And we at Informal, are clearly not the only ones being aware of this, zaki_iqlusion, jack, dev, ismail, xla, chris goes, etc, have been also talking about this for a very long time. It turns out that, although there is a ton of research on gossip and consensus based systems, there is almost no research on the mix of the two, i.e., consensus systems (includes tx gossiping, aka mempool, also vote and block gossiping, not just consensus protocol) on top of gossip systems. The closest research in our view was the line of papers around BAR (Byzantine, Altruistic, Rational)…
Excerpt (1198 of 4710 characters). Read the whole post on the forum ↗
zarko_informal:Since this was raised in terms of a security incident, we (Informal) wanted to keep a low comms profile while working on it (the right, professional thing to do), and that might have been misinterpreted as not considering the issue important and not working hard on it. We in fact, deeply care about it and have been working on this for a very long time (not just in the last two months).
Informal worked with amulet to release this information to the whole world and that is the opposite of a low comms profile
zarko_informal:The other aspect here is that comet’s existing architecture and implementation does not allow us to easily implement any novel protocol/design idea, especially that involves changing the gossip/p2p layer. The existential challenges we faced in 2022 with tendermint (before Informal took over stewardship of the comet project) were actually related to this exact problem, and hopefully we all learned that making changes in tendermint/comet gossip/p2p layer need to be done with a lot of understanding and care, and with super thorough QA process.
That isn’t what you said.
You said there was no security issue. These are radically different things.
AFAIK, one of the stateful checks is on the sending account’s balance, right? It seems that this would need to continue being single threaded.
Not just balance, pubkey, nonce… whatever is related to checking the authority of the sender. One way to optimistically parallelize is to shard CheckTx runs by sender address. Biggest state-dependent part in verifying the authority of the sender is nonce check, and fee deduction. All of these can be serialized if we maintain a per-sender lane for these. But other parts - like the “body” of a Txn - is rather interdependent across other transactions, due to the validity of a transaction largely depends on the global view of the canonical state… How about we don’t perform CheckTx at all at proposer’s mempool? May sound a little drastic, but I think it makes sense: • Computationally, CheckTx cost is at most as expensive as normal DeliverTx run. This means that validator runs every single transaction at least 2 times. All transaction runs, including CheckTx, do make use of global mutex lock, and this is one of the biggest performance upper-bound. All Txns will get run in DeliverTx call anyways - I think the notion of “Checking the transaction before inclusion” is only adding unnecessary computational load. Should a transaction fail, it will fail in DeliverTx call anyways.…
Excerpt (1198 of 2824 characters). Read the whole post on the forum ↗
I’m somewhat new to this subject, but one important role of CheckTx as I understand it is to flush the mempool of a spammer’s txs when they run out of gas. This is especially powerful with the addition of a fee market, as they will run out of gas much more quickly.
Eliminating CheckTx entirely would make it so that this flushing wouldn’t happen. Seems like it would result in blocks full of failed Tx’s.
Multithreading CheckTX would delay the flushing by a block (the spammer would have to get their txs in and have gas deducted for real before the check started failing). In this “last spam block” you would have a bunch of failed Tx’s.
Maybe it would be possible multithread the majority of the code in CheckTx, but have a special case with a shared data structure or whatever between threads just for gas. Seems tricky though. We’re getting into the domain of Cosmos-SDK here though, maybe @marbar can chime in.
However, I am excited to see how just reducing the mempool size will work. This will reduce not only network usage gossiping but also CheckTx load (I think).
All people who feel underheard, undervalued, misunderstood, dismissed, have their fight or flight mode activated. Add a dangerous element & it’s 10x worse.
How many tropes have been portrayed in films & TV & literature about somebody sounding an alarm and being repeatedly put off or waved away?
That’s why it’s such bad practice to do that -
Everybody loses.
More apologies all around. Give benefit of the doubt to get benefit of the doubt.
If you think you’re owed an apology, lead by example. Forgive by example too. Nobody is a villain in their own mind.
No more combat. No more historical justifying.
No more exclusive high table bs -and-
no more mud on the work table.
Make good processes in lieu of bad dramas.
Somebody said the Hub is paying way too much for security. What it’s been getting is a social engineer’s Swiss cheese dream target.
– Thank you for your patience during this unscheduled maintenance. –