[ mintscan link ] Written by Sam Hart (Head of Product & Strategy) and Maghnus Mareneck (Co-Founder) of Skip Protocol Background The Cosmos Hub continues to grow as a central trust layer for the AEZ and wider Cosmos. With the expansion of ICS and the addition of Neutron and Stride, the Hub has increased responsibility and importance in the security and extensibility that it can offer to the interchain. That said, some of the Hub’s functionality is still based on dated Cosmos chain design formulated just as IBC was coming online. In particular, the Hub uses a first-come-first-serve , monolithic, and generalized mempool, and does not take advantage of any of the new Cosmos SDK advancements including ABCI++, vote extensions, or enhanced block-production capabilities that Skip has deployed widely across the interchain. This has nothing to do with MEV The Hub, like Bitcoin, currently has very-little-to-no MEV (we checked). This is because Hub transactions are mostly transfers, staking events, and light client updates. This is likely how it will stay, and so nothing in this proposal suggests introducing a MEV recapture or distribution mechanism. Rather, it focuses on…

Excerpt (1193 of 7795 characters). Read the whole post on the forum ↗

Will be the easiest yes vote in quite a while. Exciting!

This is an awesome idea. Certainly in favor of it! Glad to see more developments to improve ATOM tokenomics and Hub sustainability.

easy yes. Congratulations Atom community

Seems like a no brainer to me.

Interesting. Seems an easy ‘yes’

I think that this is a important and well reasoned proposal.

The current way that we do fees adds additional danger to the systems that we are building and does not allow for demand pricing.

Notional supports this proposal and because we feel that it is an important security improvement to the stack, we are happy to help in implementation and testing.

In fact, given recent research into the fee systems used in cosmos, I think I need to say that I think that after some review and testing, it likely makes sense to attempt to upstream this into the SDK itself because it makes the systems that we build safer and more reliable.

Great team, even better product - YES!

I fully support this <3

Frankly this proposal is a very good idea and doesn’t require any refining. Even though we believe that its implementation isn’t necessarily an immediate requirement. This is still a problem that will likely grow over time and we also believe that having a proactive approach is always a good solution. Let’s profit from the current calm environment. We will also be happy to help in testing.

On behalf of the PRO Delegators’ validator, we will greatly support this proposal. Thank you for posting it

Hey, this sounds really great. The lack of a sophisticated fee market in Cosmos has bothered me for a long time.

We just put out this suggested “CHIPs” process for Cosmos Hub improvements: Cosmos Hub Improvement Process (CHIPs)

Not to hijack your post, but I was wondering what CHIPs phase you consider this proposal to be at (and whether CHIPs even looks like a good process to you)? To me, it seems like we are at phase 4: signaling proposal, since the design and implementation work has already been done.

Maintenance

Either way, my main concern here is maintenance. I have the following questions:

• How will ongoing maintenance (SDK version upgrades, etc) be paid for? Who do you expect will do it? Will the code always be owned by Skip, or will responsibility pass over to the SDK or Comet teams at some point?
• In the scenario that maintenance stops for whatever reason, what will the impact on the Cosmos Hub be?

hxrts: There is no way for clients to know what fees validators have set locally Actually, there’s a way that has been added recently to Gaia github.com/cosmos/gaia feat: add query local min-gas-price `cosmos:main` ← `cosmos:feat-query-mingas` opened 01:34PM - 27 Jun 23 UTC yaruwangway +14 -0 <!-- The production pull request template is for types feat, fix, deps, or refa … ctor. --> ## Description Closes: #2629 <!-- Add a description of the changes that this PR introduces and the files that are the most critical to review. --> --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... * [x] Included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title * [ ] Added `!` to the type prefix if API, client, or state breaking change (i.e., requires minor or major version bump) * [x] Targeted the correct branch (see [PR Targeting](https://github.com/cosmos/gaia/blob/main/CONTRIBUTING.md#pr-targeting)) * [ ]…

Excerpt (1197 of 2747 characters). Read the whole post on the forum ↗

that’s querying the grpc node run by polkachu, but is not querying the network itself.

It’s a local node parameter, so querying a specific node makes sense no?

not really. It’s supposed to be global, the hub uses x/globalfee to enforce a consistent fee level. rumor has it, everything except for: github.com cosmos/gaia/blob/0b6650c44a76fe05ae32b9f046e4f383e644cccd/x/globalfee/types/params.go#L24-L33 ``` // DefaultMinGasPrices is set at runtime to the staking token with zero amount i.e. "0uatom" // see DefaultZeroGlobalFee method in gaia/x/globalfee/ante/fee.go. DefaultMinGasPrices = sdk.DecCoins{} DefaultBypassMinFeeMsgTypes = []string{ sdk.MsgTypeURL(&ibcchanneltypes.MsgRecvPacket{}), sdk.MsgTypeURL(&ibcchanneltypes.MsgAcknowledgement{}), sdk.MsgTypeURL(&ibcclienttypes.MsgUpdateClient{}), sdk.MsgTypeURL(&ibcchanneltypes.MsgTimeout{}), sdk.MsgTypeURL(&ibcchanneltypes.MsgTimeoutOnClose{}), } ``` …is priced at 0.0025uatom because of x/globalfee – however if we look at blocks produced by certain validators, we can easily find that to not be the case. I’ll watch the block log and find an example of this for you: mintscan.io Mintscan…

Excerpt (1190 of 1388 characters). Read the whole post on the forum ↗

hxrts: • The “base fee” is a variable , dynamic fee that adjusts with demand (measured by percentage of gas limit used). This is required for inclusion, and is easy to calculate and expose to wallets from any node on the network. • The “priority fee” is an optional tip that users can add on top of the base fee to receive priority execution. Paying a higher priority fee than another transaction will ensure that the higher paying tx will be included first. We don’t anticipate priority fees being very relevant for the Hub until volume scales. What is quoted above I am in support of; having a consensus agreed base fee, and an optional tip, is perfect. My objection is in using ABCI++ as it was described when first announced. Unless I’m missing something, ABCI++ helps with prioritizing transactions that were already reaped from the mempool, but what we need is a way to customize the mempool to literally create new lanes each with their own underlying MConnection channel. If a lane doesn’t have its own MConnection channel, it doesn’t make any sense to me. If ABCI++ doesn’t support creating new lanes, I suggest we create a new message that allows for ABCI app →…

Excerpt (1191 of 1975 characters). Read the whole post on the forum ↗

This requires every validator to expose this endpoint publicly and clients to query all validators to see their gas prices each time they send a transaction. I think we can agree this is suboptimal.

Hi Jae, thanks for your reply. Cosmos SDK 0.47+ has an app-side mempool, so what’s proposed here is really an augmentation of the way the mempool works in mainline SDK, modifying this system to use an in-protocol base-fee. I see the decision about where the mempool lives as largely out of Skip’s hands. My understanding is 0.47 is planned work here: EPIC: Refactor to SDK v0.47 and IBC v7 · Issue #2540 · cosmos/gaia · GitHub

The migration to SDK 47 is pretty much done. We are currently waiting on the LSM’s upgrade and we should be able to deploy in v15.

@jtremback just to address your point above we plan to maintain the Block SDK for the Hub and work closely with Hub development teams. We’re also speaking with Binary Builders about a long-term working relationship with the Cosmos SDK team around Block SDK support given the number of chains in the ecosystem that are now planning to adopt it.

Also if the proposal does pass we can slot in the integration in whichever release makes sense for your team. The Block SDK is exceedingly simple to integrate and we can make the PR.

We’ll be putting this up shortly. Just wrapping up our team off-site, so wanted to wait till we could give this proposal our full attention.

Thank you for the additional information @hxrts

Adopting an EIP-1559 mempool is a mistake for the Cosmos Hub and Osmosis chains.

It makes absolutely no sense to constrain what fee the next block proposer must charge for transactions based on how full the previous block is.

In a Cosmos style blockchain, we need to treat each block as a unique auction that occurs within the next proposer.

It doesn’t matter if the previous proposer proposed a lot of transactions or a few transactions.

It is a mistake to treat previous blocks as a view of congestion in the network.

This is the wrong design and will result in worsening UX for Cosmos.

I think this could be accurate.

Since we have full finality, each block is fully independent of the others, especially if we do not have a mempool, aka satoshis mind prison.

The implementation of the “FeeMarket” module we have does not necessarily mandate including previous block capacity as a parameter, and doesn’t mandate copying EIP-1559. If we find previous block capacity is not a good signal (though I think in periods of high load like today on Osmosis, it’s hard for me to argue it wasn’t), then we can hot-swap other parameters. EIP-1559 would be a general improvement on the fee market today, and we’d love to be engaged on finding yet better ones over time that are dependent on other factors. Finality does not seem to change the calculus of why EIP-1559 is worse for Cosmos than it is for Ethereum. In Ethereum, even though finality takes longer, block-by-block base fees are still independent, as they would be in this implementation. However, this version of EIP-1559 allows for adjustable learning rates so we could scale up/down how much we index on block size or capacity. If we wanted a pure PGA on the Hub, it can support this too. The module is a general framework to build stateful-aware fee markets, and dynamically adjust learning parameters on a set of inputs - i.e. can create any fee market. With the Block SDK wrapping the module, we can…

Excerpt (1197 of 1487 characters). Read the whole post on the forum ↗

Here are my assumption.

1. The current Cosmos mempool is riddled with extensive security flaws and will not be a part of Cosmos in near term.

2. The idea of a widely gossiped mempool makes no sense and is not part of any successful blockchains block proposal process.

3. There are a bunch of flaw in post proof of stake Ethereum related EIP-1559. The reality is 50% of Ethereum block space is DEX swaps and CEX/DEX arbitrage is a big consumer of Ethereum block space .

zaki_iqlusion:

There are a bunch of flaw in post proof of stake Ethereum related EIP-1559.

What are the flaws? The change to proof-of-stake on Ethereum happened over a year ago and the implementation of EIP-1559 went live over two years ago, and all is well as far as I can tell. I ask sincerely since I’m 100% sure that you know more about the subject than I do.

What kind of security flaws which current cosmos mempool have?

Could you give reasoning for it?

Security flaw of the current mempool design.

The current mempool allow malicious nodes to join the network and flood their peers with transactions until they become unsynched from the network. Peers are allowed to consume unlimited compute and bandwidth by spamming nodes.

EIP-1559 was designed for a proof of work system where it was unpredictable who would propose the next block. Once Ethereum moved to proof of stake, it made a number of attacks cheaper and interacts problematic ways with the rest of the protocol. There are many documented problems with EIP-1559. It enables a bunch of cheap censorship attacks. For instance, it might be in the interest of one malicious proposer to increase the costs of the next proposer to include transactions by filling prior blocks with txs. There is really zero reason to believe that blocks being full represents real user demand not value extraction opportunities by validators. Generally Cosmos validators tend to be more altruist and less rational than Ethereum validators but there is a lot of potential problems. here are some additional references about problems that emerge once EIP-1559 was applied in a proof of stake setting. Medium – 17 Jan 22 Flaws in Ethereum’s EIP-1559 EIP-1559 added a mechanism for the Ethereum network to establish a base transaction gas price that dynamically changes depending on the… Reading time: 18 min read…

Excerpt (1154 of 2769 characters). Read the whole post on the forum ↗

zaki_iqlusion:

The current mempool allow malicious nodes to join the network and flood their peers with transactions until they become unsynched from the network.

Why it is a flaw? All txs must have tx fee. Even if there is a flood of txs, it doesn’t matter because they will pay the fee.
Txs without tx fee will be dropped by peers automatically.

It could be a flaw only if current mempool accepts FREE txs (no tx fee included).

There is a lot to be said about the references you posted here, some of the analysis hinges on control over the gas limit (in the first reference it also incorrectly assumes that you can change it without changing the gas target), which is not at all a requirement of EIP-1559 anyways. To me, the most relevant flaws of EIP-1559 are the following: • The mechanism can be defeated by (smart contract) coordination. This is pretty much true of any other mechanism, and I think the details of how this is done matter quite a bit to think through defences. • Not giving part of the fee to the block producer (BP) mechanically lowers censorship-resistance, as the value for the BP to include a transaction is no longer equal to the fee-value of its inclusion, so a censor only needs to promise to pay the partial fee received by the BP + epsilon to convince the BP to leave out a transaction. Note that the censor would actually need to pay the partial fee + the MEV induced by the transaction + epsilon, in a fully rational model and without a mechanism for mev-capture. • That being said, I am actually not convinced that the fee market should be the place to enforce censorship-resistance, since…

Excerpt (1199 of 4969 characters). Read the whole post on the forum ↗

@barnabemonnot ’s Cosmos debut! Thanks so much for writing this up. Also for anyone unaware, Barnabé is one of the people at the Ethereum Foundation who led the EIP-1559 research effort. He has published extensively on this topic, but I’ll post a short primer he wrote a while back that may be helpful for some here. Barnabé’s comment addresses Zaki’s criticism about validator’s ability to manipulate the base fee and/or block size via consecutive proposals (block size and transaction fee are dynamically coupled in EIP-1559). To summarize, it is possible to manipulate, however it is also costly to do so. EIP-1559 is not a wholistic solution to blockspace allocation, rather it introduces a commitment to pay on inclusion for anyone submitting a transaction. Now Zaki also raises a second issue regarding the mempool architecture itself, which is really an orthogonal concern. The fee market aims to price transaction inclusion. The mempool is a distribution system for candidate transactions run by the validators. Today this gossip network is part of Comet, which passes data to the Cosmos SDK and Block SDK module for processing. The Comet gossip system is probably the most cursed part…

Excerpt (1197 of 1963 characters). Read the whole post on the forum ↗

i love this discussion even if i understand 1/10 of it.

glad Barnabe came here btw!

@Elijah would be nice to have your (multiplicity’s) point of view too

Great discussion, and thanks @ barnabemonnot for chiming in - great to see you here! Just wanted to share some notes on the Comet mempool and try to decouple a few things. The mempool does have a very inefficient flooding (“push”) protocol to gossip txs, and neither the mempool nor the underlying peer connection degrade gracefully. However the expectation is that spam prevention is handled at the app-layer by tx fees. The mempool’s interface to this is the CheckTx method of ABCI, which allows the application to accept or reject txs into the mempool (ie. by checking for fees, signature, etc). Only if a tx is accepted into the mempool by the application does the mempool start gossipping it. This is not to say there’s no problems with the mempool design, but ultimately entry into the mempool is gated by the app’s fee system, and a fee system that can adequately modulate entrance into the mempool has been a gaping hole in the ecosystem forever. This is compounded further by a desire to accept low or zero fee IBC txs to improve UX and lower burden on relayers. Currently validators set subjective min fees in their local config, and increase them / restart nodes as necessary to…

Excerpt (1195 of 3814 characters). Read the whole post on the forum ↗

agree with zaki, look forward to seeing his proposal

The challenge here is that the Cosmos networking stack doesn’t actually have a concept of “validators”. It only knows about “nodes”.

Nodes then implement policies about what they propagate to other nodes but the network doesn’t know about where the next block proposal is gonna come from.

So a tx is ingested into the the network at the first node, floods through some number of nodes with unknown filtering policies and then eventually finds itself in the block proposing node which also has opaque policies. This process is subject to zero discrimination and spam, low value txs contention.

A user only gets feedback from the first node in this chain and that feedback is essential irrelevant.

Once you start introducing adaptive behavior into this architecture, the net result is that QoS starts to degrade in completely opaque ways.

Now an on chain adaptive oracle is bad because it’s constantly lagging.

Having your network reacting seconds later to issues will create the experience of profoundly unreliable system.

A design goal is have to have systems that can provide back pressure, and credible commitments to inclusion in <100 ms and provide either directly to a client.

To summarize the first half of Zaki’s argument: • The Comet network layer does not innately distinguish between validators and any other node • The transaction propagation policy of every node is discretionary and private • The gestalt behavior of such a system is highly opaque and vulnerable to attack We are 100% aligned on these points. It’s a miracle this system has worked up until now. Everyone agrees the whole transaction ingress system needs to be replaced. The Skip Sentinel was such a bypass mechanism, and the in-protocol oracle system we are working on likewise provides a side-car to bypass Comet transaction intake. We have, and will continue to build systems that provide more robust ways to ingest transactions. The part of Zaki’s argument I disagree with is that the networking layer + the EIP-1559 fee controller will have a reinforcing interaction that will further degrade the system. This argument hinges on a very nuanced premise that the timescale on which the fee controller reacts to information is of a different order than what the gossip system exposes to the validator. Let’s break down the possible scenarios to help with our intuition. So under normal…

Excerpt (1197 of 4158 characters). Read the whole post on the forum ↗

We’re sorry we couldn’t provide feedback on this proposal before having it on-chain. This mempool architecture discussion was much needed indeed. As we are late to the party, we will skip on our regular 3-step breakdown (context, analysis & solution). If we had to share our opinion on the matter, this is probably the best quote we would like to take out of this entire discussion: hxrts: Full DDOS prevention will require addressing both the fee market and transaction intake system, so it’s important to note that the addition of EIP-1559 is only a partial mitigation, and an incremental step toward making the Hub more secure. We are entirely aligned with this analysis. CONCLUSION: We think the best strategy to overcome the situation has to be set in a two step process. One adressing the base fee to prevent the basic direct DDOS attack via economic measures. The second should be addressed directly at the SDK layer regarding the Comet gossip system which clearly needs a refresher. There seems to be very interesting paths explored in this thread and we think @ebuchman have raised an elegant longer term solution by making the mempool more “pull” based. We also…

Excerpt (1196 of 1553 characters). Read the whole post on the forum ↗

Govmos: we think @ebuchman have raised an elegant longer term solution by making the mempool more “pull” based. We also believe this is the most suitable strategy for a durable cure. In the meantime, if development takes a longer toll than expected, then it would be worth exploring more short term fix by simply limiting the number of peers to gossip mempool transactions. This would at least propose a temporary solution until the more sophisticated solution is implemented. You think wrong. Notional did the research, learned of the CAT mempool by @cmwaters from @valardragon , and made the first PR to comet to adopt cat. @AdiSeredinschi told us that CAT was non viable, and bucky – as well as all of informal, was fully disengaged from the process of solutions engineering for the p2p storms issue. The only contribution Informal made here was slowing down the fix and creating horrific vibes. Proof: • pull celestia/cmwaters CAT mempool implementation into cometbft by joe-bowman · Pull Request #1426 · cometbft/cometbft · GitHub • cat by faddat · Pull Request #1429 · cometbft/cometbft · GitHub Our intention was to get CAT into the v0.34.x branch as rapidly as…

Excerpt (1192 of 1773 characters). Read the whole post on the forum ↗

I don’see why we shouldn’t give this a try, it is an improvement compared to the current fee market. YES!

Thank you Jacob for the answer and the appropriate rectifications. We couldn’t find the time to do the deep down analysis we usually do. We clearly acknowledge that we didn’t have all the necessary depth for this particular matter. You provided some very worthy points here with all the verifiable sources. Thanks a lot for this.

Regarding the political implications, we don’t want to position ourselves one way or the other. We value both Notional and Informal, all we could say is that the Hub need you both. Hopefully someday these deeply rooted tensions will be a thing of the past.

Govmos:

Thank you Jacob for the answer and the appropriate rectifications. We couldn’t find the time to do the deep down analysis we usually do. We clearly acknowledge that we didn’t have all the necessary depth for this particular matter. You provided some very worthy points here with all the verifiable sources. Thanks a lot for this.

That’s not your fault at all.

I doubt informal will ever tell the truth on these matters.

I value some people at informal, and hold some people at informal in the highest regard.

Sadly their org has failed them and the cosmos.

Just want to add some substance to a couple of claims above: made the first PR to comet to adopt cat Grateful for the PRs because we’ve missed feedback on CAT mempool until this P2P issue was surfaced. It validates some of the work we’ve been doing to reduce bandwidth consumption, and makes the team confident we should continue investing time in it, thanks for that! @AdiSeredinschi told us that CAT was non viable I said the Comet team in Informal has reservations about the code. It’s not ready to ship, period. Not in `main` and particularly not in 0.34. The team has been doing experiments with mempool (CAT and otherwise) since this summer. The reason for these experiments is because we want to reduce bandwidth consumption in Comet, and the CAT (push/pull) design seems elegant for that purpose. For example, this branch in August has some of the earliest CAT mempool ported code. No PR was opened because we found problems with the ported implementation and were not able to replicate findings from earlier experiments with CAT. Think of that as failed experiments, of which there are many hehe (some tracked here ). Since then, we fixed some things & opened another PR…

Excerpt (1199 of 2691 characters). Read the whole post on the forum ↗

Thanks for chipping in man. Glad to see you here. DMd you (well iw as going to, but dont seem to find how anymore on the forum - how can i contact you outside the forum?

AdiSeredinschi:

Grateful for the PRs because we’ve missed feedback on CAT mempool until this P2P issue was surfaced. It validates some of the work we’ve been doing to reduce bandwidth consumption, and makes the team confident we should continue investing time in it, thanks for that!

Sir, your team has been saying they’re going to ban me from contributing to comet.

I’m sorry that does not sound even remotely like gratitude sir.

I urge everybody to remove all funding from informal systems in perpetuity.

Sir, your team has been saying they’re going to ban me from contributing to comet.

You’re talking off-topic again. Stop.

I stand by what we wrote: continued spamming, harassment or abuse of our team, or off-topic comments, will result in you being blocked from GitHub repositories. It should be the same with this forum BTW.

It doesn’t mean there’s no value to your technical contribution/feedback, I remain grateful for that.

Block SDK MEV extraction will definitely be extremely profitable once core DeFi primitives get deployed on chain.