[on-chain proposal] - Cosmos Hub IBC relayer gas cost restitution plan - FeeGrants

Dear community and validators, This forum post is a request for feedback and Proposal draft seeking community funding for a relayer fee-grant support system for IBC traffic on the CosmosHub. Abstract: This governance proposal addresses the urgent need for supporting IBC relayers on the Cosmos Hub following the recent gas fee increase. It proposes the establishment of a fee-grant support system, funded by the community, to cover gas fees for IBC relaying activities. The proposal outlines the critical role of relayers in maintaining the network’s interoperability and the negative impact of increased operational costs on their sustainability. The proposing team consisting of respected Cosmos Hub relayers suggests forming a multisig group to manage the distribution of fee-grants to vetted relayers, offering a temporary solution to prevent service degradation and ensure the continuity of efficient IBC operations on the Cosmos Hub. What is a relayer and IBC: The Cosmos ecosystem is known for its interoperability technology dubbed, the Inter-Blockchain Communication protocol (IBC) that uses light client verification systems to allow for trust-minimized asset and message…

Excerpt (1191 of 19144 characters). Read the whole post on the forum ↗

So the choice is between “cover the ibc gas fees” or “no IBC on the Hub”.

What are we waiting for? Let’s go on chain and approve this!

Of course I like that the relayers are not trying to cover their ops costs or profits (defo cudos for this and thank you for your service) + that this is a 3 month time bound prop.

I do believe that in-protocol incentivization is probably a few months away from full, wide spread adoption. So hopefully, we will only need to renew this FeeGrant agreement just once or twice.

This sounds good to me. support

Strong no. Instead of reducing inflation, now validators ask for additional payment, covering a service, with which relayers actively try to capture delegates from ICF and by users. This is also not the first time validators have received payments for relaying, so this is no solution to the underlying problem. In my opinion relaying is a wallet / UX issue. Instead of relying on validators to relay, users should be able to relay their transaction themselves, or having an open bid market. Stop centralizing power around validators, it is already bad that all decisions are made by a small group of validators. Giving relaying power to a selected few just centralizes the issue further essentially allowing them to put proposals like this up: “cover the ibc gas fees” or “no IBC on the Hub” (which is not true btw.) Also: Why should Cosmos Governance incentivize bot traffic? Let the bot operators pay for themselves. Please provide an analysis on the distribution of wallets for the relaying fees. I expect only a small amount of fees is generated by normal users. Give us some data where the fees are coming from. Transaction costs are one of the revenues for the network and this prop…

Excerpt (1197 of 1657 characters). Read the whole post on the forum ↗

Please be aware that relayers are NOT being paid, they are merely asking for the pool to cover the gas fees that are involved in relaying. All infrastructure (full nodes + relayer machines), time spend, support and other activities are still costs incurred by the relayer operator. A choice they make to indeed, as you say, strengthen their brand and attract delegations. as to your question about where the TXs come from, an over majority is Transfers between Osmosis and the hub and ICA to Stride. There will probably be arbitrage bots within that set of users but they too are just using the network and delivering a service. They will still have to pay the Gas to initiate their own IBC transaction. And just to be clear, we are not stating that IBC will stop on the hub if this proposal does not pass. I think it is likely to see a reduced service level though as we see already. I agree that self-relaying can be a good solution, but is that in the room with us right now? And just to re-iterate, everyone is welcome to get a feegrant as the above draft clearly stated. This is exactly the reason why we did NOT go with a RFP like proposal for now, as that locks out all relaying to a…

Excerpt (1198 of 1403 characters). Read the whole post on the forum ↗

I see this a very well made proposal and the matter is urgent but the cure to pay costs for the relaying validators is just short-term revival. Cosmos core should brainstorm a sustainable solution for one of the major usecase of the Hub, IBC. None of the relayers should run the infrasfructure with an loss and quit validating bc it comes unsustainable, none of the users should suffer from longer transactions times and feel stucked or timeouts.

… wonder how much mid-high txs would cost if the users pays the costs fully instead of relayers and what if these paid fees would then come back as staking rewards

Let’s do this. Witval will support the proposal

Very supportive of this, IBC is the backbone of the ecosystem and that we still don’t have users actually paying their IBC costs is insane to me. @Syed is there any plan from informal to address this effectively?

Susannah | IBC at Interchain

Thank you for the thoughtful write up, whilst I agree that the current approach to funding relayers with off-chain agreements and delegations is unsustainable, my main concern with the proposal is as you highlighted: This proposal will most likely remove some of this competition and “free market” principles for relayer operators, that is something voters should take into consideration for better or worse. Relayers ensure liveness of the network and timely packet delivery. Putting certain relayers in a privileged position to have their relaying fees funded whereas other relayers would not be subsidised does not seem like the ideal solution and seems somewhat unfair. I understand that you would want to avoid subsidising malicious transactions that are simply to drain the allocated amount for the feegrant but it would be good to clarify in more detail the criteria for being eligible for the feegrant. I think it is reasonable to look at the existing relayers in the past 3 months and already sign up all of those to be eligible within the 500 message criteria, without requiring them to manually sign up. Additionally feegrant only covers costs on one chain and not for an entire…

Excerpt (1199 of 3953 characters). Read the whole post on the forum ↗

WhisperNode fully supports this proposal. We went through 11 $ATOM in 1/2 days with the new fees already relaying. Without some support we’d be forced to shut down relaying certain chains. Great job drafting and thinking this through.

Agreed - we need a tech solution - and we have it already, aka Fee Middleware. With that, costs for relaying can be baked into the tx fees we pay - part going to validators on each chain for validating the tx, part going to relayers in each chain for both costs of relaying, and maybe even a profit percentage.

It’s just that current IBC channels can’t be upgraded to include that feature yet.

So yeh, this Fee Grant isn’t a clean or ideal solution, but it’s the best short term fix till Fee Middleware is deployed everywhere (soon?)

I disagree that a long term service contract removes the free market.

The hub can fire the privileged set if service is unsatisfactory.

Sincerely, long time ICS29 hater.

Yes from us of course. There is no logical reason to vote no for this.

Proposal is very modest compared to everything else we have seen in Cosmos Hub in last months.

Supportive of this. Although it is a quick fix, the general idea is good and can be improved over time if a permanent solution doesn’t come. a question, There are many validators who are already funded/supported by the ICF FD program, I know that is not enough but are we setting that aside for human and tech resources?

Hello everyone ,

Inter Blockchain Services is not validating the hub but as IBC supporters we could not afford the cost , just for osmosis / cosmos we used morea than an ATOM by hour( like the other active one ) , and with the use of ICA more and more the hight cost is clearly a problem . Authz is a fast and easy way to “fix” the problem temporarily

I support this idea.

Validators and Relayers should be rewarded as much as they work for sustainable network. I think it’s OK to pay a little bit too much for them not only gas-fee. But as it’s written in Q&A and others says, basically users should pay that money. So I want to know what and when solution will come next. ICS-29? Self-relaying?(I don’t know what it it and even non dev ppl like me can do it.). Or this is just short term solution while team are discussing about it.

Strongly against this.

Either there is a way to fund all relayer or we shouldnt fund them at all. Only funding a self declared subset is market distortion since foundation delegations are often bound to also running relayers which will be easy for these teams but costly for others.

On the short term I see clear benefits for this proposal. Also supportive of ICS29 short/medium-term. On the long term I would be very curious to hear how a design or approach to a solution should look like? And what can core teams do to help, from the perspective of the signers of this proposal? My weakly held opinion is as follows: Some say “IBC should be invisible” and users should not even know what protocol or infrastructure is underlying their interoperability needs. But IBC is already invisible for most teams, and also to users. I think of IBC relaying as an end-to-end concern, reaching into the domain of application development/SDK, and up to wallets and front-ends. Most interest in solving the issue of IBC operational costs historically was from teams lower in the stack (IBC-go, Skip, Hermes relayer, validators). I think this is simply because incentive misalignment, not trying to blame or make a hero of anyone. So I’m wondering how can (dis)incentives bring more involvement from teams higher in the stack and how can we move away from the mentality that IBC should be free. The signing teams on this proposal are probably best positioned to signal the need for such an…

Excerpt (1197 of 1213 characters). Read the whole post on the forum ↗

Thank you for that write-up! Yes ICS-29 can work although for now we at least (lavender.Five) remain sceptical. I dont think thats for now what we want to discuss with the proposal though. It is intended as an interim solution to have Cosmos Hub relaying function again while not paying anyone directly to do that. All teams are welcome as long as they can comply with a small and simple ruleset. susevans: I’m assuming this is what you mean by a race to the bottom for relayers? The race to the bottom mentioned refers to that incentivizing based solely on # of packets transferred causes the system to default to a low-latency competition environment. One where charging a smaller fee, frontrunning, over-spending on Gas and setting up cheaper and cheaper infrastructrure is incentivised. It moves importance of redundancy to being the fastest to get to a packet in the cheapest way. In such a system (as we see at low latency trading firms and other similar businesses) costs will keep increasing until transferring the most packets will not necesarilly generate enough fees to cover them anymore, as a result efforts centralise around the few fastest businesses or fees have…

Excerpt (1199 of 1552 characters). Read the whole post on the forum ↗

waqarmmirza:

but are we setting that aside for human and tech resources?

Yes we could see that as an incentivisation to run the required infrastructure and provide support, it
doesnt come close to covering fees at the current gas price.

Flo:

Only funding a self declared subset is market distortion

ALL relayers are welcome to apply, we do NOT expect to hinder anyone from obtaining the feegrant.

The requirements will be low and are only to ensure no extremely malicious behaviour takes place.

Anyone with 500Txs a month already can be fast-tracked to ensure a fast and smooth rollout.

We are doing experiments with millions of Atoms each year, and I think this is a very reasonable ask for a public good.

Thank you thats great. Looking forward to see how we can apply.

It’s an interesting problem to solve but I can see some similarities to incentivizing validators. Let’s say there was no mechanism for validators to get a portion of rewards, no one will be running validator as a business. To solve that, there is an option for validators to get a percentage of rewards allocated based on their total delegation. Whatever rewards delegators get, validator gets a configurable percentage of that as fee. If validator feels that their percentage of rewards is not worth running a validator, they have the option to shut down. We need something similar for relayers too. Now i know devil is in the details, however relayer incentivization also needs to be transparent to relayer operators. We need a way to pass on a portion of relayer fees (or a small portion of inflation or a portion of dedicated pool for relayers paid out per transaction or ???) to relayers so it becomes self sustainable. I am not suggesting what exactly contributes to relayer incentivization, just making a point that mechanism needs to be automated and meaningful without requiring every chain to have a different solution or significant impacts to all chains and front ends. Just sharing…

Excerpt (1198 of 1227 characters). Read the whole post on the forum ↗

Agree - users should pay their own fees, which Fee Middleware will partly fix, when it gets adopted post-Channel Upgrability release (+ some time for all chain pairs to upgrade, etc etc)

I personally am for this prop, with the understanding that this is just a short-term, stop gap measure. This arrangement should last longer than ~6months (I know this prop is for 3ish months, but I don’t think that will be enough time for Fee Middleware to become available on every chain)

With regards to “any plan from informal” - the question is better suited to @AdiSeredinschi or @jtremback from Informal
(I’m not from Informal )

And just want to add - the relayers are ONLY asking for fee reimbursement - not profit, time/labour or infra costs - a very chad move.

Hub’s vals & relayers are the best!

luisqa: @Syed is there any plan from informal to address this effectively ? On Informal side, we have put significant effort into ICS29 specs, implementation, and adoption. Hermes relayer has been ready for use with relayer fees and we ensured UX is smooth for relayer operators wanting to use it: Hermes has telemetry support for fees, a few CLIs to interact with the fee middleware, and support to automatically register the counterparty payee on the destination chain , and operators can also filter packet relaying by incentivization . This is with the understanding that ICS29 is not a solution to all problems. Beyond ICS29, there was not much progress since we could not land on a problem statement that would be appropriate from a public goods perspective: What makes sense for Osmosis would not make sense for Hub or other chains, or it doesn’t make sense for relayer operators. Put differently, what is the problem to solve exactly? DEXes/DeFi app-chains need relaying to be free for users and seamless to attract activity; relayer operator want relaying to be sustainable (i.e., include fees/revenues); end-users need relaying to be transparent and non-fragmented.…

Excerpt (1191 of 1297 characters). Read the whole post on the forum ↗

Last Call

Please provide additional feedback or mention your name to be on the multisig, if not we will make the multisig with people from the proposal.

We will go on-chain later today with the following onboarding requirements and multisig participants: Onboarding requirements • Be a relayer on the hub and relay on channels to active and important counterparty chains** • stride • neutron • osmosis • kujira • crescent • persistence • juno • secret • evmos • quicksilver • injective • kava • archway • comdex • umee • lum network • akash • noble • axelar • gravity bridge • Have > 10 IBC txs relayed for any of the channels connected to the listed chains in the past 30 days. For the baseline we use Map of zones • The multisig will re-evaluate the Chains and thereby the grantees on a monthly basis • Operation is allowed with the following relayer softwares: ts-relayer, go-relayer (RLY), rust-relayer (Hermes) • relayer must set batch_delay / polling frequency of at least 500ms or more • block time / batch delay = ~5s / 0.5s => max 10 txs per block per account • if any account posts more than 10 txs into one block: misbehaviour • packet efficiency > 10% • Multiple wallets per entity are eligible > every account that is eligible within the outlined criteria **relaying to other chains…

Excerpt (1196 of 3046 characters). Read the whole post on the forum ↗

This is a fine stop gap solution for the short term.

We still need to figure out the long term solution for this as its the crux of the Cosmos IBC philosophy

Sounds fucking amazing

Agree - users should pay their own fees, which Fee Middleware will partly fix, when it gets adopted post-Channel Upgrability release (+ some time for all chain pairs to upgrade, etc etc)

I personally am for this prop, with the understanding that this is just a short-term, stop gap measure. This arrangement should last longer than ~6months (I know this prop is for 3ish months, but I don’t think that will be enough time for Fee Middleware to become available on every chain)

Please provide additional feedback or mention your name to be on the multisig, if not we will make the multisig with people from the proposal.

Agreed - we need a tech solution - and we have it already, aka Fee Middleware. With that, costs for relaying can be baked into the tx fees we pay - part going to validators on each chain for validating the tx, part going to relayers in each chain for both costs of relaying, and maybe even a profit percentage.

It’s just that current IBC channels can’t be upgraded to include that feature yet.

So yeh, this Fee Grant isn’t a clean or ideal solution, but it’s the best short term fix till Fee Middleware is deployed everywhere (soon?)

I support this idea.

Validators and Relayers should be rewarded as much as they work for sustainable network. I think it’s OK to pay a little bit too much for them not only gas-fee. But as it’s written in Q&A and others says, basically users should pay that money. So I want to know what and when solution will come next. ICS-29? Self-relaying?(I don’t know what it it and even non dev ppl like me can do it.). Or this is just short term solution while team are discussing about it.

anilCSE:

Let’s do this. Witval will support the proposal

Let’s do this. Witval will support the proposal

We still need to figure out the long term solution for this as its the crux of the Cosmos IBC philosophy

Basically IBC is the backbone of this will be a great foundation, can’t wait for great things with Namada

Ertemann:

**We are seeking 3 community representatives to join the multisig along with 2 relayer representatives. Please reach out to @Ertemann on Telegram if you are interested in participating or indicate so in this forum post! - There will be no compensation for this role or any role performed by the proposers for that matter.

I fully support this proposal, but I don’t know if the applications are full and how can I join?

Let’s do this. Witval will support the proposal

MoonBoy:

wonder how much mid-high txs would cost if the users pays the costs fully instead of relayers and what if these paid fees would then come back as staking rewards

wonder how much mid-high txs would cost if the users pays the costs fully instead of relayers and what if these paid fees would then come back as staking rewards

Can the node still participate?

Ertemann: feedback It successfully identifies a critical issue arising from the recent gas fee increase and proposes a collaborative solution. Here’s a breakdown of my feedback: Strengths: • Clear Problem Definition: The proposal effectively outlines the adverse effects of the gas fee increase on IBC relayers, emphasizing its potential impact on the Cosmos Hub’s usability and brand image. • Community Collaboration: The involvement of various respected relayer teams in crafting the proposal strengthens its credibility and fosters a collective effort to address the issue. • Transparent Solution: The proposal offers a transparent solution, detailing the multisig structure, fee-grant distribution, and a fair usage agreement. This transparency is essential for community trust and understanding. • Risk Management: Anticipating potential issues and proposing risk mitigation measures, such as hard caps and public analytics dashboards, demonstrates a thoughtful approach to risk management. • Community Involvement: The inclusion of community representatives in the multisig aligns with the principles of decentralized decision-making, ensuring a diverse perspective and…

Excerpt (1195 of 2763 characters). Read the whole post on the forum ↗

@lexa are the above replies all bots that are farming potential engagement aidrops that have become more pertinent with the Celestia and Namada criteria or are these genuine replies?

You can count me in already!

Dont understand why the introduction of a tiered system is necessary. Its just creates inequality that isnt needed or am i missing something there?

Aside from that happy about the positive feedback.

This is good to discuss, in my opinion relayers have similarities with validators in that both are important activities in a blickchain, on the one hand validators get rewards which in my opinion are commensurate with what they do and on the other hand relayers should also get rewards like validators get. . But in the public interest, I strongly agree that relayers pay their own costs temporarily to address existing gaps, and please note that this is only good for the short term, not the long term. Let’s wait and see what the future plans are to overcome this problem (long term solution)!

*How will the community representatives be selected?

Yes. Cosmos forever. belive

Support this initiative. Easy yes.

when the relayer registration to be included in the fee grant will be opened?

Yes from us of course. There is no logical reason to vote no for this.

fully support this proposal

Hey community and relayers, The Relayer Feegrant Working group is now in full effect and we are open to PRs by IBC relayers! We are incredibly grateful for the technical leader of the group @clemensGG (CryptoCrew validators) who has delivered an awesome Github workflow for the FeeGrant implementation, Live Monitoring, Multisig integration and a lot more! Please consider delegating to CryptoCrew validators for all the effort they put in. Relayers can make Pull Requests noting their address and other needed information in the following repository: GitHub - cryptocrew-validators/relayer-feegrant-wg Please follow the Readme.Md for instructions. The Multisig aims to do batch approval TXs every week or so after the Pull request was merged and confirmed. There is an automated alerting service integrated into the repository for stuck packets that might alarm operators if channels are not properly serviced. All metrics and live monitoring of the relayers can be found here: Grafana We hope to see as many Relayer applications as possible. Best regards, The Relayer Feegrant Working Group! IcyCRO - CryptoCrew - CosmosSpaces - CrosNest - Architect Nodes -…

Excerpt (1187 of 1201 characters). Read the whole post on the forum ↗

Just wanna shout out to the work you guys did there so far. Its a good start to push things off the ground for relayers and for smaller validators teams to start relaying too. Well done

Great initiative!

How often do you convene on adding new relayers? We submitted an issue + PR 3 weeks ago

The ATOM received from this proposal has now been depleted by 26 relayers to serve almost 3 million transfers over 6 months.

We are seeking additional gas restitution from the AADAO to continue this effort.

Find a full report here: relayer-feegrant-wg/REPORT.md at main · cryptocrew-validators/relayer-feegrant-wg · GitHub

Just a question why didn’t you pay to Validao or to GATA Hub? They both applied for it months ago. And I can see both are actively relaying.

Hi @waqarmmirza, looking into the PR history it seems they have been onboarded but were actually missed in a review meeting! Thank you for pointing this out, we’ll update the feegrant state as soon as possible in an expedited meeting.

Hey, mura from ValiDAO here. We applied for a feegrant in early February, and our PR was merged early March. Yet, we have not received a feegrant. In the mean time, I understand that the pool of ATOM allocated for this initiative has been depleted, and has even been replenished. Still, we’re without a feegrant ~7 months in after PR merge, 8 months after application time. After a few attempts I managed to get a hold of the maintainers, who let me know that the reasons for the delay was technial problems related to batched authz message signing. I never heard any mention of the above ^ that our applications were missed during a meeting. I hven’t looked into it, so I’m not sure if other later applicants have received a grant since then, or if this is a miss that’s isolated to ourselves. Would like to for clarification: were we simply missed, or was there problems with authz message signing, or are both true? I echo the sentiment of @susevans ’s reply ~nov last year: Putting certain relayers in a privileged position to have their relaying fees funded whereas other relayers would not be subsidised does not seem like the ideal solution and seems somewhat unfair. Only…

Excerpt (1199 of 1248 characters). Read the whole post on the forum ↗