Preparing for Replicated Security

Introduction Interchain Security (ICS) is launching in 2023 and the Cosmos Hub will become a security provider after the Lambda upgrade. After having funded Proposal #72 and Proposal #77 from the Community Pool, the community has already signalled soft consent around the adoption of Replicated Security, but the Lambda upgrade will finalize its inclusion in the Hub. The Hub being able to offer security to new projects means accessing the upside of any projects launching on Hub-secured consumer chains. ATOM delegators can benefit from getting staking rewards made up of not only Hub block rewards, but also of the Hub’s portion of revenue on all consumer chains accessing Hub security. This is a great opportunity to start conversation around the governance and business relationships between provider and consumer chains. Replicated Security will create an interconnected economic zone centred around a provider chain (such as the Hub), so we need to know what kind of partnerships and expectations we can have of both the provider and various consumer chains. Some norms will evolve as we actively use the technology and learn more about it, but this essay is intended to lay out the…

Excerpt (1197 of 20353 characters). Read the whole post on the forum ↗

Skimmed through the post and I echo the sentiment of subsidising the early consumer chains.

I feel like there will be some chains which will underpay the validator set (specially problematic for lower ranked validators) for the value they provide. Can we also subsidise such validators?

Will also like to see some data to study the incentive alignment between {cost to run consumer chain infra + slashing risk} vs the incentive structure of the consumer chain vs {the amount of stake needed on Hub for a validator to break even on a particular consumer chain}

Maybe a simple solution is to create a form for small validators to apply for grants. Throw them some Atom to assist with up front costs in this initial phase. I think that suits the interests of the hub community to keep these small validators afloat and supported while this new tech is being ironed out.

other than circle, what benefits do the other potential consumer chains offer to ATOM? im not entirely sure how many consumer chains validators will be expected to include, but why would the chain fund consumer chains at a loss to both small validators and the chain?

@tknox35 Interesting thought. I think it leads us to a broad question - is it the responsibility of the Hub to keep the small validators afloat during Replicated Security, or is it the responsibility of each consumer chain? Or a combination (e.g., Some Hub support and some consumer chain subsidy as well)? @common_spelling My perspective is that by coming together in a shared economic system, consumer chains are incentivized to act with the wellbeing of ATOM in mind. One of the ideas I mention is of a token swap, which would keep ATOM on the balance sheet of the consumer chain, which would be motivated to act in the best interests of ATOM - I think that’s sort of a heavy-handed example of what I mean by an incentive though! That would be my personal expectation for why the Hub could fund consumer chains even if there’s an expected initial loss. Realistically, I don’t think the Hub would continue to secure chains operating at a significant loss. There is no number of consumer chains that validators will be expected to include. The launch of Replicated Security is just about making it possible, and there isn’t a consensus on how many chains the Hub should provide for. I…

Excerpt (1195 of 1750 characters). Read the whole post on the forum ↗

To understand correctly: • (2.1) with core consumer chains you mean for example a chain focussed on smart contracts which is tightly bound to the provider chain? Such that without implementing the smart contract code in the provider chain, you still have the benefits of that feature? • (3) With the completely same validator set, how can you expect to run fully sovereign? Is that only applicable if you lease a subset of the provider chain? Otherwise local governance would be quite hard to achieve imo. Furthermore if the provider chain validators can choose to drop a consumer chain, you will never get a fully sovereign set, unless they fork away. • (7) Adding the regular call option in there makes RS automatically only suited for bigger validators. Smaller validators who are running (near) solo who are expected to keep their infrastructure up-to-date AND participate in calls… that is a lot to ask imo. Although I very much love the work you have put into this essay (for which I say; thanks @lexa !) I am a bit worried that this is only being discussed now, a few weeks before the proposal goes on-chain. Which means that ( again ) something is developed tech-wise without…

Excerpt (1192 of 1730 characters). Read the whole post on the forum ↗

(2.1) - That could be an example of it, yes. When I was figuring out this thought, I was also thinking about things like the Allocator and Scheduler being hosted on core consumer chains. (3) - Right now, there is no possibility for leasing only a subset of the provider chain, so I wasn’t thinking in terms of a semi-sovereign or sovereign validator set. More so that the Hub’s validators and community can vocally commit to giving a consumer chain the right to make their own decisions. By ‘local governance’ I mean, for example, that if the consumer chain has a native token then the entire chain’s governance could operate as a DAO with that token, and the provider chain’s validators are not at all involved in the local governance of that DAO beyond also being DAO members by holding that token. ‘Local governance’ might involve budgeting for the consumer chain’s treasury, signaling proposals about new features, etc. It would look bad for the Hub to secure a consumer chain and then start censoring blocks or changing their local gov decisions, so a commitment to allowing for ‘self-determinism’ is saying, at the very least: “We want you to have the power to make your own decisions,…

Excerpt (1197 of 1722 characters). Read the whole post on the forum ↗

lexa:

By ‘local governance’ I mean, for example, that if the consumer chain has a native token then the entire chain’s governance could operate as a DAO with that token, and the provider chain’s validators are not at all involved in the local governance of that DAO beyond also being DAO members by holding that token. ‘Local governance’ might involve budgeting for the consumer chain’s treasury, signaling proposals about new features, etc.

I really appreciate the detailed thought about upcoming replicated security. I think it would be a game with whole new rules especially for consumer chains. About the local governance, I agree that it would be similar to DAO governance or Optimism’s model. One thing that stands out is that few(or, many) proposals would not only affect the consumer chain itself, but also the provider chain, hub especially.
So, how can we design a governance structure where soveriengty of consumer chain is preserved, but also hub’s stakeholders opinions are also reflected? Maybe a bicameralism like Lido? I don’t know, but I think its a topic we should deeply research & think about!

I’m interested in the economic cooperation between provider and consumer chains. As validators of course we would like to receive good revenue in exchange for our services of providing security. But we also think that projects would need time to be successful and generate a lot of revenue.

How do we assess the consumer chain performance? What are the metrics? Timeline?
How can we support them?

One question, if the consumer chain doesn’t have a token, they can pay in any currency?

Thanks.

I am really impressed, “bravo” for this work, I will take the time to read everything carefully to make my comments.

Anyway, the idea of formalize the relationship between the HUB and the consumer chain is obviously very important and you did a nice job by bringing the subject to the table.

what does keep ATOM on the balance sheet mean? How does it motivate consumer chains to atom’s benefit?

lexa:

One of the ideas I mention is of a token swap, which would keep ATOM on the balance sheet of the consumer chain, which would be motivated to act in the best interests of ATOM - I think that’s sort of a heavy-handed example of what I mean by an incentive though!

it takes the permission of <25 validators to adopt a chain that 175 have to run nodes for.

why not create an auction consumer chain that lets people buy into project white papers instead of using the liquidity that secures the chain to bootstrap solutions looking for problems. allocator consumer chain etc.?

why would you subsidize a consumer chain with atom’s liquidity in addition to accepting its illiquid farming/fee token to use ATOM’s security?*

@VK_S16 How do we assess the consumer chain performance? What are the metrics? Timeline? How can we support them? Great questions. I think the low-hanging fruit of assessment (from the Hub’s perspective) would be in the revenue being generated, but I’m not sure how we would check on softer metrics. How do we measure the success/performance of any blockchain in the space? The timeline is something I think each chain would present since their project leads are in a better position to lay out their intentions from the start. One question, if the consumer chain doesn’t have a token, they can pay in any currency? If the consumer chain doesn’t have a token, they’d pay in ATOM if the Hub is securing them. However, the sdk allows chains to whitelist tokens that can be used for fees (‘fee tokens’) so it’s possible that a chain could use a token other than ATOM…I just don’t know why they would. Since Hub validators would be accepting the fees, it might have to be a token that’s enabled on the Hub too…and I think we only use ATOM right now. I’m a bit fuzzy on this detail so if anyone has a correction, I’m open to it @common_spelling what does keep ATOM on the…

Excerpt (1198 of 3738 characters). Read the whole post on the forum ↗

I don’t want to dismiss this out of hand, but I also don’t know how to solve it strictly in the context of ‘launching ICS’, y’know? On a personal level, I want this feature to launch so that we get the opportunity to see it in action and actually give these prospective consumer chains a chance to see how they’ll solve the problem. My hope is that some of these teams will have ideas that help us brainstorm ways to decentralize our validator set on the Hub itself. Agreed on this Maybe ICS can reward lower ranked validators a bit more than higher ranked. The high ranked will then still have a decent APR, but the lower ranked slightly more (just thinking out loud) I think it’s an investment - the Hub might offer this subsidy with the expectation that it will pay off later. I don’t think we can access lucrative investment opportunities without being willing to take some risks, and the actual risks we’re willing to tolerate are up to the voters. I’m generally risk averse, but I’d still rather have the opportunity to invest than not. Not to forget that RS adds a layer of utility to ATOM. Which means that subsidizing the concept (at least in the early stage) might prove the…

Excerpt (1198 of 1420 characters). Read the whole post on the forum ↗

lexa:

I think the low-hanging fruit of assessment (from the Hub’s perspective) would be in the revenue being generated,

what revenue? they are paying us back with an illiquid consumer chain token.

If we make the farming/fee token liquid using a token swap thing like the allocator, it is just bleeding Atom liquidity and security from the chain to validators and devs/founders that get oversized allocations.

lexa:

It means that when a chain looks at all its assets (various tokens), ATOM is on that list - it could be in their treasury or whatever equivalent of a community pool they have.

who controls their balance sheet? and how does an illiquid farming/fee token holding on to Atom or Atom Debt incentivize them to act in alignment with Atom?

common_spelling:

they are paying us back with an illiquid consumer chain token.

I mean harsh intonation, but this:

they are paying us back with an illiquid consumer chain token

Is kinda true… I mean, what can be said here is the same what is applicable to supply chain rules in a free market. If a supplier delivers rotten tomatoes in one of the boxes which they keep hiding, they will lose reputation and the buyers will simply switch supplier. True. If a chain ends up being useless and that’s what they offer, I’m assuming, the validators would drop it. Or even more - that chain would lose all reputation as a consumer chain and will to be able to hire new validators. And then again… On a free market, there will always be someone willing to take on the job.

With all of that said. That sentence is still kinda true. And I think it should be thought about in detail

it seems like a mistake for ATOM to incentivize chains willy-nilly that dont offer clear benefit to the chain.

IMO LSD chains offer no value to the chain, adds cost to small validators, and pays for transactions with it’s minimally liquid farming token. Further funding such an obvious burden on the chain seems either negligent or corrupt.

circle has many obvious benefits. Neutron seems beneficial and provides a SC LSD. not sure what other consumer chains will be proposed, but I hope that economic impact isnt only gauged on, untested, projected volume when they are paying in an illiquid farming/fee token.

If a supplier delivers rotten tomatoes in one of the boxes which they keep hiding, they will lose reputation and the buyers will simply switch supplier.

This is pretty much it, imo. The Hub isn’t going to take on chains willy-nilly; we’re a premium provider and can afford to be choosy but we also have to treat our customers well. This is purely personal speculation, but I doubt we’ll end up securing more than a dozen chains at the absolute most. I think we’ll launch 3-6 (incl projects funded by prop #72) and the bar for quality will be super high.

Neutron and Circle are among the first consumer chain we’re going to see try to launch on the Hub and, as you’ve said, they seem to have obvious benefits. If we can conceive of consumer chains that have benefits to the Hub, why the uncertainty about Replicated Security as a feature? Am I misinterpreting what you’re concerned about?

technically how many channels can be secured by the hub? is there a limit?

There’s no physical or technical limit that I know of. If we had all the infra and person-power in the world, we could keep adding forever.

More realistically, there are personal and business limits such as:

• Number of nodes validators are willing to run (since it takes one per chain)
• Decreasing ROI (return on investment) for securing new chains as we scale up
• (Maybe) Increased network activity affecting block time in the long run

These business reasons are why I personally think we’ll end up with a pretty low number of really high quality consumer chains. Chains other than the Hub will also likely become security providers, so in the future, consumer chains will probably be shopping around for a good provider and it’ll be about finding a good fit.

suddenly the Hub could provide security to a subordinate hub which itself could provide its security technically? It could be the definition of really high quality consumer chains ?

I’m not sure what you mean by ‘subordinate hub’…something like this, maybe:

The Cosmos Hub blockchain taking on another chain (Blockchain A) as a consumer chain, and Blockchain A then providing security to yet another chain (Blockchain B)? Like a subletting situation?

That hadn’t even occurred to me, honestly. I don’t know enough about the technical specifications to know if that’s possible but my gut feeling would be no, the code doesn’t let you provide security unless you have your own sovereign validator set.

A ‘high quality consumer chain’ imo, is one that is successful at filling its product niche and participates in a mutually beneficial relationship with its security provider.

yes thats what i meant. ok thx for your time.

lexa:

If we can conceive of consumer chains that have benefits to the Hub, why the uncertainty about Replicated Security as a feature? Am I misinterpreting what you’re concerned about?

im not concerned about replicated security as a feature, my concern is with funding liquid staking chains with atom liquidity. funding LSDs doesnt benefit atom.

I am just unable to conceive of a way an LSD consumer chain would return value back to the provider chain to justify its funding. 25% of the txn fee paid in an illiquid farming token is not much of a carrot.

common_spelling:

im not concerned about replicated security as a feature, my concern is with funding liquid staking chains with atom liquidity. funding LSDs doesnt benefit atom.

But replicated security is providing security, not liquidity.

common_spelling:

I am just unable to conceive of a way an LSD consumer chain would return value back to the provider chain to justify its funding. 25% of the txn fee paid in an illiquid farming token is not much of a carrot.

There will be different token designs for different LSDs. I’m not an expert, but it seems silly to write them all off as “illiquid farming tokens”.

Do you mean that your concern is that you don’t want to use up a slot in the Hub’s replicated security capacity to fund LSDs because you don’t see how a LSD token will drive value back to the Hub or act as adequate payment?

@ala.tusz.am the discussion also involves prospective funding. lexa: 4.1) Investing in consumer chains Provider chain validators will need to take on upfront costs in acquiring hardware and additional labour hours in setup, onboarding, and maintenance. ICS is an emerging technology, and the benefits to both validators and consumer chains might not fully manifest until the technology solidifies. The relationship between a provider and consumer can be seen as an investment that may not pay off until later. These investments might look like: • A discount on security services for early consumer chain adopters of the technology who will help iron out bugs and work out issues that arise, such as the funding generated by Proposal 72 . Investing in projects like this could also be way to attract talented teams to continue contributing to the ecosystem. • An initial token swap between consumer and provider chains that could ensure that a consumer chain is well-capitalized early on, and that the provider will have direct exposure to how the project pays off in the long term. ala.tusz.am: Do you mean that your concern is that you don’t want to use…

Excerpt (1197 of 1871 characters). Read the whole post on the forum ↗

Thanks @lexa for this clarifying post. I’m copying over a post from another thread to see if I can get some help with trying to estimate ICS costs: Trying to figure out how much ICS costs would actually be for new consumer chains, and would appreciate any clarifications you’re able to give re my back-of-the-envelope calculations below. A figure I’ve heard being thrown around is roughly 600-800 dollars per month for validators, times 175 validators. In addition to that, on top of the rewards coming from consumer chains the validators get an average fee of around 10% (meaning we’ll have to multiply the results by 10 to get the revenue needed to cover such costs). The result would look something like this: $600 x 175 validators x 12 months x 10 = $12.6 million per year $800 x 175 x 12 x 10 = $16.8 million per year Although this does not consider the weight of different validators, these figures are quite extraordinary, and surely cannot be correct? Hard to see new chains being able to cover this kind of amount, especially so during a bear market like this. Or am I missing something? Further, I’m wondering whether or not there is a normal investment risk for validators and…

Excerpt (1199 of 1524 characters). Read the whole post on the forum ↗

@common_spelling Thanks for clarifying! Honestly, I’m not that knowledgeable about the way liquid staking chains might work so it’s possible that my generalized thoughts on replicated security as a whole don’t map perfectly onto how it works for LSDs in particular. Relying a bit on your response to @ala.tusz.am here, sure, this is an adequate approximation of what I mean. My main response is just that if something doesn’t make sense to ‘take up a slot’ on the Hub, I think we have to trust the community and validators to not let it take up a slot. I’m definitely not here to argue that there’s a way for every kind of consumer chain to be profitable and should be given a chance regardless of demand or accounting! @saint.Ericus The cost per validator per chain is the real thing that’s hard to nail down here, and it’s also the thing that will wildly swing the back-of-napkin cost from ‘reasonable’ to ‘what the heck’. I haven’t been directly involved in many conversations about this parameter but I’ve heard $300-450 quoted, but that’s not an accusation that your quotes are wrong. The Hub has 175 validators, but inevitably many of them have very different per-chain…

Excerpt (1199 of 3246 characters). Read the whole post on the forum ↗

lexa: The cost per validator per chain is the real thing that’s hard to nail down here, and it’s also the thing that will wildly swing the back-of-napkin cost from ‘reasonable’ to ‘what the heck’. I haven’t been directly involved in many conversations about this parameter but I’ve heard $300-450 quoted, but that’s not an accusation that your quotes are wrong. The Hub has 175 validators, but inevitably many of them have very different per-chain costs. Just to break your math into a few more steps (particular the fee split part, because it took me a few reads to understand what you’re saying): • $600 x 175 validators = $105,000 per month • $105,000 per month x 12 months = $1,260,000 per year And then, assuming that the validators’ additional revenue from Replicated Security comes in the form of a revenue split where the consumer chain takes 90% and the Hub validators take 10%: • Total revenue = ($1,260,000 per year)/(10% fee split) • Total revenue = $12.6 million My conclusion here is that I can’t really get behind any napkin math at all, even if nothing jumps out as glaringly wrong. It’s just too complex to model like this - the variation in validator costs,…

Excerpt (1199 of 2761 characters). Read the whole post on the forum ↗

Thanks for the post @lexa !

Do I understand correctly that if a validator decides to not validate the consumer chain although it has been approved by 2/3, it would get slashed on the Cosmos Hub? Would the normal Cosmos Hub slashing rules apply?

Further, do I understand correctly that Cosmos Hub validators would still earn the same as now and ON TOP the rewards from the consumer chain?

What will happen to the “CEX Validators” such as Kraken or Coinbase. I assume they are not allowed to onboard new tokens that easily (here: potential consumer chain tokens). What if such a token would be classified as a security? Would these validators then simply vote against them and maybe overrule the “community validators” or is there some middle way? For instance, Validators could opt-in to receive the token of the consumer chain and it they don’t they would receive ATOM. Maybe the consumer chain could directly swap their token to ATOM and then send these as a rewards.
What do you guys think?

I am always surprised about how many costs validators make on validating chains. You need proper resources, that is for sure. But I am also sure that a lot are overspending by several lightyears.

I have been playing around on some chains (also spoke to @jtremback last week) where when there is not a lot of traffic you can easily validate with servers of costing $20 a month or so. (I really like to find out what the bottom is where validating is simply not reliable anymore, just out of curiousity)
It most likely is a bit less on performance, but why go for a Ferrari when you are not allowed to drive of 30 km/h?

Suppose every validator runs 5 of those servers (Horcrux, sentries, etc); then the expected costs go down dramatically:

• $100 x 175 validators = $17,500 per month
• $17,500 per month x 12 months = $210,000 per year

That makes quite the difference…

@BuilderBot Do I understand correctly that if a validator decides to not validate the consumer chain although it has been approved by 2/3, it would get slashed on the Cosmos Hub? Would the normal Cosmos Hub slashing rules apply? That is correct. I think it is absolutely critical that everyone knows that there is no ‘opt-in’ to validating a consumer chain once it has met those two criteria (governance approved, adopted by >2/3 of the validator set). ‘Deciding not to validate’ = downtime = slashing on the Hub. That said - the window for downtime is going to be much more flexible for consumer chains; i think we’re looking at something like 4 days whereas the Hub’s window is approximately 19 hours (note that both of these are approximate since the downtime window is really in the form of “missed blocks”, not “days absent”). What will happen to the “CEX Validators” such as Kraken or Coinbase. I assume they are not allowed to onboard new tokens that easily (here: potential consumer chain tokens). This is such an interesting idea to explore - thanks for bringing this up! I think the first relevant thing is that CEXs aren’t necessarily sending the exact rewards to their…

Excerpt (1199 of 3551 characters). Read the whole post on the forum ↗

@lexa thanks a lot!
Two follow-up questions:

If the consumer chains would pay out rewards in ATOM, where do they get the ATOM from? Would the inflation increase or do they need to buy ATOM?

Running the consumer chain nodes - could this be outsourced to third parties? For instance, if a small validator is overwhelmed with creating and maintaining nodes for multiple chains, is it feasible then to outsource it to infrastructure providers? In other words, how does the Cosmos Hub validator “talk” to its consumer chain node?

If the consumer chains would pay out rewards in ATOM, where do they get the ATOM from? Would the inflation increase or do they need to buy ATOM? That’s a level of detail/specificity that feels more up to the consumer chain to figure out. Maybe the project was bootstrapped with ATOM (like some of the Prop 72 chains) so it already has an ATOM-heavy treasury, maybe they do a tokenswap, maybe the system isn’t ready to pay ATOM up front but transitions to ATOM once their token is established and can swap for ATOM in liquidity pools? There’s lots of creative ways for a project to get a hold of ATOM, but I don’t think I’m the most creative person to lay them all out. Running the consumer chain nodes - could this be outsourced to third parties? Not running a validator setup myself, I can’t be sure. But we know that not all validators are running bare metal machines - plenty already outsource even their Hub node to AWS or other white label providers. I imagine it would be possible but I don’t know how practical or likely it would be. From a philosophical perspective, I find it hard to imagine that a bare metal validator would decide to outsource part of their setup. Bare…

Excerpt (1194 of 1555 characters). Read the whole post on the forum ↗

I indeed this bare metal (self hosted) validators are really proud of the fact that they own, control and run their own hardware. Having them go over the line of going back to cloud hosted servers is probably a sign that the load on validators is too high and that we have to think hard about simplifying the work required to run validators on the Hub in combination with consumer chains.

And really outsourcing the operations of a node for a consumer chain feels a bit like the line of work we see for AllNodes. Where everything (server, OS, binary) is managed by a 3rd party provider, which is really a bad thing imo. Validators should be in control of their own nodes at all times, being able to maintain it. Running a validator is not a job on the side, it requires learning how things work. So the max we should ever want for renting stuff from 3rd party is the hardware, nothing more.

sounds reasonable, thanks a lot for your answers! They were all very helpful.

Another question came up, assuming Validators will receive the consumer chain tokens. How does the claiming work?

• Can I claim once and it claims all consumer chain tokens including the normal Cosmos Hub ATOM rewards?
• I set a separate reward address for my val where I receive rewards currently. Will I need one reward address per consumer chain?
• I assume Keplr wallet would support the consumer chain tokens? I guess one could just add it there.

Thoughts?

thanks! Makes sense!

BuilderBot:

• Can I claim once and it claims all consumer chain tokens including the normal Cosmos Hub ATOM rewards?
• I set a separate reward address for my val where I receive rewards currently. Will I need one reward address per consumer chain?
• I assume Keplr wallet would support the consumer chain tokens? I guess one could just add it there.

I think these questions are more about the specific wallet app you use than about the details of ICS so now this is just two folks guessing at how things will work! I’m not an expert here!

1. Probably claim all at once? Since ‘claiming’ is an interaction with your validator and your validator would be holding all those rewards for you.
2. I have no idea…I’m not sure if you mean a ‘reward address’ held by a validator to receive block rewards?
3. I think if ICS launches and a consumer chain gets added, it would be a very weird choice on the Keplr team’s part to not support it

okay thanks a ton! Alright, I thought ICS also handles how the rewards process of the consumer chain works but I guess not then, or at least not that specific part.

What i understand…
Preparing for replicated security involves the following steps in brief:

1. Identify critical data and systems.
2. Choose appropriate replication technology.
3. Define security policies and procedures.
4. Implement encryption for data protection.
5. Conduct regular security audits.
6. Train employees on security measures.
7. Continuously monitor and update security.

@Amarjeet_Singh

I’m not sure what you mean by this. Those steps don’t seem related to what I’ve written or to the Replicated Security technology.

lexa:

Point being - CEXs don’t need to support consumer chain tokens, and they don’t need to actually give consumer chain tokens to their customers. They could decide to just provide an interest rate in ATOM and their customers could either accept that or move to a custodial wallet if they really want to see the direct upside of a consumer chain.

Sure, if customers are willing to forfeit the Consumer Chain Tokens. But this wouldn’t change the fact that CEXs now potentially trade (in case they sell it on the market to cover costs) and receive securities as a reward for validation work. Does not really matter whether the customer receives it or not, harm is already done