Edit: We have abandoned this proposal and posted this as a replacement - https://hubble.figment.network/cosmos/chains/cosmoshub-2/governance/proposals/15 Looking for feedback on this: docs.google.com Figment Networks - Cosmos Hub 3 Upgrade Proposal B Cosmos Hub 3 Upgrade Proposal B Figment Networks August 23, 2019 Context Cosmos SDK is software that all validators must run in order to participate in the Cosmos Hub blockchain. The Tendermint team, All in Bits, has upgraded the Cosmos Hub testnet... This proposal is intended to signal acceptance/rejection of the precise software release that will contain the changes to be included in the Cosmos Hub 3 upgrade. A high overview of these changes was successfully approved by the voters signalling via Cosmos Hub 3 Upgrade Proposal A . We are proposing to use this code Release v2.0.0 · cosmos/gaia · GitHub to upgrade the Cosmos Hub. We are proposing to export the ledger’s state at Block Height 1774000, which we expect to occur on Wednesday, September 11, 2019 at or around 4:22pm UTC. We are proposing to launch Cosmos Hub 3 at 6:22pm UTC on Wednesday, September 11, 2019. Instructions for…

Excerpt (1191 of 1848 characters). Read the whole post on the forum ↗

Looks great to me. Looking forward to pulling out the bowtie!

…Gentleman Jack!

1/2 hours early will much friendly to Asian friends.

Heya! Not sure I follow–are you asking for the upgrade to be earlier?

Yes. launch at around 4:00PM UTC.

Makes the state export around 7am PST, difficult for Tendermint team on the west coast.
Perhaps we could alternate between 4pm and 6pm UTC each time there’s an upgrade. Thoughts?

It seems that 16:00pm UTC is best time for all of us.

time zone.png989×633 44.2 KB

Don’t need 2 hours for state export. Maybe 1 hour is enough.

September 12~14 is thanks giving holiday for most Asian countries. It is the biggest family holiday in Korea and China. It is strongly recommended to do an upgrade after the holiday or 1~2 day earlier.

This statement is from strong agreement of 4 Korean validators, B-Harvest, A-Team, Cosmostation, and CoinOne.

Our holiday start from 13th to15th September.

+1

Thanks for posting this!

Agreed, 6:22 pm UTC equals 2:22 am in China

Thank you @Gavin for planning this.

On timing:
I think what @ping suggest is good. But maybe 1 hour earlier so our friends in Korean can rest earlier.

On dates:
Having it prior to the holidays (before 11 Sept) might be better - reason being there is Wanxiang Blockchain Week in Shanghai right after the holidays (16 to 18 Sept), which I believe many validators will attend.

Time is not friendly to Asian friends, I disagree with the time.It seems that 16:00pm UTC is best time for all of us.

1 or 2 hours earlier would be best for all of us including Korea and Japan. I would say 2 pm or 3 pm UTC. Thanks.

Sept 11 1600 UTC is fine for us as long as it won’t touch 12-14 Sept for the Mid Autumn Festival. Otherwise, it’s better be the week after.

Hey all! Thanks for all the info, I’m glad to see so much participation.

I’m going to revise the proposal:
State export for Block Height 1,823,000 to target Sep 15 ~2pm UTC (7am PDT).
Genesis epoch 1568563077, so upgrade at 3:57pm UTC (~1am KST).

FYI, doing Sep 15 because I don’t want everyone in China/Korea away on holiday directly after a major upgrade…!

Awaiting deposits: https://hubble.figment.network/cosmos/chains/cosmoshub-2/governance/proposals/14

Apologies for not bringing this up prior to proposal creation (just read this thread), but I am quite concerned about the specification of an exact restart time instead of one relative to the export block timestamp.

Have you calculated the expected deviation (error bars) on the time estimate for block 1,823,000? If network conditions and/or the stake distribution change, even a little, over the next month, that height may come more quickly - in which case the network might experience an unnecessarily long downtime - or more slowly, in which case operators might not have enough time to upgrade, creating unnecessary operational risk. It is perfectly possible that block 1,823,000 happens after 3:57 PM UTC, in which case it is unclear what would happen.

I think a version of this proposal which specified a relative, as opposed to absolute, restart time would be safer, and I would advise that the relative offset be on the order of four or five hours instead of two - the cost of a few hours of downtime is low, better to ensure that validators have sufficient leeway to upgrade carefully.

Glad you’re mentioning this concern. I raised it here (let us know if you’re not in this channel).

The feedback I received was that we could signal on-chain if the block time average deviates substantially. I also like the idea of the time being relative to the export block, and agree that there should be adequate time between the two. The only feedback I received was to shorten the two hour window.

How would an on-chain signal work? Do you mean another proposal closer to the deadline? That is possible, although if it were created last-minute, some validators might not see it & confusion could ensue - I think a relative restart time is likely to be safer (unless there are problems that I haven’t thought of yet).

A second signaling proposal is safer if the first doesn’t include an exact time and mentions that a second will come, so that seems OK too (but this proposal does include an exact time and doesn’t mention a second signaling proposal).

I don’t have a strong opinion on window length, just a general preference for safety over liveness - network upgrades conducted like this are risky operations for validators & users alike, better if all involved have time to proceed carefully without worrying about downtime if they don’t export & restart fast enough.

How about a specific time rather than a specific block height? eg. “the last height at September 15, 2019 at 12:00 pm UTC”

and then a specific genesis time eg. “September 15, 2019 at 4:00 pm UTC”

A specific time won’t work as we don’t have clock synchrony (validators may have differing clocks).

“Last block with timestamp before X time” would work as long as Tendermint enforces that subsequent timestamps are non-decreasing (I don’t remember), although I think it’s a bit more likely to result in operator error since we don’t have an easy way to query that from a node (though we could build one).

I’m in favour of a second proposal to override the genesis time of Gov14 to be relative to the timestamp of block height 1823000.

Main issue I can see with a bigger window is that we start to push our friends in Asia into the middle of the night. If we use a four hour window, genesis may be 3 am in Korea.

How about an 8-hour window? From block height 1,823,000, 9 hours makes genesis at 7pm EDT, 4pm PDT, 8am KST, 1am CET.

Summary of proposed solution

Another proposal that does these:

1. Amends the genesis time
2. Signals genesis time as 9 hours after the timestamp of Block 1,823,000
3. Targets genesis time being 11 pm UTC

So do we need to vote for the proposal 14 or wait for the new proposal?

I would suggest a second proposal, #15, that would override first one "Just in case the conditions on Proposal 14 fails due to block time deviation"

The idea is to script a condition similar to: If at block 182XXXX ( few days before the plan ) the UTC time is too late and we are in risk of not having enought time to upgrade, then the genesis time will be relative to the proposed block, in order to give at least 2 hours window. If there is no risk then we keep going with first proposal #14.

Bests

This sounds fine to me, shall we adjust the block height forward a few days so validators have the same amount of time? There seemed to be some concern with 9 hours downtime - I think it’s fine, but probably 4 or 6 hours would also be reasonably safe.

Glad to make the proposal if you like.

There seemed to be some concern with 9 hours downtime - I think it’s fine, but probably 4 or 6 hours would also be reasonably safe.

I am in favor of not stopping it. but as everyone is in favor of stopping it or most believe it normal, then the best will be the shortest possible time. (minutes unless hours) But if most believe it is the right thing, Im ok too stopping my nodes some hours.

I believe that 2 hours is sufficient time for a validator with zero automation to securely distribute the genesis file, restart multiple sentry and validator node and KMS nodes before an upgrade.

2 hours is plenty of time to do this at a deliberate pace with opportunity to correct errors.

I’m leaning toward proposing a revised genesis time of 120 minutes after the timestamp of Block 1,823,000.

In future, let’s educate the validator set about the importance of downtime relative to the risk of double-signing. I’d like us to consider closing the window between export block and genesis time for upgrades to come. We’ll have a better opportunity to make competing proposals after this upgrade.

Fair enough, I don’t have a strong preference.

Independently, can we push the block height a few days? September 15th conflicts with Tel Aviv Blockchain Week which I think a few validators are involved in.

I don’t think we can avoid conferences (unless they are very big ones). For example, Wanxiang Blockchain Week in Shanghai is apparently Sep 16 to 18. How big is Tel Aviv Blockchain Week, @cwgoes?

Besides having enough time for the gov proposal to pass, I’m optimizing for a day that targets a block height that is most ideal globally, ie. a day that is:

1. Not on or before a long weekend for many people
2. Not on or before a large holiday
3. Not on or right before a large conference
4. Well outside of daylight savings time changes

I’m also making it a “round” block number of four sig figs and a time that ideally isn’t the middle of the night any validators. Would love to know if there’s anything else we should be watching out for.

I agree that we are unlikely to be able to avoid all potential conflicts. It would be nice if there were a better way to poll the validators, I really doubt the subset who are reading & replying to this thread is a representative sample.

Tel Aviv Blockchain Week is probably a medium-size conference; I think Wanxiang Blockchain Week is pretty large, in both cases I do not know how many validators plan to attend.

An interesting future option could be a grace period of a few days after upgrades during which downtime slashing does not happen, that would make the process a bit less synchronous (as long as at least 2/3 of stake is online).

Neat idea, re: downtime slashing, although it is already a pretty big window at 18 or 19 hours before a very small slashing.

I’d like to see additional on-chain governance features for coordinating this sort of thing. Eg. a proposal could have multiple options and ranked choice.

Ranked choice proposals are a great idea!

Anyone want to play around with ranked choice voting? I set up this ranked vote for a launch window time: http://t.me/RankedPollBot?start=gm6xciimfcak7ipta656ibfe

Nice, submitted responses (hopefully).

I’m drafting the amendment proposal here: Cosmos Hub 3 Upgrade Proposal C

Please participate! and RT for visibility: https://twitter.com/Ether_Gavin/status/1166820073107075074

We are abandoning Gov 14 and have proposed Gov 15 to replace it: https://hubble.figment.network/cosmos/chains/cosmoshub-2/governance/proposals/15