Testnet Wednesday Reports

Stride Launch Rehearsal (July 5, 2023) This Wednesday, Hypha hosted Stride’s second launch rehearsal on the Replicated Security Persistent Testnet immediately after a very smooth Neutron mainnet upgrade The testnet program is stewarded by Hypha Worker Co-op and provides a testing environment for pre-launch consumer chains on the Replicated Security Persistent Testnet. This work supports the success of the Cosmos Hub mainnet by catching issues before they go live and letting dev teams and infrastructure providers alike practice with the new technology introduced by RS. Stride will be the second Hub consumer chain to go through launch rehearsals on the RS testnet and the first sovereign-to-consumer transition. With Stride going live on the Cosmos Hub in two weeks, the stakes were high for this rehearsal. In the last month, Hypha and the Stride team have worked closely to iterate on feedback from our last rehearsal on June 7. Naming convention clarity For chains that start off as consumer chains (such as Neutron), a genesis file is provided ahead of time and then modified after spawn time to include the CCV state. This modification can only happen after spawn time, as the…

Excerpt (1198 of 4259 characters). Read the whole post on the forum ↗

Conversation of here proposals need voting before approved.

We’re going to use this thread as a running log of Testnet Wednesday events! July 12, 2023 We launched Duality’s second chain on the Replicated Security testnet on July 12. `duality-testnet-1` will remain online as Duality’s persistent test chain. The launch event took about one hour from proposal passage until interchain secured. Shortly after spawn time, we ran into a JSON unmarshalling issue that caused the Duality binary to panic. After a bit of digging, we discovered that the `app_state.group` value had been incorrectly defined as an array of objects, rather than one object. With the aid of a few helpful validators, we fixed the bug, published an updated final genesis file, reset the node state, and started seeing blocks. Today’s launch also reminded us to always run genesis files through `jq` before generating checksums. Without standardizing the format of genesis files with `jq`, whitespaces and differently-ordered keys will lead to different checksums, even though the data is the same. This is especially important considering that some validators build their own final genesis files after spawn time. We unfortunately did not manage to have enough time to upgrade…

Excerpt (1197 of 1799 characters). Read the whole post on the forum ↗

July 26, 2023 We had THREE testnet upgrades in one day! This is the highest number of operations performed in one day so far. • Upgrade `theta-testnet-001` to v11 • Upgrade `provider` to v11 • Upgrade `pion-1` to v1.0.4 The Gaia upgrades on the Theta testnet and the provider chain went smoothly. Both were major version upgrades, and these were coordinated upgrades via governance proposal. In both cases, the chain came back online within a few minutes after hitting the upgrade height. `pion-1`’s block height also drifted forward a bit, enabling us to start and conclude all three events in under 90 minutes. Snappy! Coincidentally, we observed an issue with Hypha’s validators around the same time as the first scheduled upgrade. On the Replicated Security testnet, the nodes that were running Stride ran out of memory. This led to Hypha’s Stride validators going offline for a bit, which led to a temporary loss of consensus on the provider chain. Thanks to alerting from some proactive validators, we were able to provision more memory and recover in time for the Gaia upgrade. Validator shout outs • Bosco from Silk Nodes for posting step-by-step instructions for the Gaia…

Excerpt (1199 of 1339 characters). Read the whole post on the forum ↗

Wednesday August 23

The last few weeks have been pretty quiet on the testnets! Here’s a quick roundup.

August 9, 2023:

• No coordinated Testnet event, but all validators were asked to upgrade their Gaia version from the v11 release candidate to the official v11 release.

August 16, 2023:

• No testnet event, but we upgraded mainnet to v11.

August 23, 2023:

• Upgraded release testnet from v11 to v12-rc
  • Upgrade height at 13:41 UTC; blocks at 13:47 UTC
• Upgraded provider chain from v11 to v12-rc
  • Upgrade height at 14:20 UTC; blocks at 14:25 UTC

Very boring and smooth upgrades all around! Thanks everyone for your help in making this another successful Testnet Wednesday

Composable joins the Replicated Security Testnet today! It’s been over two months since we last launched a consumer chain, so today’s rehearsal involved a bit of shaking off some dust. The Composable testnet chain, `banksy-testnet-3`, will remain online for about a week so the dev team can run some tests. Today’s rehearsal timeline was as follows (all times in UTC): • 14:52 - Composable upgrade proposal submitted • 15:03 - First cc addition proposal submitted on provider chain • 15:22 - Composable upgrade proposal passed • 16:12 - Composable upgrade height reached, chain paused • 16:15 - Consumer chain banksy-testnet-3 offboarded from provider chain • 16:20 - Second cc addition proposal submitted on provider chain • 16:27 - Spawn time for banksy-testnet-3 • 18:12 - Enough Composable validators upgraded to reach quorum, Composable chain starts the “three block” countdown • 18:23 - Blocks on consumer chain • 18:29 - Relayer up, interchain secured We had to resubmit the consumer-addition proposal on the provider side because > the proposal was initially configured with incompatible parameters for the chain ID suffix and the revision number (`“banksy-testnet-3”` and…

Excerpt (1198 of 2586 characters). Read the whole post on the forum ↗

Fantastic news to hear

This Wednesday marked the start of our first period for the Testnet Incentives Program (Nov 22 - Dec 20, 2023). Today we performed a minor upgrade from `v14.0.0` to `v14.1.0` (forum post for this version here ). `v14.1.0` contains cryptographic equivocation code to automatically process equivocation evidence and slash offending validators without the need for an equivocation proposal via governance. We will be testing the consensus-breaking piece of code in this upgrade on Friday November 24, 2023 at 15:00 UTC. Nodes that are still running `v14.0.0` at this time will have apphash errors. To avoid this, make sure you have upgraded to `v14.1.0` before Friday. The Testnet Incentives Program is a pilot program, so we are continuing to gather feedback on process, criteria, etc! Some of the feedback we’ve received: Verifying mainnet validator status Validators are telling us: Testnet validators want a way to verify that they operate on mainnet without using their mainnet key on the testnet, for ease-of-completion, operational security, etc, We updated the Google form to allow verification using only mainnet keys on mainnet. Hypha’s validators decide when the upgrade…

Excerpt (1197 of 3628 characters). Read the whole post on the forum ↗

November 29, 2023 Last week, we upgraded the `pion-1` chain to `v2.0.0`. This upgrade contains a lot of migrations, which made it a very long upgrade (which was to be expected). Hypha’s validators used up an extra 100 GB of disk space during the upgrade. This and memory resources are major considerations for mainnet – validators will need to make sure they have enough disk space and memory to perform the upgrade! Min gas price requirement • `v2.0.0` makes minimum-gas-prices a mandatory parameter in app.toml. This means that a value must be given, even if it is “0untrn”. • The recommended value is 0.02untrn. • Validators who had not set this parameter saw errors after upgrading. • Some validators reported that their node would not start if there was only an empty string in this field. Time- vs height-based expectations for testnet upgrades Context: • Software upgrades are technically height-based, and various tools are used to predict the time at which a particular block height will occur. • On mainnet, the voting period for governance proposals (either via governance or a subdao) are typically quite long and validators know that the upgrade block height may…

Excerpt (1197 of 3886 characters). Read the whole post on the forum ↗

@lexa one more point I raised on Discord: is it possible to launch an upgrade proposal on chain way more earlier? My reasoning: I use a monitoring that generates metrics for all my nodes if there’s an upcoming upgrade (e.g. proposal that has passed) which I use for building 2 alerts: 1) to fire if I do not have a binary prepared in Cosmovisor folder, and 2) if an upgrade is in less than 30 minutes, to remind me to be present. If I made a mistake somehow (for example if I put the binary into a wrong folder), or for example if the proposal name somehow doesn’t match the one specified in README/instructions provided by the team (so the binary should be put into a different folder), I won’t have enough time to react and fix things accordingly. On mainnet there’s usually a big timespan between the date the proposal is accepted and the actual chain upgrade, this time on testnet it was merely a few minutes, so here 1) it doesn’t match the mainnet conditions and 2) it leaves really short time for validators to react. Do you think considering what I wrote above it’s possible to submit such proposals, let’s say one day before the estimated time of the block it’s supposed to be executed…

Excerpt (1199 of 1203 characters). Read the whole post on the forum ↗

The upgrade proposal for consumer chains is managed by the consumer chain team, which is how it would be on mainnet. I can work with them on committing to putting on testnet-chain earlier. You’re right – we want to be letting people use scripts and monitoring the way they do on mainnet, and mainnet gets way more lead time than testnet.

Occasionally we do try to run two events on the same day, in which case I’d aim to have the highest priority event submit their proposal earlier and then have the second event waiting in the wings once we know our block drift won’t cause the events to collide.

Really appreciate the feedback

Today’s Testnet Wednesday was delayed as validators and testnet coordinators were on deck to execute and support an emergency upgrade for mainnet `neutron-1`. neutron-1 emergency upgrade The Neutron team coordinated an emergency upgrade for `neutron-1` to include a security-critical patch to cosmwasm. The Neutron team used a two-stage plan: • Distribute a halt height to the validator community • Distribute the upgrade information, including the new release binary link By distributing the halt height first, they allowed validators to prepare their nodes and be aware of the upcoming upgrade without even having a new release available. `neutron-1` halted at 10:04:35 ET and returned at 10:08:05, after a 3m 30s downtime. amazing work all around, especially from the Neutron team coordinating this upgrade! pion-1 emergency upgrade • The pion-1 upgrade was scheduled right after it was determined that the mainnet (neutron-1) chain upgrade was complete • The halt height was set to 9723450, at approximately 16:45 UTC (11:45 ET) • A signaling proposal was submitted to the provider chain of the Replicated Security testnet to use this event as eligibility criteria for…

Excerpt (1197 of 2443 characters). Read the whole post on the forum ↗

We had a particularly busy Testnet Wednesday this week. Both Neutron and Aether got their proposals in earlier in the week and we anticipated upgrading `pion-1` first at ~15:30UTC and then launching Aether at 16:00 UTC. Hypha distributed voting power more on Tuesday and this slowed down `pion-1`’s block time by about 0.2s, making the upgrade time closer to 17:00 UTC. This is expected behaviour – a more decentralized set means that the proposers are more geographically distributed and so communication takes longer. Lesson learned for us – don’t make changes to voting power distribution just before an upgrade. As a result, we launched Aether’s `ethera_9000-1` chain first. `ethera-9000-1` chain launch 46/61 validator participation • 13:44 UTC: Hypha announces that we’ll launch `ethera_9000-1` at 15:30 UTC • 14:00 UTC: spawn time is reached and CCV is generated, validators begin joining • 15:30 UTC: Hypha brings validators online • 15:32 UTC: Blocks are being produced and chain is interchain-secured This was a very efficient rehearsal with a few notes: • The Aether chain requires the `chain-id` flag to be used in the start command, whereas previous chains have…

Excerpt (1195 of 2392 characters). Read the whole post on the forum ↗

We ran our first-ever game day on the testnet today! In this game day, we investigated two new ideas on a brand new chain: • Cancelling a software upgrade proposal using a governance-gated `cancel-upgrade` proposal • Skipping an upgrade using the `--unsafe-skip-upgrades <block-height>` flag We had 39 validators participating today on this new `gameday01` chain. cancel-upgrade For our first event, we submitted a proposal to upgrade from Gaia `v13.0.2` to `v14.0.0-rc0` and then cancelled it using a `cancel-upgrade` proposal which validators had to vote on. 31/39 validators voted successfully on the `cancel-upgrade` proposal and we avoided upgrading to `v14.0.0-rc0`. However – this is not a realistic scenario for the Hub. Because the Cosmos Hub voting period is 2 weeks and we usually upgrade roughly 1 week after an upgrade proposal passes, there is no time for a `cancel-upgrade` proposal before the upgrade height arrives. On mainnet, it’s much more practical to just skip the upgrade height. unsafe-skip-upgrades For our second event, we pretended that the issue had been resolved and we were ready to upgrade to `v14.0.0` via a software upgrade proposal. As…

Excerpt (1196 of 4290 characters). Read the whole post on the forum ↗

February 7, 2024 Lately there has been a ton of spam on the Hub getting vetoed. Historically, it hasn’t been really clear which outcomes lead to a deposit being burned so we decided to do a demo day of different proposal outcomes. Proposals on the Hub can have five outcomes: • PASS: Quorum achieved, >50% of participating voting power votes YES • REJECT: Quorum achieved, >50% of participating voting power votes NO or NWV • VETO: Quorum achieved, >33.3% of participating voting power votes NWV • INCONCLUSIVE: Quorum achieved, neither YES/NO side achieves >50% of participating voting power • NO QUORUM: Quorum not achieved We ran five different proposals, each intended to showcase a different voting outcome and how the deposit was handled. Each proposal was submitted by a unique wallet that was initially funded with 11 ATOM (for a 10 ATOM deposit on each proposal). • Vote YES on Proposal 101 • Vote NO on Proposal 102 • Vote NOWITHVETO on Proposal 103 • Vote ABSTAIN on Proposal 104 • Do NOT vote on Proposal 105 Kinda neat to see all those voting options show up so clearly in the explorers! Outcomes resulting in a returned deposit: PASS Wallet…

Excerpt (1194 of 2262 characters). Read the whole post on the forum ↗

February 14, 2024 v15 is coming to the Hub ! This is going to be a big one because of state migrations (executing on decisions made in proposals 826 and 860 ) and moving to sdk 0.47. On the testnets, we upgraded `theta-testnet-001` on Tuesday and did `provider` today to ease the load on validators operating on both networks. We had 41 validators participating on `provider`. • Theta runtime: 55 minutes • Provider runtime: 41 minutes Lessons learned Major feedback from operators who have now gone through this upgrade process at least once (some went through it twice): • Don’t panic at the long wait and lack of updates in the logs! Logs are not very verbose and it will look like nothing is happening, but just be patient. Hypha and Informal are working with the Comet team to see about getting more informative logs during upgrades in the future. • Do not restart your node mid-upgrade unless you’re seeing OOM errors. Having learned from our big Neutron upgrade, we know that restarting mid-upgrade can make the situation worse and lead to a corrupted database. • Set min gas price to `0.005uatom` or your node will fail on restart. Failing on restart just means that…

Excerpt (1194 of 1536 characters). Read the whole post on the forum ↗

February 21, 2024 Today we did a coordinated upgrade to bump provider to rc1, then went through a changelog review game day. Minimum deposit ratio Deposits to proposals in v15 must be made according to a minimum deposit ratio of 0.01 (i.e., the minimum amount that can be sent to a proposal in the deposit period is 10% of the total minimum deposit). Parameter change proposals • Gaia-specific parameters such as `globalfee` can be changed can still be changed via a `submit-legacy-proposal` and take effect. • Parameters from the sdk (such as those within the `staking` module) cannot be changed using `submit-legacy-proposal` • Parameters from the sdk can be changed using `submit-proposal` However, parameter change proposals now require the submitter to list all parameters in the module and the value of each parameter. Previously, parameter change props could be made and name a single param to adjust. This represents a risk to the voting process as validators and voters need to confirm that a parameter change proposal does not accidentally or maliciously change a parameter which isn’t clearly communicated in the proposal text. This can be observed in Proposals 112…

Excerpt (1197 of 2548 characters). Read the whole post on the forum ↗

Unsure if this is the place to do this, but i want to say cudos again to the Hypha team here. Our (Citizen Web3) Devops has been visiting the events for 2 weeks and has only the most positive feedback about what is being done there. If you want to improve your understanding of gaia/gaiad, this is a must!

I second the statement above, as I’ve been a testnet member for over than 2 months already.

Firstly, it’s really a great playground for validators to learn to deal with things in unexpected consequences and therefore be more prepared for things going wrong way on mainnet.

Secondly, it’s also a great way to see the features that would soon land on mainnet beforehand and know what’s coming in the future onto a mainnet and how to use it.

Lastly, I cannot say anything except the positive things on how the whole thing is coordinated, both from Hypna’s side (as they were extremely helpful and willing to listen to validators’ suggestions on improving the whole project and also willing to help those in need) and from validators’ side (everybody is super helpful and I’ve seen a lot of validators helping others in need).

Happy to be the part of it and hope some of my support was also helpful. Way to go, guys!

March 6, 2024

With two emergency upgrades this week, I’m a bit late getting this report out
Fortunately both Neutron and Hub upgrades went super smoothly – hardly a noticeable halt!

On the testnet front, we had a smooth scheduled upgrade of pion-1` to v3.0.0. The upgrade itself took < 30 minutes and we had 35 validators participating. We also had a new Hypha member running the show behind the scenes with Dante off on a much-deserved vacation

March 13, 2024 Today we ran through a fairly simple demo day on two topics: • Assigning consumer keys • Setting a security contact Consumer key assignment In the early days of ICS, validators were encouraged to either assign a consumer key or reuse their provider key for consumer chains. That recommendation has changed! It is now only recommended to assign a consumer key and use a unique key for each chain. Never reuse keys, and keep track of which keys are used for which chains. To reinforce this, we demonstrated key assignment on `pion-1` The transaction to assign a consensus key to a consumer chain is in the provider module: ``` gaiad tx provider assign-consensus-key pion-1 <consumer pubkey> ``` Of the 56 testnet validators, 42 have now assigned a consumer key. Prior to this demo day, it was only 26 Set security contacts Setting a security contact is a lightweight way to verify that an email or social media handle is approved by the keyholders of your validator node, making it easier to speak for your validator without having to touch your keys. Make sure your security contact is up to date though – letting it go stale could mean being you’re being…

Excerpt (1194 of 1733 characters). Read the whole post on the forum ↗

I forgot to post last week so we have a double update today!

March 27, 2024

We packed a ton into this week:

1. Upgrade to v15.1.0 to keep up with mainnet
2. Game day: bond shares to your validator
3. Game day: tokenize shares to obtain liquid staked tokens
4. Game day: (optional) collect as many liquid tokens as possible!

Validators worked through the game day tasks, gaining familiarity with the transactions used to bond and tokenize shares, with samples given in our repo here.

Items 2 and 3 on this list were TIP criteria and we had 31 validators pass.

April 3, 2024

This week we upgraded to v15.2.0-rc0 on both provider and theta-testnet-001. This version patches a governance issue in which the proposal length is limited to 255 characters, the sdk default for 0.47. Lately we’ve seen some proposals on the Hub that include just a link to IPFS for the full proposal!

We expect to see a coordinated upgrade on mainnet to address this next week.

As this was a minor coordinated upgrade on testnet, it did not count for TIP criteria.

April 17, 2024

pion-1 upgrades to v3.0.4

April 24, 2024

provider upgrades to v.16.0.0-rc2

v16 has an active post on the forum and incorporates ICS epochs, a critical part of Partial Set Security.

[PROPOSAL 914][PASSED] Gaia v16 Software Upgrade Software Upgrade

Change log: 2024-4-20 Created initial post 2024-4-26 Revised title for on-chain proposal & added link to release 2024-5-13 Revised title for passed vote Background The Gaia v16 release is a major release that will follow the standard governance process by initially submitting this post on the Cosmos Hub forum. After collecting forum feedback (~ 1 week) and adapting the proposal as required, a governance proposal will be sent to the Cosmos Hub for voting. The on-chain voting period typically l…

With ISLE concluded, we’re back on the regular Testnet Wednesday schedule! TIP has been renewed by AADAO and we’ve adjusted our criteria and event planning to be more lenient in some ways (no need to do things perfectly every week – showing up and trying is enough) and stricter in others (you must assign consumer keys on mainnet now). Read more here ! May 29, 2024: ICS 2.0 Validator Set Cap Per our adjusted criteria, we’re making events more open-ended for how validators can participate. As long as validators are present and able to be recorded (voting on a proposal, signing blocks, sending a particular tx), we’ll count their work towards TIP eligibility. For this event, we launched the `test-valsetcap-1` chain and validators had to vote YES on the launch proposal in order to be officially counted. We demonstrated one of ICS 2.0’s power shaping mechanisms: the validator set cap. Capping the size of the validator set : The consumer chain can specify a maximum number of validators it wants to have in its validator set. This can be used to limit the number of validators in the set, which can be useful for chains that want to have a smaller validator set for faster…

Excerpt (1193 of 2606 characters). Read the whole post on the forum ↗