Nonce account migration and re-enabling durable nonce transactions

[jstarry]

Problem

Durable nonce transactions were disabled after they contributed to the mainnet-beta outage on June 1st. Under certain circumstances, durable nonce transactions could be re-processed if they previously failed. For more details, read the outage report here: https://solana.com/news/06-01-22-solana-mainnet-beta-outage-report-2

Before re-enabling durable nonce transactions, the re-processing issue needs to be fixed.

Proposed Solution

1. Ensure that new nonce transactions cannot be processed as normal transactions by no longer using raw blockhashes for durable nonces: separates durable nonce and blockhash domains #25744

   1. Since nonces are already versioned, introduce a new version of the nonce state which uses hash("NONCE_BLOCKHASH", recent_blockhash) as the nonce hash.
   2. Introduce new features called separate_nonce_from_blockhash and enable_durable_nonce
   3. When the separate_nonce_from_blockhash feature is activated:
      1. also activate the enable_durable_nonce feature unless another nonce issue is identified
      2. require that durable nonce transactions make use of the new nonce version to be processed.
      3. support a new system program nonce instruction called UpgradeNonceAccount which can be executed by anyone to migrate a nonce account from the old version to the new version.

2. Ensure that past transactions cannot be processed as nonce transactions by rejecting any durable nonce transaction that uses a raw blockhash: permanently disables durable nonces with chain blockhash domain #25788

3. Ensure that nonce transactions always advance the nonce hash so that they cannot be processed again

   1. Enforce that nonce hash is different from what it will be advanced to: Reject durable nonce txs that don't use an advanceable nonce #25832
   2. Enforce that nonce account is writable: Nonce accounts must be writeable #21260

4. Ensure that nonce accounts can only be advanced by the nonce authority: Reject durable nonce transactions not signed by authority #25831

Rollout Progress

• Audits (scheduled to be finished by Friday, June 10th)

  • separates durable nonce and blockhash domains #25744
  • permanently disables durable nonces with chain blockhash domain #25788
  • adds system instruction to upgrade legacy nonce versions #25789
  • Reject durable nonce transactions not signed by authority #25831
  • Reject durable nonce txs that don't use an advanceable nonce #25832
  • Publish audits to: https:/solana-labs/security-audits

• Release the patches in a new v1.9 and v1.10 release [v1.9.29 and v1.10.25 have all patches]

  • separates durable nonce and blockhash domains #25744
  • permanently disables durable nonces with chain blockhash domain #25788
  • adds system instruction to upgrade legacy nonce versions #25789
  • Reject durable nonce transactions not signed by authority #25831
  • Reject durable nonce txs that don't use an advanceable nonce #25832

• Activate features on testnet and devnet clusters

  • Feature Gate: Nonce accounts must be writeable #25837
  • Feature Gate: separate durable nonce and blockhash domains #25772
  • Feature Gate: Reject durable nonce txs that don't use advanceable nonce #25836
  • Feature Gate: Reject durable nonce transactions not signed by authority #25835
  • Feature Gate: enable durable nonce #25773

• Wait for 95% of mainnet-beta staked validators to upgrade to v1.9.29 / v1.10.25

  • Latest adoption progress: 96.63%

• Wait for mainnet-beta RPC nodes to upgrade to v1.9.29 / v1.10.25

  • Latest adoption progress: 88%

• Activate features on mainnet-beta cluster

  • Feature Gate: Nonce accounts must be writeable #25837
  • Feature Gate: separate durable nonce and blockhash domains #25772
  • Feature Gate: Reject durable nonce txs that don't use advanceable nonce #25836
  • Feature Gate: Reject durable nonce transactions not signed by authority #25835
  • Feature Gate: enable durable nonce #25773

• Automate nonce upgrades on the testnet cluster

• Automate nonce upgrades on the devnet cluster

• Automate nonce upgrades on the mainnet-beta cluster

Communication

• The CLI doesn't support versioned nonces so users will need to update their CLI version to the latest release otherwise commands like solana nonce and solana nonce-account will not work correctly after nonce accounts are upgraded
• All past created durable nonce transactions will be invalidated. Users that have pre-signed a durable nonce tx with the intent of sending at a future date will need to drop pre-signed transactions and sign new transactions.
• All past queried nonce account blockhash values will be invalidated. Users that have cached a nonce account blockhash to use for creating a durable nonce transaction at a future date will need to drop cached blockhashes and cache the upgraded nonce account blockhash instead.
• All clients that do client-side parsing of nonce account data and require the version to be 0 will need to be updated to properly parse nonce accounts that have been upgraded to version 1. The data layout is otherwise unchanged. (Note that the web3 NonceAccount.fromAccountData method was implemented to be forwards compatible and is not impacted)

[CriesofCarrots]

An alternative to the UpgradeNonceVersion instruction would be to wholesale rewrite/upgrade all the old nonce accounts on the epoch boundary when nonce_version_upgrade activates. The advisability of this approach probably depends on how many such accounts there are.

[CriesofCarrots]

Trent informed me there are ~60k, so that's probably a deal-breaker for that alternative ^

[jstarry]

An alternative to the UpgradeNonceVersion instruction would be to wholesale rewrite/upgrade all the old nonce accounts on the epoch boundary when nonce_version_upgrade activates. The advisability of this approach probably depends on how many such accounts there are.

I think it's better to avoid runtime specific changes whenever possible

[nguyenkha]

Hi, I have a question

support a new system program nonce instruction called UpgradeNonceAccount which can be executed by anyone to migrate a nonce account from the old version to the new version.

Is anyone must be the nonce account authority? or anyonce can upgrade any nonce account?

[jstarry]

Is anyone must be the nonce account authority? or anyonce can upgrade any nonce account?

The authority will not need to sign for the nonce account upgrade. Anyone can upgrade any nonce account and the upgrade can only happen once.

[steveluscher]

This is awesome. Really excited for this barrage of fixes.

I've read through this plan, and most of the source, and have concluded that – except for maybe wanting to add the UpgradeNonceAccount instruction, nothing much has to change in the web3.js library; is that a fair assessment? From the client's perspective, it's just ferrying a value in and out of the account. Whether it's a recent blockhash or a sha256 hash makes no difference from the perspective of signing and confirming transactions, right?

[jstarry]

I've read through this plan, and most of the source, and have concluded that – except for maybe wanting to add the UpgradeNonceAccount instruction, nothing much has to change in the web3.js library; is that a fair assessment? From the client's perspective, it's just ferrying a value in and out of the account. Whether it's a recent blockhash or a sha256 hash makes no difference from the perspective of signing and confirming transactions, right?

I don't think any web3 changes are needed. The explorer could be updated to parse the new instruction though. It makes use of the parsed RPC response rather than relying on web3 parsing methods.

[ptrdsh]

Heya! thx for tracking this so nicely.
In our cold storage infrastructure, which stores the asset keys on an HSM and solves multisig quorum approvals off-chain with approval devices signing approvals for the unsigned raw tx, we have to rely on durable nonces, because 2min is very tight between TxRequest to approve and HSM asset key signature plus broadcasting.. we are quite stuck because of this.

1. Any guesstimate ETA on re-enabling durable nonce txs (Feature Gate: enable durable nonce #25773)? Ack on today's plan to finish audits. Something like "next week" or "next month" would be accurate enough. Are there maybe known workarounds?
2. Ive read the communication above and ack client-side parsing of nonce acc data, but "UpgradeNonceAccount" sounds as if there might be something else not backwards compatible. When will the docu be updated and will there be a "migration" guide, should there be other needed changes?

[aeyakovenko]

@ptrdsh audits are done, releases and feature activation will start next week, but each one needs an epoch.

[behzadnouri]

@ptrdsh audits are done, releases and feature activation will start next week, but each one needs an epoch.

@aeyakovenko that is no longer necessary. We can activate them all at the same epoch.

Ive read the communication above and ack client-side parsing of nonce acc data, but "UpgradeNonceAccount" sounds as if there might be something else not backwards compatible.

@ptrdsh The main change is that the nonce account cannot be used for durable transactions until it is advanced once. If you execute either of AdvanceNonceAccount or UpgradeNonceAccount instruction on the nonce account, then all is good to use that nonce account for durable transactions.

[KobeBlockify]

Are durable nonces activated or not yet?

[behzadnouri]

Are durable nonces activated or not yet?

They are activated on testnet and devnet.
Once the mainnet cluster upgrade to recent releases, we can activate the feature on mainnet as well.

[jstarry]

Durable nonce transactions have been re-enabled on mainnet-beta and also fixed on devnet and testnet clusters. Security audit reports should be publicly viewable soon.