Polygon’s zkEVM rollup encountered an emergency situation on March 23 when the chain ceased processing blocks. Polygon recently released an analysis detailing the outage that affected its zkEVM network.
Late Saturday, Polygon disclosed that its zkEVM sequencer had experienced a failure, attributing it to a reorganization of the Ethereum mainnet.
Polygon zkEVM Mainnnet Beta faced an issue with its sequencer due to an L1 reorg. We are working on fully resolving the issue and will follow up with a detailed post-mortem.
This only impacts Polygon zkEVM (which is the only rollup) and does not impact Polygon PoS, Polygon CDK,…
— Polygon | Aggregated (@0xPolygon) March 23, 2024
The postmortem revealed that the synchronizer for Polygon zkEVM mishandled the reorg, causing the chain to process batches of transactions with incorrect timestamps.
A blockchain reorganization, or reorg, occurs when a network rejects blocks from a previous version and adopts a new chain of blocks as the accurate history of transactions. This can happen when different parts of the network temporarily disagree on recent transaction order or validity. While reorgs help ensure network security and consistency, they can cause temporary uncertainties in transaction histories.
The failure of Polygon zkEVM’s sequencer, responsible for ordering transactions and assigning timestamps, led to the processing of transactions based on outdated information. This resulted in inaccuracies, disrupting the network’s ability to maintain a consistent ledger.
During the incident, several transactions were processed as no-operation, essentially being ignored by the network. Synchronizing these empty transactions contributed to the prolonged downtime of the network.
Overall, it took approximately 14 hours for the rollup to resume withdrawals, with around 4,000 transactions potentially affected by the outage. Notably, only Polygon zkEVM was impacted, as it is in a mainnet beta phase with potential bugs and availability issues, unlike Polygon’s PoS layer-1 and chains using its CDK.
Polygon’s zkEVM, a zero-knowledge Ethereum Virtual Machine, utilizes zero-knowledge proofs to verify transactions before settling batches of transactions on Ethereum. StarkWare and zkSync are also developing similar products. This differs from layer-2 solutions like Arbitrum and Optimism, which operate as optimistic rollups without zero-knowledge proofs.
Following the outage, Polygon’s zkEVM entered an emergency state, allowing an upgrade to the network’s prover and verifier to be implemented alongside a fix for the sequencer malfunction during reorgs.
Upon completing the upgrade, the emergency state was deactivated, subjecting further system updates to a 10-day timelock before taking effect. This marked the first use of Polygon’s zkEVM emergency state feature, requiring approval from the network’s Security Council, operating as a 6/8 multisig group composed of reputable members of the Ethereum community.
The outage highlights ongoing centralization concerns across Ethereum layer-2 networks, according to Jarrod Watts from Polygon Labs.
What happened with Polygon zkEVM over the weekend is a great discussion point for Layer 2 upgradeability.
> There is an issue
> zkEVM stops producing blocks
> Security council sets emergency state to true (to bypass 10d wait period for upgrades)
> Team fixes the issue via…— Jarrod Watts (@jarrodWattsDev) March 25, 2024
Despite these challenges, Polygon zkEVM aims to enhance its offerings with upcoming upgrades, including support for EIP-4844’s “blobspace” following its “Feijoa” upgrade.
