56 posts tagged with "consensus"

Unexpected Ledger State Replay in the Conway era​

An issue was identified shortly before the Chang hard fork: it was found that ledger state snapshots would break ledger replay in the Conway era under mainnet conditions. The ledger and consensus teams worked rapidly to resolve the issue with a hotfix released within 24 hours of the hard fork. In order to avoid pauses in node availability, it was recommended that users should not restart their node process until they had upgraded their node to the hotfix - this included any node type: relays, block producers, DB-Sync nodes, etc.

The issue is documented here. The cause was a slight inconsistency between the ledger state snapshots that were written and those that could be read back; a side effect of the removal of pointer addresses in the Conway era. Nodes version 9.1.1 and later resolve this issue.

Further Details​

Node version 9.1.1

GitHub Issue

Issue Caused by Large Reference Scripts on Cardano Mainnet​

On 25th June 2025, a Cardano user inserted a series of transactions, each containing 194 large reference scripts onto the mainnet chain, funded from 3 wallets containing around 20K ada each. High deserialisation costs for these reference scripts impacted the node, resulting in network disruption to block producer nodes, an increase in network load, and some slowdown in transaction throughput.

The direct effect lasted about 12 hours until it was stopped by a community member, at a cost of 4603 ada to the user who had created the transactions. Overall, the network responded extemely well to the increased load, showing a high level of resilience, with some reduction in transaction throughput related to the overall high system load. The community response to the event was positive, praising the speed of response, the robustness of the Cardano network, the cohesion of the Cardano community, and its ability to diagnose and manage issues such as this.

Mitigations Deployed​

The general issue had already been identified, and a mitigation (costing for reference scripts) had been prepared as part of the Chang hard fork, but not yet deployed to mainnet. Based on the event, stronger mitigations were prepared, including restricting large reference scripts, and changing the cost model. These mitigations were deployed via node versions 8.9.4, 8.12.1 or 8.12.2, and incorporated into node version 9.0.0 or later for the Chang hard fork.

Public Reports on the Incident​

Coindesk Report

Nasdaq Report

Blocks from the future​

We identified two issues relating to "blocks from the future".

1. Blocks from the near future
2. Blocks from the far future

While blocks from the near future have been known to occur on mainnet as a result of clock skew/misconfiguration, there are no known instances on mainnet of blocks from the far future. In both cases, restarting an affected node would resolve the issue.

What is meant by a Block from the Future?​

A node considers a block to be from the future if its slot is ahead of the current slot. Ouroboros Praos mandates that all chains containing blocks from the future (at that time) are ignored during chain selection. As Praos assumes that all nodes have access to perfectly synchronized clocks, this will never cause nodes to disregard blocks that have been minted by other honest nodes. In an actual real-world deployment, this assumption is unrealistic due to the imperfections of protocols like NTP as well as leap seconds.

The issues that were identified meant that blocks from the future could potentially be used by malicious actors to create denial-of-service attacks.

Both issues were fixed by Cardano node 8.8 or later, and were eradicated at the Chang hard fork.

Further Details​

Report on Blocks from the Near Future

Report on Blocks from the Far Future

High level summary​

• Added a snapshot-converter tool, which will be merged soon. This tool converts non-UTXO-HD ledger snapshots, into UTXO-HD ones, so that the user does not have to replay from Genesis when using an UTXO-HD capable node. This patch also solves an issue with deserialization of TxOuts in Conway in the UTxO-HD implementation.
• Solved a memory leak in the UTxO-HD implementation. This patch will be benchmarked this week.
• Wrote a test for adding large txs to the mempool.
• Expanded the Mempool capacity beyond just byte size.
• @amesgen discovered and advised on a Conway ledger snapshot deserialization bug.

High level summary​

During the past week the team:

• Incorporated minor improvements to the ChainSync client test (#529).
• Documented tasks of a caught-up node (#1215), which can be useful for SPOs.
• Tweag presented the last Genesis SoW to the Consensus team. The next steps are reviews and phased (opt-in) rollout.

Regarding the two problems found during UTXO-HD benchmarks, namely, increase in heap size (#1192) and a newly found race condition (#1193), #1208 fixed the race condition and was merged, however #1194 showed no improvements so it will not be merged yet.