Skip to main content

57 posts tagged with "consensus"

View All Tags

· 2 min read
Damian Nadales

High level summary

  • Debunked our working theory on the cause of performance degradation when taking a ledger snapshot. We are now back to the UTXO set as the first contributing cause to said degradation, and together with the Ledger team we have proposed a way decrease the number of allocations when serializing the ledger state.
  • Developed the first and second draft scripts for estimating the bandwidth necessary to ensure the CPU is the bottleneck when syncing (#1240). This is informing us and the Networking Team how to refine BlockFetch for the syncing node (especially for Genesis).
  • On the UTXO-HD front:
    • After addressing several issues found during benchmarking and testing, the performance team ran benchmarks on the utxo-hd-9.1 branch, yielding positive results. The nodes function without errors. The memory and CPU usage is almost on par with the 9.1 node.
    • A tool has been provided to convert ledger state snapshots from pre-UTxO-HD nodes to UTxO-HD nodes, allowing users to use UTxO-HD right away without needing to replay the chain (since they can use their locally stored ledger state after converting it with the aforementioned tool).
    • The SDET team will run integration tests on the utxo-hd-9.1 branch. If the tests pass, we will start working on wrapping up the documentation and preparing the branch for merging once it is decided to release this feature.
    • Bear in mind that:
      • This UTxO-HD release uses an LMDB backend (but it also provides an in-memory backend). The LSM-tree backend should arrive Q1 2025.
      • UTxO-HD is just the first step of a bigger initiative for moving parts of the ledger state to the disk storage, lowering the memory requirements of the node and contributing to long term sustainability of Cardano.

· One min read
Kevin Hammond

Unexpected Ledger State Replay in the Conway era

An issue was identified shortly before the Chang hard fork: it was found that ledger state snapshots would break ledger replay in the Conway era under mainnet conditions. The ledger and consensus teams worked rapidly to resolve the issue with a hotfix released within 24 hours of the hard fork. In order to avoid pauses in node availability, it was recommended that users should not restart their node process until they had upgraded their node to the hotfix - this included any node type: relays, block producers, DB-Sync nodes, etc.

The issue is documented here. The cause was a slight inconsistency between the ledger state snapshots that were written and those that could be read back; a side effect of the removal of pointer addresses in the Conway era. Nodes version 9.1.1 and later resolve this issue.

Further Details

Node version 9.1.1

GitHub Issue

· 2 min read
Kevin Hammond

Issue Caused by Large Reference Scripts on Cardano Mainnet

On 25th June 2025, a Cardano user inserted a series of transactions, each containing 194 large reference scripts onto the mainnet chain, funded from 3 wallets containing around 20K ada each. High deserialisation costs for these reference scripts impacted the node, resulting in network disruption to block producer nodes, an increase in network load, and some slowdown in transaction throughput.

The direct effect lasted about 12 hours until it was stopped by a community member, at a cost of 4603 ada to the user who had created the transactions. Overall, the network responded extemely well to the increased load, showing a high level of resilience, with some reduction in transaction throughput related to the overall high system load. The community response to the event was positive, praising the speed of response, the robustness of the Cardano network, the cohesion of the Cardano community, and its ability to diagnose and manage issues such as this.

Mitigations Deployed

The general issue had already been identified, and a mitigation (costing for reference scripts) had been prepared as part of the Chang hard fork, but not yet deployed to mainnet. Based on the event, stronger mitigations were prepared, including restricting large reference scripts, and changing the cost model. These mitigations were deployed via node versions 8.9.4, 8.12.1 or 8.12.2, and incorporated into node version 9.0.0 or later for the Chang hard fork.

Public Reports on the Incident

Coindesk Report

Nasdaq Report

· 2 min read
Kevin Hammond

Blocks from the future

We identified two issues relating to "blocks from the future".

  1. Blocks from the near future
  2. Blocks from the far future

While blocks from the near future have been known to occur on mainnet as a result of clock skew/misconfiguration, there are no known instances on mainnet of blocks from the far future. In both cases, restarting an affected node would resolve the issue.

What is meant by a Block from the Future?

A node considers a block to be from the future if its slot is ahead of the current slot. Ouroboros Praos mandates that all chains containing blocks from the future (at that time) are ignored during chain selection. As Praos assumes that all nodes have access to perfectly synchronized clocks, this will never cause nodes to disregard blocks that have been minted by other honest nodes. In an actual real-world deployment, this assumption is unrealistic due to the imperfections of protocols like NTP as well as leap seconds.

The issues that were identified meant that blocks from the future could potentially be used by malicious actors to create denial-of-service attacks.

Both issues were fixed by Cardano node 8.8 or later, and were eradicated at the Chang hard fork.

Further Details

Report on Blocks from the Near Future

Report on Blocks from the Far Future

· One min read
Damian Nadales

High level summary

  • Added a snapshot-converter tool, which will be merged soon. This tool converts non-UTXO-HD ledger snapshots, into UTXO-HD ones, so that the user does not have to replay from Genesis when using an UTXO-HD capable node. This patch also solves an issue with deserialization of TxOuts in Conway in the UTxO-HD implementation.
  • Solved a memory leak in the UTxO-HD implementation. This patch will be benchmarked this week.
  • Wrote a test for adding large txs to the mempool.
  • Expanded the Mempool capacity beyond just byte size.
  • @amesgen discovered and advised on a Conway ledger snapshot deserialization bug.