4 posts tagged with "incident"

Unexpected Ledger State Replay in the Conway era​

An issue was identified shortly before the Chang hard fork: it was found that ledger state snapshots would break ledger replay in the Conway era under mainnet conditions. The ledger and consensus teams worked rapidly to resolve the issue with a hotfix released within 24 hours of the hard fork. In order to avoid pauses in node availability, it was recommended that users should not restart their node process until they had upgraded their node to the hotfix - this included any node type: relays, block producers, DB-Sync nodes, etc.

The issue is documented here. The cause was a slight inconsistency between the ledger state snapshots that were written and those that could be read back; a side effect of the removal of pointer addresses in the Conway era. Nodes version 9.1.1 and later resolve this issue.

Further Details​

Node version 9.1.1

GitHub Issue

Issue Caused by Large Reference Scripts on Cardano Mainnet​

On 25th June 2025, a Cardano user inserted a series of transactions, each containing 194 large reference scripts onto the mainnet chain, funded from 3 wallets containing around 20K ada each. High deserialisation costs for these reference scripts impacted the node, resulting in network disruption to block producer nodes, an increase in network load, and some slowdown in transaction throughput.

The direct effect lasted about 12 hours until it was stopped by a community member, at a cost of 4603 ada to the user who had created the transactions. Overall, the network responded extemely well to the increased load, showing a high level of resilience, with some reduction in transaction throughput related to the overall high system load. The community response to the event was positive, praising the speed of response, the robustness of the Cardano network, the cohesion of the Cardano community, and its ability to diagnose and manage issues such as this.

Mitigations Deployed​

The general issue had already been identified, and a mitigation (costing for reference scripts) had been prepared as part of the Chang hard fork, but not yet deployed to mainnet. Based on the event, stronger mitigations were prepared, including restricting large reference scripts, and changing the cost model. These mitigations were deployed via node versions 8.9.4, 8.12.1 or 8.12.2, and incorporated into node version 9.0.0 or later for the Chang hard fork.

Public Reports on the Incident​

Coindesk Report

Nasdaq Report

Security Issue Report: SECP256k1 bug​

Date Occurred: July 15, 2022 Severity: Potentially Very High if exploited on Mainnet Authors: Iñigo Querejeta Azurmendi

Date of Report: August 17, 2023

Summary of Issue​

Criticality Level​

Actually low (since the issue was detected prior to deployment) but potentially very high if it had been deployed to mainnet Context

New SECP256k1 Plutus bindings were being introduced in order to support interoperability with other major chains, such as Bitcoin and Ethereum. The intention was to deploy these as part of the Vasil hard-fork. The bindings were considered to be a low-security risk since the underlying library functions were well tested and had been deployed on other blockchains. How was the Issue Detected

The issue was detected via specific End-to-End tests that had been commissioned. It was (accidentally) triggered on the Cardano Testnet before a fix could be deployed there.

What Action was Taken​

The Cardano Testnet was permanently halted, and new test environments were deployed (Preview and Pre-Prod). Fixes were applied to prevent the use of the primitives. A full security audit was carried out on the bindings. The rollout of the primitives was postponed to a new hard fork (Valentine)

Potential Effect​

The potential effect was that an adversary might be able to craft invalid Plutus transactions to crash any node, requiring execution of the Cardano disaster recovery plan to revert to a safe state and bypass the transaction.

Actual Effect​

• Delay to the Vasil hard-fork
• Temporary removal of SECP256k1 primitives
• Additional hard-fork to introduce SECP256k1 primitives

Ongoing Mitigations Needed, if any​

None

Responsibility for Mitigations​

Core team

Detailed description of Incident​

New Plutus secp256k1 cryptographic primitives for Plutus v2 failed to apply the necessary validity checks on the input data, meaning that the primitives could theoretically be used in an unsafe environment. The vulnerability was present in recent node versions (1.35.0 onwards), including ones deployed to Cardano Testnet.

The problem was not in the deserialization functions of the underlying library (Bitcoin's library) but rather that the Haskell functions that implemented the Plutus builtins were not calling them correctly. In particular, the library functions were designed to take structured data as input. However, the Haskell FFI implementation that was produced for the Plutus builtins allowed a caller to pass in (possibly) unstructured data. There were no checks that these data were structured in the correct way. This issue was detected during End-to-End testing.

• This is the ECDSA signature verification algorithm that was used. It takes a SECP256k1_pubkey as input. That type is an opaque type with an expected structure: a parsed and valid public key. It was not immediately obvious that structured data needed to be passed to allow the function to be used safely.
• The same happened with the Schnorr verification function. It takes as input a SECP256k1_xonly_pubkey, which is again an opaque structure that holds a parsed and valid public key.

The FFI skipped checks over these structured keys and directly passed the raw bytes that were given as arguments. If an adversary were to pass in data that was not properly structured, then it could result in unexpected behavior of the library. This could perhaps translate into an adversary being able to crash the nodes that ran these functions. All nodes in the network could be crashed by a single transaction that would then be executed repeatedly, so stalling the network until the disaster recovery process was initiated.

The fix was addressed in this PR. It consisted of using the external representation that the deserialization function expects and running the deserialization prior to signature verification. This was audited by security experts.

Recommendations​

• Check all new Plutus bindings for correct use.
• Audit all new Plutus built-in bindings.
• Continue to develop specific End-to-End tests for all new Plutus features.
• Do not assume that any existing library functions are "safe". Treat all external calls circumspectly.

Incident reporting: Cardano block production temporary outage​

On Sunday, January 22, 2023, an incident occurred resulting in block production pausing for a brief period of time (approximately two minutes, similar to the usual pause at an epoch boundary). Around 50% of block-producing nodes and relays restarted during this period. Having restarted, nodes continued to produce blocks without failure. While the network continued to operate, the issue did have the potential to affect network integrity, so was flagged as a ‘critical’ incident, thus warranting immediate response and investigation by IOG engineers. The investigation (with SPO & Cardano Foundation collaboration) quickly revealed the cause of the issue – a complex bug in data structure handling code related to the precise order of insertion/deletion of multi-asset tokens into the internal ledger record. Input Output Global (IOG) engineers, along with SPOs and DApp developers, collectively identified how to reproduce the issue as a unit test that could be included in the standard Cardano node test suite. Following successful testing, this led to a bug fix being implemented, tested, benchmarked, and deployed as a hotfix in the node v.1.35.5 release on Friday, January 27, 2023. Care was taken not to highlight the exact cause of the bug during this process so that it could not be exploited prior to SPOs deploying this new node version. With the fix deployed, the Cardano SPO and developer community have not seen any further instances of this issue.

Further Details​

You can read more details on the incident and how it was managed from SundaeSwap’s Pi Lanningham here. Thanks again to all the community for its support in identifying and fixing this bug.