Incident reporting: Cardano block production temporary outage
On Sunday, January 22, 2023, an incident occurred resulting in block production pausing for a brief period of time (approximately two minutes, similar to the usual pause at an epoch boundary). Around 50% of block-producing nodes and relays restarted during this period. Having restarted, nodes continued to produce blocks without failure. While the network continued to operate, the issue did have the potential to affect network integrity, so was flagged as a ‘critical’ incident, thus warranting immediate response and investigation by IOG engineers. The investigation (with SPO & Cardano Foundation collaboration) quickly revealed the cause of the issue – a complex bug in data structure handling code related to the precise order of insertion/deletion of multi-asset tokens into the internal ledger record. Input Output Global (IOG) engineers, along with SPOs and DApp developers, collectively identified how to reproduce the issue as a unit test that could be included in the standard Cardano node test suite. Following successful testing, this led to a bug fix being implemented, tested, benchmarked, and deployed as a hotfix in the node v.1.35.5 release on Friday, January 27, 2023. Care was taken not to highlight the exact cause of the bug during this process so that it could not be exploited prior to SPOs deploying this new node version. With the fix deployed, the Cardano SPO and developer community have not seen any further instances of this issue.
Further Details
You can read more details on the incident and how it was managed from SundaeSwap’s Pi Lanningham here. Thanks again to all the community for its support in identifying and fixing this bug.