Incident reporting: Cardano block production temporary outage​

On Sunday, January 22, 2023, an incident occurred resulting in block production pausing for a brief period of time (approximately two minutes, similar to the usual pause at an epoch boundary). Around 50% of block-producing nodes and relays restarted during this period. Having restarted, nodes continued to produce blocks without failure. While the network continued to operate, the issue did have the potential to affect network integrity, so was flagged as a ‘critical’ incident, thus warranting immediate response and investigation by IOG engineers. The investigation (with SPO & Cardano Foundation collaboration) quickly revealed the cause of the issue – a complex bug in data structure handling code related to the precise order of insertion/deletion of multi-asset tokens into the internal ledger record. Input Output Global (IOG) engineers, along with SPOs and DApp developers, collectively identified how to reproduce the issue as a unit test that could be included in the standard Cardano node test suite. Following successful testing, this led to a bug fix being implemented, tested, benchmarked, and deployed as a hotfix in the node v.1.35.5 release on Friday, January 27, 2023. Care was taken not to highlight the exact cause of the bug during this process so that it could not be exploited prior to SPOs deploying this new node version. With the fix deployed, the Cardano SPO and developer community have not seen any further instances of this issue.

Further Details​

You can read more details on the incident and how it was managed from SundaeSwap’s Pi Lanningham here. Thanks again to all the community for its support in identifying and fixing this bug.

The team works on applied research and consulting in formal methods that is directly applicable to evidence based engineering in Core Tech and beyond.

High level summary​

This sprint the team has submitted two papers for publication, carried out consultancy with other teams and has an opening for an intern.

Details​

• Finalised and submitted two papers to ICE 2023

• Studying Praos paper.

• The team is hiring a performance engineering intern - https://apply.workable.com/io-global/j/BFEC5B0AFF/

• Design exploration for congestion management.

High-level summary​

This week, the Hydra team focused on improving the smoke test, fixing developer tooling, and improving the API for voting use cases. They reviewed progress on auction, payments, and voting projects and made worked on reproducing a bug with handling rollbacks. Moving forward, the team plans to update dependencies, implement a dirt road fix for the rollbacks bug, and explore adding Hydra support to kupo.

What did the team achieve this week​

• Reviewed progress on auction, payments and voting projects
• Improved smoke tests so they can run on mainnet
• Fixed a regression in the development environment and updated cardano-node used in tests
• Improved API with more configurability to unblock voting use case
  • Exclude utxo in SnapshotConfirmed outputs #808
  • Addressed a user request by only sending Greetings once #813
• Reproduced the rollback bug by improving our model-based test suite #784

What are the goals of next week​

• Update dependencies to match cardano-node master
• Dirt road fix for rollbacks #784
• Update Hydraw to maintain state locally
• Explore adding Hydra support to kupo
• Put disclaimer texts and closing mainnet compatibility feature #713

High-level summary​

This week, the Hydra team published the monthly report for March, created separate links for different versions of documentation, and clarified potential solutions for the Rollbacks bug. The team also discussed Query API requests for the Hydra Voting project and met with community members to answer questions about Hydra. Moving forward, the team plans to integrate the specification into the repository, implement a short-term fix for the Rollbacks issue, and draft a Query API ADR.

What did the team achieve this week​

• Published monthly reports on website
• Separate last released and latest versions of docs (e.g. normal released vs. /unstable)
• Discussed Query API concerns from Hydra Voting project (link)
• Clarified rollbacks bug and defined possible solutions (short and long term) #784
• Met with community members to answer questions about Hydra

What are the goals of next week​

• Finally get the docs integrated into the repository #693
• Dirt road fix for rollbacks #784
• Disclaimer text and detail known issues about mainnet compatibility #713

High level overview​

The Mithril team released a new 2313.0 distribution that implements minor improvements and bug fixes. They continued implementing the migration of the aggregator stores to a relational design. They finished adaptating and migrating the certificate, epoch settings, signed entity type, signer registration and open message stores. Additionally, they completed the refactoring of the dependency injector of the aggregator and simplified its multi-signer in order for it to sign multiple concurrent messages.

Finally, the team has worked on designing the new architecture of the network to support signing multiple types of data and upgraded the version of the Cardano node used in the Mithril testnets and devnet to 1.35.6.

Low level overview​

• Released the new distribution 2313.0
• Worked on the epic that implements a relational store in the aggregator #779:
  • Completed the migration/adaptation of the epoch_settings table #813
  • Completed the migration/adaptation of the signed-entity-type table #815
  • Completed the migration/adaptation of the certificate table #817
  • Completed the migration/adaptation of the open_message table #827
  • Completed the migration/adaptation of the signer_registration table #828
• Completed the refactoring of the dependency injection of the aggregator #823
• Completed the cleanup of the multi-signer of the aggregator #824
• Completed the upgrade of the Cardano node to 1.35.6 #843