2025-07 - 2025-09​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, some highlights of SRE achievements for this quarter include:

• All IOE cardano-parts supported node environments, including preview, preprod, and mainnet were upgraded to cardano-node release 10.5.1 with various additional pre-release candidate versions tested.

• Changes required for the new tracing system to become the default tracing system have been largely implemented in cardano-node, including various scripts, nixos services, and OCI container updates. Similar changes supporting the new tracing system have been also implemented elsewhere, such as the iohk-nix configuration repo, blockperf monitoring tool, and nixos based cardano-parts node cluster tooling stack, allowing for the upcoming cardano-node 10.6.x release later this year to default to the new tracing system.

• With the exception of a few canary machine deployments, all IOE testnet and mainnet cardano-node infrastructure are now running using the new tracing system.

• Cardano-node release binaries have been cleaned up and OCI containers now include the snapshot-converter binary.

• Adawallet and the cardano-airgap image were feature upgraded to be compatible with operations required for the Midnight Glacier Airdrop, including cardano-signer signing and native token UTxO handling.

• The devx-ci cluster has been expanded to include Oakhost Darwin build capacity.

Details for this and many other SRE work items can be found in the biweekly SRE reports.

2025-04 - 2025-06​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, some highlights of SRE achievements for this quarter include:

• All IOE cardano-parts supported node environments, including preview, preprod, and mainnet were upgraded through cardano-node releases including 10.3.1 and 10.4.1 with various additional pre-release versions tested.

• The cardano performance cluster used for benchmarking was upgraded to ephemeral instance configuration to additionally benchmark LMDB ledger backend.

• Peer snapshot files (and checkpoint file for mainnet) are now a regular part of the release binary artifacts in preparation for genesis mode.

• Genesis mode has been rolled out as the default consensus mode for preview and preprod testnets.

• Adawallet and Cardano-airgap repos received significant updates to prepare for some complex but infrequent pool operations.

• Aarch64-linux resources were brought online in the Hydra build cluster.

Details for this and many other SRE work items can be found in the biweekly SRE reports.

2025-01 - 2025-03​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, some highlights of SRE achievements for this quarter include:

• All IOE cardano-parts supported node environments, including preview, preprod, sanchonet, mainnet were upgraded through cardano-node releases including 10.1.4 and 10.2.1 (for pre-release configured machines).

• Sanchonet was respun to test the hard fork mechanism to protocol version 11 with community participation.

• A nixos buildkite agent module was ported to cardano-playground to support custom QA testing workflows.

• A nixos AWS ephemeral block device module was added to cardano-parts to support upcoming cardano-node UTXO-HD features which leverage fast NVMe block devices.

• The sanchonet cardano network was retired and handed over to community as it had fulfilled its purpose as an early testing network for Voltaire era governance features.

• Cardano node and submit-api container improvements were made to reduce prior inconsistencies, add a new useful mode of operation, add useful configuration references and improve documentation.

• The cardano-node flake closure was optimized and reduced in size 18 percent.

• The nix hydra machine used for cardano-node builds was migrated to upgraded hardware allowing for better evaluation concurrency and therefore an enhanced developer ci experience.

Details of this and other SRE work can be found in the biweekly SRE reports.

2024-10 - 2024-12​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, some highlights of SRE achievements for this quarter include:

• Cardano-parts started offering regular releases to ease the process of upgrading rather than only PR merges with migration notes.

• All IOE cardano-parts supported node environments, including preview, preprod, sanchonet, mainnet were upgraded through various cardano-node releases including 10.0.0-pre, 10.1.0-pre, 10.1.1, 10.1.2, and finally into 10.1.3 by the end of December.

• All environments were migrated from deprecated grafana agent to grafana alloy.

• Sanchonet environment was re-spun twice; once for cardano-node 10.0.0-pre and once for a community scheduled PV11 hardfork test.

• Preview and preprod networks were successfully forked into Conway era, Plomin protocol version 10.0.

• Private and shelley-qa chains were retired after fulfilling their intended goal.

• Remaining cardano-world legacy resources were retired, replaced by cardano-playground.

Details of this and other SRE work can be found in the biweekly SRE reports.

Next steps​

• Switch to the new cardano-node tracing system as default and retire use of the legacy tracing system, including dashboards, alerts, cardano-tracer and cardano-node nixos service updates as well as container image required changes.

• Continue cardano-parts and operations improvements.

2024-07 - 2024-09​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, SRE achievements for this quarter include:

• All IOE cardano-parts supported node environments, including preview, preprod, sanchonet, mainnet and other clusters were upgraded through various cardano-node releases of 9.0.0, 9.1.0, 9.1.1, 9.2.0, and finally into 9.2.1 by the end of September.

• All IOE cardano-parts supported node environments had dual stack ipv4/ipv6 capability added and configured, including supporting scripts and recipes, module updates, terraform/openTofu resource changes and software updates to make previously ipv6 incompatible software ipv6 compatible, example: cardano-faucet. Cardano-parts clusters can now seamlessly participate in ipv6 cardano-node traffic and other ipv6 traffic.

• Preview, preprod and mainnet networks were hard forked to Conway.

• Legacy mainnet cluster shelley-era high-load relays were scaled down over the quarter and stopped now that p2p has removed the need for them.

• Legacy cardano explorer was retired and Cardano Foundation is now providing the replacement landing page which links to several community explorers.

• Cardano-smash production load was retired from equinix metal hosting from the cardano-world repo and transferred to the new cardano-mainnet cluster.

• New cardano-mainnet cluster scaling capability was added for the bootstrap machines. Block performance analysis was used to tune RTS parameters on the bootstraps and other mainnet pool machines.

• Sanchonet environment was re-spun for cardano-node 9.1.0 and greater compatibility.

• Private chain was stopped and re-spun with 2 hr epochs for testing.

• New nixosModules were added to cardano-parts and cardano-playground, including: profile-blockperf, profile-tcpdump (for saving node traffic pcaps to s3) and ogmios.

• Documentation for playground and mainnet cluster operations was improved, such as documents for: debugging of peer-to-peer connections; governance voting with the playground stakepools; faucet setup; faucet pool de-delegation and mainnet dbsync cardano-snapshot operations. See the docs/explain directory of both the cardano-playground and cardano-mainnet repos for details.

• The cardano-monitoring repository received a lot of documentation and improvements and now also serves as the home for devx-ci metrics after migration away from Grafana cloud hosting.

• An improved cardano-airgap image for secure signing operations was created and made available.

• Hydra CI performance was improved with changes to our custom Nix evaluator and optimized resource usage while waiting for IFDs.

Next steps​

• Add a production protocol-parameters cardano-api based server to facilitate community transaction creation without requiring a live node.

• Migrate from deprecated grafana agent to grafana alloy.

• Finalize support for the new cardano-node tracing system once the service is rewritten for general consumption.

• Extend govtool frontend and backend to a process-compose stack once govtool is publicly buildable again.

• Continue cardano-parts and operations improvements.

2024-04 - 2024-06​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, main SRE achievements for this quarter include:

• All cardano release environments, including preview, preprod, mainnet legacy and mainnet new clusters were upgraded through various cardano-node releases of 8.9.2, 8.9.3, 8.9.4, 8.12.1, and finally into 8.12.2 by the end of June

• Cardano pre-release environments additionally iterated through pre-release upgrades of 8.11.0-pre, 8.12.0-pre, and finally into 8.12.2 by the end of June with the exception of sanchonet which remains pinned at 8.11.0-pre until the next respin to support node version 9.0.0 or greater

• Sanchonet environment was re-spun two times for pre-release Conway testing of cardano-node versions 8.10.0-pre and 8.11.0-pre respectively

• Private chain environment was re-spun three times to support fast epoch Conway testing

• Cardano-playground and cardano-mainnet repos have added ten operations oriented documents for knowledge transfer

• Block producers which participate in mithril signing will now produce metrics and can have them scraped with the default metrics agent

• A cluster spin-up job to utilize the new cardano-cli create-testnet-data sub-command was created

• A nixosModule, dashboards and alerts were added supporting the new cardano tracing system

• Many new operations scripts and features were added, including a template diff and patch recipe to pull the latest cardano-parts improvements to consuming repositories more easily

Next steps​

• Finalize support for the new cardano-node tracing system once the service is rewritten for general consumption

• Add IPv6 cardano-parts support

• Extend govtool frontend and backend to a process-compose stack once govtool is publicly buildable again

• Continue cardano-parts and operations improvements

2024-01 - 2024-03​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, main SRE achievements for this quarter include:

• All cardano release environments, including preview, preprod, mainnet legacy and mainnet new clusters were upgraded through cardano-node releases of 8.7.3, 8.9.0 and finally into 8.9.1 by the end of March

• All cardano pre-release environments, including sanchonet, private chain, and shelley-qa clusters were upgraded through cardano-node releases of 8.7.3, 8.8.0-pre, 8.8.1-pre, 8.9.0 and finally into 8.9.1 by the end of March

• Sanchonet and private chain environments were both re-spun once each during this quarter to support new pre-release versions of cardano-node in the Conway era

• Cardano-parts added a cardano-db-sync process-compose stack for each environment

• Cardano-parts added a cardano-node process-compose stack for each environment

• Cardano-parts added enhancements for topology related nixos modules and functions to accomodate new bootstrapPeer functionality, new topology attributes and increased complexity network deployments.

• Cardano-parts added support for mithril signers integrated with block producers and a mithril-signer-verifier service for monitoring

• Sanchonet, preview, preprod and mainnet IOG block producers are now signing mithril certificates

• Cardano-parts added support for mithril clients in nixos cardano-node systemd service, process-compose job stacks and nix cardano-node entrypoint, all of which also require any mithril snapshot to be signed by a trusted IOG pool prior to use

• Cardano-parts added ip integration tooling, so that similar to other deployer tools like nixops, nixosConfigurations possess ip information which can be used in module configuration

• Cardano-db-sync snapshots server migration from legacy mainnet cluster to new mainnet cluster with a rewrite of the snapshot service was completed

• Cardano metadata server migration to Cardano Foundation was completed

• BlockPerf, a cardano-node performance monitoring tool, was integrated into the new mainnet cluster relays

• Cardano-node bootstrapPeer functionality was added with node 8.9.x, requiring effort to align nixos service module code between cardano-node nixos services, iohk-nix topology generation, cardano-ops legacy code, and cardano-parts module compatibility as well as feature test under various edge cases

• Cardano-playground added govtool backend support for private chain voltaire testing team

Next steps​

• Add support for the new cardano-node metrics system

• Add IPv6 cardano-parts support

• Extend govtool frontend and backend to a process-compose stack

• Adapt network spin-up tooling for the new create-testnet-data cardano-cli command

• Continue cardano-parts and operations improvements

2023-10 - 2024-01​

Main achievements​

• Release benchmarking, leading up to next mainnet release
• Conway benchmarking of existing Babbage workloads
• P2P benchmarks, validating viability as default topology
• Added basic PlutusV3 capability of our tooling
• Publication of benchmarking reports accompanying a mainnet release
• GHC9 performance investigation
• Finalized and validated all optimizations for the new tracing system
• New Nomad benchmarking cluster: production use
• Adjustment of our infrastructure to cover the migration to IntersectMBO
• Conensus QTAs: prototype developed into alpha-stage benchmark
• Successful on-boarding of a new team member

Release benchmarking​

We've performed and analyzed benchmarks for node versions 8.6.x to 8.7.3, which is projected to be the next mainnet release, throughout Q4. Along that way, we have identified, located and handled all performance blockers.

Additionally, we've started publishing benchmarking reports here on Cardano Updates. The format is meant to increase transparency and provide insight into those measurements that accompany mainnet releases - demonstrating the absence of performance regressions and development of specific metrics over time.

Conway benchmarks​

Furthermore, we've done first ever benchmarking of the Conway ledger. To that end, we've ported our Babbage workloads to Conway for immediate comparability. Fortunately, we've have not found any performance regression in the Conway ledger.

P2P benchmarks​

In Q4, we've validated P2P topology to be viable as default for both relay and block producer nodes. As a consequence, we've switched to P2P topology for benchmarking baselines ourselves.

GHC9 performance​

In Q4, evaluation of GHC9.2's and GHC9.6's optimizer in the context of the Cardano code base has been completed. Eventually, GHC9.6 has shown itself to be much more suitable from a performance perspective. We're convinced that with a few select annotations in the code, GHC9.6's optimizer can produce a result on par performance-wise with GHC8.10 - which just was a great release in that regard. With GHC9.2 unfortunately, the changes would have to be more invasive - and thus more time-consuming.

New nomad cluster​

We’ve moved the new Nomad cluster into production use and established new baselines for each workload on it. Additionally, we’ve shut down the legacy cardano-ops benchmarking cluster, and archived all raw data from it.

Consensus component QTAs​

We’ve developed the existing prototype into an automatable, self-contained benchmark called beacon, as well as systematized workloads and run structure for it. Moreover, we’ve demonstrated usefulness of the metrics, their reproducibility, and identified domains viable for QTAs with system-level benchmarks.

New team member​

We're happy to welcome a new joiner to our team! We've successfully onboarded him in Q4; he has taken over the cardano-tracer service - the node-external component of the new tracing system - and has already landed several valuable contributions.

Areas of focus​

Highlights​

Implement legacy mode for UTxO-HD to keep baseline performance​

• ✅ We managed to run a UTxO-HD capable node in legacy mode, maintaining the baseline memory usage while keeping all the ledger state in memory.
  • While the legacy mode is not production-ready (it requires further integration and testing), it remains as a plan B should the need arise to release UTxO-HD if our stakeholders so demand it.
• ✅ We pivoted to redesigning the Ledger DB API because:
  • This is needed for integrating the LSM-tree backend.
  • The redesign opened the possibility of implementing an in-memory backend that would keep the same performance and resource requirements as the baseline version (which needs to be confirmed by benchmarks).
• ✅ We created a more general Ledger DB API.
• 🛠️ We are integrating (into the feature branch) the existing Ledger DB implementations with the new API.
• 🛠️ We are implementing the new in-memory backend.

Assist mainnet node release with initial Conway capabilities, test, benchmark, and improvements to CIP 1694​

• ✅ We recognized that Conway introduces a new challenge in the versioning of NTC queries, and we resolved it (see 864 and 4770).

Assist with P2P IOG relay network shut down​

• ✅ We created a prototype for the pre-Genesis State Machine for bootstrap peers, which is currently under test (see this PR).

Assist with repo transfer to Intersect​

• ✅ We transferred the ouroboros-consensus repository to the Intersect GitHub organization.

Support vendors to deliver contracts​

• Genesis
  • ✅ Interacted with the Consensus team and addressed resulting feedback on past deliverables.
  • ✅ Finished implementation of the testing infrastructure of Genesis
  • ✅ Started to refine the Proof of Concept demo into an actual implementation of the core components of the Genesis design.
• 💾 LSM-tree implementation. Well Typed:
  • ✅ Finished the design of the public facing API.
  • ✅ Defined the LSM-tree database file-type formats.
  • ✅ Implemented property and model-based tests.

Operation serenity Q4 2023​

• 🎉 We welcomed our newest team member @RenateEilers and assisted with her (ongoing) onboarding.
• ✅ We implemented a simplification in the ChainSync mini-protocol that is also a step towards Ouroboros Chronos.
• ✅ We added tests to check Consensus emits valid CBOR, which prevents the generation of invalid binary encoding.
• ✅ We established and implemented an interface between Consensus tooling and P&T tooling, which constitutes a step towards incorporating component level benchmarks in our development process.

2023-10 - 2023-12​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, main SRE achievements for this quarter include:

• Cardano-parts support was added for cardano-db-sync, cardano-smash, cardano-faucet, cardano-metadata, grafana monitoring along with a number of other features

• Completed migration of testnets from cardano-world to the cardano-playground cluster

• Completed migration of the cardano book from cardano-world to the cardano-playground cluster

• Completed migration of pools from cardano-ops to the cardano-mainnet cluster

• Creation of a mainnet p2p bootstrap cluster

• Cardano sanchonet environment respins during the quarter for testing new cardano-node pre-release Conway era functionality

• All environments were upgraded to cardano-node 8.7.2 or 8.7.3 by the end of the quarter

• Completion of a govtool backend deployment for Voltaire chain testing

• Creation of a cardano-monitoring repository, cardano-monitoring:

  • A new repository enabling agile deployment of EC2 monitoring servers, compatible with OpenTofu grafana and mimir providers

Next steps​

• Scale down the mainnet non-p2p legacy cluster

• Add deployment support for new network services, such as Mithril

• Continue cardano-parts and operations improvements