Skip to main content

SRE Q3 2024 Update

· 3 min read
John Lotoski
John Lotoski
Service Reliability Engineer

2024-07 - 2024-09​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, SRE achievements for this quarter include:

  • All IOE cardano-parts supported node environments, including preview, preprod, sanchonet, mainnet and other clusters were upgraded through various cardano-node releases of 9.0.0, 9.1.0, 9.1.1, 9.2.0, and finally into 9.2.1 by the end of September.

  • All IOE cardano-parts supported node environments had dual stack ipv4/ipv6 capability added and configured, including supporting scripts and recipes, module updates, terraform/openTofu resource changes and software updates to make previously ipv6 incompatible software ipv6 compatible, example: cardano-faucet. Cardano-parts clusters can now seamlessly participate in ipv6 cardano-node traffic and other ipv6 traffic.

  • Preview, preprod and mainnet networks were hard forked to Conway.

  • Legacy mainnet cluster shelley-era high-load relays were scaled down over the quarter and stopped now that p2p has removed the need for them.

  • Legacy cardano explorer was retired and Cardano Foundation is now providing the replacement landing page which links to several community explorers.

  • Cardano-smash production load was retired from equinix metal hosting from the cardano-world repo and transferred to the new cardano-mainnet cluster.

  • New cardano-mainnet cluster scaling capability was added for the bootstrap machines. Block performance analysis was used to tune RTS parameters on the bootstraps and other mainnet pool machines.

  • Sanchonet environment was re-spun for cardano-node 9.1.0 and greater compatibility.

  • Private chain was stopped and re-spun with 2 hr epochs for testing.

  • New nixosModules were added to cardano-parts and cardano-playground, including: profile-blockperf, profile-tcpdump (for saving node traffic pcaps to s3) and ogmios.

  • Documentation for playground and mainnet cluster operations was improved, such as documents for: debugging of peer-to-peer connections; governance voting with the playground stakepools; faucet setup; faucet pool de-delegation and mainnet dbsync cardano-snapshot operations. See the docs/explain directory of both the cardano-playground and cardano-mainnet repos for details.

  • The cardano-monitoring repository received a lot of documentation and improvements and now also serves as the home for devx-ci metrics after migration away from Grafana cloud hosting.

  • An improved cardano-airgap image for secure signing operations was created and made available.

  • Hydra CI performance was improved with changes to our custom Nix evaluator and optimized resource usage while waiting for IFDs.

Next steps​

  • Add a production protocol-parameters cardano-api based server to facilitate community transaction creation without requiring a live node.

  • Migrate from deprecated grafana agent to grafana alloy.

  • Finalize support for the new cardano-node tracing system once the service is rewritten for general consumption.

  • Extend govtool frontend and backend to a process-compose stack once govtool is publicly buildable again.

  • Continue cardano-parts and operations improvements.