Skip to main content

55 posts tagged with "sre"

View All Tags

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node 10.5.1 has now been promoted to a full release for mainnet use and has been deployed to all of IOE's node SRE testnet and mainnet clusters.

  • A new docker OCI tag of 10.5.1-docker is additionally available for any OCI image users who want to use the snapshot-converter binary from within the image. The snapshot converter can be found at path /usr/local/bin/snapshot-converter. New cardano-node OCI release images going forward will contain the snapshot converter to facilitate ledger backend state changes without having to rely on host level tooling or a fully ledger replay.

  • Cardano-parameters is a new repo which maintains the mainnet, preprod and preview protocol parameters with a daily update, as reported by the BlockFrost Cardano API service.

Repository Work -- Merged​

Cardano-mainnet​

  • Cardano-node release has been updated to 10.5.1. Rotates KES, cleans up 10.5.0 module code no longer needed and deploys select bootstraps with an EgressPollInterval modifier. cardano-mainnet-pr-37

Cardano-monitoring​

  • A modernization and security update PR which updates most flake pins, patches grafana for CVE vulnerabilities, updates the opentofu-registry and adds filtering for alpha/beta versions, updates tofu resource declarations and recipes, migrates to SSH over SSM, fixes some race conditions on reboot and more. See the PR header for details. cardano-monitoring-pr-2

Cardano-node​

  • For the purposes of a 10.5.1-docker OCI image tag, adds snapshot-converter to the nix overlay and bundles it into the cardano-node OCI image. Adds documentation with example command usage. cardano-node-compare

Cardano-parts​

  • Cardano-node pre-release has been updated to 10.5.1. A nix packaged version of cardano-signer has been added, blockperf and credential-manager tools updated. Nix jobs for facilitating governance activities have been improved, easing operations. cardano-parts-release-v2025-07-23

  • Cardano-node release has been updated to 10.5.1. cardano-parts-release-v2025-07-25

Cardano-playground​

  • Cardano-node pre-release has been updated to 10.5.1. Adds playground gov action support scripts, preview/preprod committee state changes and relevant gov action artifacts. Includes various improvements with cardano-parts release v2025-07-23. See the PR header for details. cardano-playground-pr-49

  • Cardano-node release has been updated to 10.5.1. Fixes a playground voting script to work with sops decrypt using bash array args; drops some deprecated code and updates cardano-book. cardano-playground-pr-50

Devx-ci​

  • Nix-darwin was updated to 25.05 requiring updating of the Darwin guest bootstrap scripts as well as the Buildkite modules. Deprecated Hetzner Darwin materialization for a legacy ssh problem was removed and Hydra impermanence config was updated. devx-ci-compare

Repository Work In Progress -- PRs and Branches​

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Blockperf new tracing system compatibility PRs have been merged to master.

  • Cardano-node binary release artifacts have been cleaned up, removing some irrelevant binaries and adding tx-generator for the upcoming node 10.5.1 release.

  • SRE has undertaken governance activities on preview and preprod networks to refresh the constitutional committee expiry and reduce committee minimum size to 3. This process has completed on preview and will complete soon on preprod.

Repository Work -- Merged​

Blockperf​

  • Adds a publishing control bool via env var BLOCKPERF_PUBLISH which defaults to True if unset. This supports the use case where published metrics to CF don't make sense, such as temp testnet machines, but blockperf metrics would still be valuable to scrape for internal review. If publishing is disabled, the startup requirements are loosened; for example, cert key declarations are not required. This PR also adds the preview network. blockperf-pr-31

  • Supports new node tracing in a backward compatible manner: adds support for new node tracing system; adds checks for new node tracing system config; adds a boolean env var tracing system selector, BLOCKPERF_LEGACY_TRACING, defaulting to True; updates the readme to reflect the new options and operating modes. blockperf-pr-32

Cardano-airgap​

  • Makes boot relevant files available at predictable paths. An external grub can use these to load the iso image from block device or local boot storage accessible file path cardano-airgap-pr-5

Cardano-node​

  • This PR adds tx-generator and removes several release binaries. To do so, musl builds needed to be added and git hash stamping was applied to set commit version in the binary. Workbench use of tx-generator was changed to a passthru noGitRev package version to avoid redundant building of the same source because of differing commit hash. Release bin filtering was changed from a denyList to an acceptList approach to avoid inclusion creep if the denyList isn't maintained. cardano-node-pr-6271

  • Adds cardano tracer basic startup and connect tests to hydra job checks/nixosTests/cardanoNodeArtifact and also adds cardano tracer basic startup and connect tests to hydra job checks/nixosTests/cardanoNodeEdge. cardano-node-pr-6272

  • Adds cardano-tracer, tx-generator to the release bins, and cleans up the release bins with an allowList approach. Hotfix backport to 10.5.1 cardano-node-pr-6280

Cardano-signer (nix packaged)​

Repository Work In Progress -- PRs and Branches​

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • New tracing system supporting PRs in iohk-nix and cardano-node have merged, making the new tracing system the default starting with upcoming node 10.6.0 release.

  • A dedicated aarch64-linux machine has been added to the devx-ci build cluster to improve CI performance for that arch.

  • Additional darwin resources will also be joining the devx-ci build cluster in the near future.

Repository Work -- Merged​

Cardano-mainnet​

  • Dbsync snapshots machine was switched from ramdisk based tmpfs to blockdevice tmpfs. Cardano-node pre-release has been updated to 10.5.0, cardano-cli pre-release to 10.11.0.0 and mithril to v2524.0. Opentofu resources, just recipes and other code has been updated to transition to use of ssh over AWS SSM with the closure of port 22 ingress. Includes various improvements with cardano-parts release v2025-06-24. Additional details are included in the PR header. cardano-mainnet-pr-36

Cardano-node​

  • This PR changes the default tracing system to the new tracing system and implements support for this where needed including: nixos cardano-node service, nixos cardano-tracer pre-existing service, nixos cardano-tracer new service, nixos scripts, nixos tests, docker, workbench service tracer changes, configuration reference files, bash scripts, CI. Additional details are included in the PR header. cardano-node-pr-6133

Cardano-parts​

  • Cardano-node pre-release has been updated to 10.5.0, cardano-cli pre-release to 10.11.0.0 and mithril to v2524.0. Opentofu resources, just recipes and other code has been updated to transition to use of ssh over AWS SSM with the closure of port 22 ingress. Additional details are included in the release notes. cardano-parts-release-v2025-06-24

Cardano-perf​

Cardano-playground​

  • Cardano-node pre-release has been updated to 10.5.0, cardano-cli pre-release to 10.11.0.0 and mithril to v2524.0. Opentofu resources, just recipes and other code has been updated to transition to use of ssh over AWS SSM with the closure of port 22 ingress. Includes various improvements with cardano-parts release v2025-06-24. Additional details are included in the PR header. cardano-playground-pr-48

Devx-ci​

  • Adds a new dedicated aarch64-linux hetzner rx170 machine to the CI cluster, including supporting nixos hardware and config module, re-keyed secrets, new wireguard ip range assignments based on machine namespace, bootstrap script updates and removal of emulated linux builds. Additional details are included in the PR header. devx-ci-pr-132

Iohk-nix​

  • This PR changes the default tracing system to the new tracing system and implements support for this where needed including: cardano-lib, generic log config, generic tracing config, testnet template config, legacy generic log config, legacy testnet template config. Additional details are included in the PR header. iohk-nix-pr-594

Repository Work In Progress -- PRs and Branches​

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Ouroboros-network-ops cluster was upgraded to the latest cardano-parts release with ssh over ssm usage.

  • The performance and tracing benchmarking cluster was upgraded to utilize ephemeral NVMe block devices for use in benchmarking ledger LMDB backend.

  • Blockperf, a software for measuring block propagation times in cardano networks, was updated to support the new tracing system.

  • Work to utilize the new tracing system as default for shipped configs such as in release artifacts, docker oci and cardano book as well as nixos module services continues and is near ready for merge. The cardano-node new tracing system related PRs currently listed in the "Repository Work In Progress" section are planned for merge prior to node 10.6.0 tagging.

Repository Work -- Merged​

Cardano-node​

  • Bumps iohkNix flake pin and updates cardano configurations to match for ci. cardano-node-pr-commit

  • Updates the nixos cardano-node service so that targetNumber* module option parameters inherit from ouroboros-network by default. cardano-node-pr-commit

Cardano-perf​

  • Upgrades the performances and tracing team's infra cluster to nixpkgs 25.05, latest colmena, fixes a wireguard startup race condition, and converts the cluster to usage of ephemeral NVMe volumes for LMDB capable benchmark testing on fast block devices. Details in the PR header. cardano-perf-pr-4

Iohk-nix​

  • Use ouroboros-network defaults for deadline and sync target related config. Update peer snapshots and useLedgerAfterSlot per env. iohk-nix-pr-601

Ouroboros-network-ops​

Repository Work In Progress -- PRs and Branches​

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano parts, cardano-playground and cardano-mainnet clusters were updated to nixpkgs 25.05 with nix 2.29.1 and all machines deployed.

  • Cardano-airgap repo now has the ISO available for download via hydra CI server.

  • The cardano-node new tracing system related PRs currently listed in the "Repository Work In Progress" section are planned for merge prior to 10.6.0 tagging. This will make the binary artifacts, docker images and book config default to use of the new tracing system.

Repository Work -- Merged​

Adawallet​

  • An adawallet maintenance PR, bumping flake inputs, repackaging cardano-hw-cli from source, fixing cardano-cli and cardano-hw-cli breaking changes in the new versions, and adding various features and improvements. Details in the PR header. adawallet-pr-21

Cardano-airgap​

  • An airgap maintenance PR, updating nixpkgs, integrating adawallet, fixing misc version breaking change issues, adding the ISO image to hydra and other various features and improvements. Details in the PR header. cardano-airgap-pr-4

Cardano-parts​

  • Sets nixpkgs to 25.05, and nix to 2.29.1. NixosModules and just recipes with breaking changes from those updates were fixed along with other miscellaneous improvements detailed below in the PR/release header. cardano-parts-release-v2025-06-05

Cardano-mainnet​

  • Nixpkgs has been updated to 25.05 and nix to 2.29.1. NixosModules and just recipes with breaking changes from those updates were fixed. Includes various improvements with cardano-parts release v2025-06-05. cardano-mainnet-pr-35

Cardano-node​

  • Bumps iohkNix flake pin to adjust QueryBatchSize declared default from 100 to 100000 to match the Consensus UTXO HD Migration Guide. Adjusts corresponding config files to match iohkNix update and for related CI jobs to pass. cardano-node-pr-6235

Cardano-playground​

  • Nixpkgs has been updated to 25.05 and nix to 2.29.1. NixosModules and just recipes with breaking changes from those updates were fixed. Includes various improvements with cardano-parts release v2025-06-05. cardano-playground-pr-47

Iohk-nix​

  • Adjust the QueryBatchSize declared default from 100 to 100000 to match the Consensus UTXO HD Migration Guide. iohk-nix-pr-600

Repository Work In Progress -- PRs and Branches​