Skip to main content

56 posts tagged with "sre"

View All Tags

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • With the exception of a few canary machines still running the legacy cardano-node tracing system, the majority of IOE playground and mainnet cardano-node machines are now running the new tracing system.

  • Adawallet received a feature upgrade to sign messages, enabling it to complete glacier drop claims for adawallet accounts. The cardano-airgap image was updated to include this capability.

Repository Work -- Merged​

Adawallet​

  • Bumps to node 10.5.1 for cardano-cli 10.11.0.0, adds cardano-signer v1.29.0 to the devShell, adds a yarn devShell, improves the cardano-hw-cli package by switching to default nodejs pkg and greatly simplifies the build. adawallet-pr-22

  • Add an adawallet sign-msg feature which signs with either payment or stake key, enabling glacier drop claims on adawallet accounts. Cleans up some more legacy cardano-hw-cli packaging and adds back bash auto-completion. adawallet-pr-23

Cardano-airgap​

  • Updates adawallet for sign-msg support for glacier drop and adds cardano-signer and misc support packages and services to the devShell and iso. cardano-airgap-pr-6

Cardano-mainnet​

  • This PR primarily upgrades all machines to the cardano-node new tracing system. It provides alert, dashboard and nix module upgrades for compatibility with the new tracing system. This PR includes improvements from cardano-parts release v2025-08-05. Additional details can be found in the PR description. cardano-mainnet-pr-38

Cardano-node​

  • Adds the snapshot-converter binary to the nix overlay and the node OCI container. Adds documentation on how to use the snapshot-converter within the image for changing ledger state type. cardano-node-pr-6299

Cardano-parts​

  • This cardano-parts release changes the default tracing system from legacy to the new cardano-node tracing system for deployed machines. See the release notes for details. cardano-parts-release-v2025-08-05

  • Updates cardano-signer to v1.29.0 which allows for Byron era address claims for Midnight Glacier drop. Bumps mithril unstable and adds some flakeModule cluster options for more service granularity. cardano-parts-release-v2025-08-14

Cardano-playground​

  • This PR primarily upgrades all playground testnet machines with a few canary exceptions to the cardano-node new tracing system. It provides alert, dashboard and nix module upgrades for compatibility with the new tracing system. This PR includes improvements from cardano-parts release v2025-08-05. Additional details can be found in the PR description. cardano-playground-pr-45

Cardano-signer (nix packaged)​

  • The nix packaging for upstream cardano-signer was updated for release 1.29.0 for byron address glacier drop compatibility and a GHA for ci build testing was added. cardano-signer-pr-2

Repository Work In Progress -- PRs and Branches​

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node 10.5.1 has now been promoted to a full release for mainnet use and has been deployed to all of IOE's node SRE testnet and mainnet clusters.

  • A new docker OCI tag of 10.5.1-docker is additionally available for any OCI image users who want to use the snapshot-converter binary from within the image. The snapshot converter can be found at path /usr/local/bin/snapshot-converter. New cardano-node OCI release images going forward will contain the snapshot converter to facilitate ledger backend state changes without having to rely on host level tooling or a fully ledger replay.

  • Cardano-parameters is a new repo which maintains the mainnet, preprod and preview protocol parameters with a daily update, as reported by the BlockFrost Cardano API service.

Repository Work -- Merged​

Cardano-mainnet​

  • Cardano-node release has been updated to 10.5.1. Rotates KES, cleans up 10.5.0 module code no longer needed and deploys select bootstraps with an EgressPollInterval modifier. cardano-mainnet-pr-37

Cardano-monitoring​

  • A modernization and security update PR which updates most flake pins, patches grafana for CVE vulnerabilities, updates the opentofu-registry and adds filtering for alpha/beta versions, updates tofu resource declarations and recipes, migrates to SSH over SSM, fixes some race conditions on reboot and more. See the PR header for details. cardano-monitoring-pr-2

Cardano-node​

  • For the purposes of a 10.5.1-docker OCI image tag, adds snapshot-converter to the nix overlay and bundles it into the cardano-node OCI image. Adds documentation with example command usage. cardano-node-compare

Cardano-parts​

  • Cardano-node pre-release has been updated to 10.5.1. A nix packaged version of cardano-signer has been added, blockperf and credential-manager tools updated. Nix jobs for facilitating governance activities have been improved, easing operations. cardano-parts-release-v2025-07-23

  • Cardano-node release has been updated to 10.5.1. cardano-parts-release-v2025-07-25

Cardano-playground​

  • Cardano-node pre-release has been updated to 10.5.1. Adds playground gov action support scripts, preview/preprod committee state changes and relevant gov action artifacts. Includes various improvements with cardano-parts release v2025-07-23. See the PR header for details. cardano-playground-pr-49

  • Cardano-node release has been updated to 10.5.1. Fixes a playground voting script to work with sops decrypt using bash array args; drops some deprecated code and updates cardano-book. cardano-playground-pr-50

Devx-ci​

  • Nix-darwin was updated to 25.05 requiring updating of the Darwin guest bootstrap scripts as well as the Buildkite modules. Deprecated Hetzner Darwin materialization for a legacy ssh problem was removed and Hydra impermanence config was updated. devx-ci-compare

Repository Work In Progress -- PRs and Branches​

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Blockperf new tracing system compatibility PRs have been merged to master.

  • Cardano-node binary release artifacts have been cleaned up, removing some irrelevant binaries and adding tx-generator for the upcoming node 10.5.1 release.

  • SRE has undertaken governance activities on preview and preprod networks to refresh the constitutional committee expiry and reduce committee minimum size to 3. This process has completed on preview and will complete soon on preprod.

Repository Work -- Merged​

Blockperf​

  • Adds a publishing control bool via env var BLOCKPERF_PUBLISH which defaults to True if unset. This supports the use case where published metrics to CF don't make sense, such as temp testnet machines, but blockperf metrics would still be valuable to scrape for internal review. If publishing is disabled, the startup requirements are loosened; for example, cert key declarations are not required. This PR also adds the preview network. blockperf-pr-31

  • Supports new node tracing in a backward compatible manner: adds support for new node tracing system; adds checks for new node tracing system config; adds a boolean env var tracing system selector, BLOCKPERF_LEGACY_TRACING, defaulting to True; updates the readme to reflect the new options and operating modes. blockperf-pr-32

Cardano-airgap​

  • Makes boot relevant files available at predictable paths. An external grub can use these to load the iso image from block device or local boot storage accessible file path cardano-airgap-pr-5

Cardano-node​

  • This PR adds tx-generator and removes several release binaries. To do so, musl builds needed to be added and git hash stamping was applied to set commit version in the binary. Workbench use of tx-generator was changed to a passthru noGitRev package version to avoid redundant building of the same source because of differing commit hash. Release bin filtering was changed from a denyList to an acceptList approach to avoid inclusion creep if the denyList isn't maintained. cardano-node-pr-6271

  • Adds cardano tracer basic startup and connect tests to hydra job checks/nixosTests/cardanoNodeArtifact and also adds cardano tracer basic startup and connect tests to hydra job checks/nixosTests/cardanoNodeEdge. cardano-node-pr-6272

  • Adds cardano-tracer, tx-generator to the release bins, and cleans up the release bins with an allowList approach. Hotfix backport to 10.5.1 cardano-node-pr-6280

Cardano-signer (nix packaged)​

Repository Work In Progress -- PRs and Branches​

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • New tracing system supporting PRs in iohk-nix and cardano-node have merged, making the new tracing system the default starting with upcoming node 10.6.0 release.

  • A dedicated aarch64-linux machine has been added to the devx-ci build cluster to improve CI performance for that arch.

  • Additional darwin resources will also be joining the devx-ci build cluster in the near future.

Repository Work -- Merged​

Cardano-mainnet​

  • Dbsync snapshots machine was switched from ramdisk based tmpfs to blockdevice tmpfs. Cardano-node pre-release has been updated to 10.5.0, cardano-cli pre-release to 10.11.0.0 and mithril to v2524.0. Opentofu resources, just recipes and other code has been updated to transition to use of ssh over AWS SSM with the closure of port 22 ingress. Includes various improvements with cardano-parts release v2025-06-24. Additional details are included in the PR header. cardano-mainnet-pr-36

Cardano-node​

  • This PR changes the default tracing system to the new tracing system and implements support for this where needed including: nixos cardano-node service, nixos cardano-tracer pre-existing service, nixos cardano-tracer new service, nixos scripts, nixos tests, docker, workbench service tracer changes, configuration reference files, bash scripts, CI. Additional details are included in the PR header. cardano-node-pr-6133

Cardano-parts​

  • Cardano-node pre-release has been updated to 10.5.0, cardano-cli pre-release to 10.11.0.0 and mithril to v2524.0. Opentofu resources, just recipes and other code has been updated to transition to use of ssh over AWS SSM with the closure of port 22 ingress. Additional details are included in the release notes. cardano-parts-release-v2025-06-24

Cardano-perf​

Cardano-playground​

  • Cardano-node pre-release has been updated to 10.5.0, cardano-cli pre-release to 10.11.0.0 and mithril to v2524.0. Opentofu resources, just recipes and other code has been updated to transition to use of ssh over AWS SSM with the closure of port 22 ingress. Includes various improvements with cardano-parts release v2025-06-24. Additional details are included in the PR header. cardano-playground-pr-48

Devx-ci​

  • Adds a new dedicated aarch64-linux hetzner rx170 machine to the CI cluster, including supporting nixos hardware and config module, re-keyed secrets, new wireguard ip range assignments based on machine namespace, bootstrap script updates and removal of emulated linux builds. Additional details are included in the PR header. devx-ci-pr-132

Iohk-nix​

  • This PR changes the default tracing system to the new tracing system and implements support for this where needed including: cardano-lib, generic log config, generic tracing config, testnet template config, legacy generic log config, legacy testnet template config. Additional details are included in the PR header. iohk-nix-pr-594

Repository Work In Progress -- PRs and Branches​

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Ouroboros-network-ops cluster was upgraded to the latest cardano-parts release with ssh over ssm usage.

  • The performance and tracing benchmarking cluster was upgraded to utilize ephemeral NVMe block devices for use in benchmarking ledger LMDB backend.

  • Blockperf, a software for measuring block propagation times in cardano networks, was updated to support the new tracing system.

  • Work to utilize the new tracing system as default for shipped configs such as in release artifacts, docker oci and cardano book as well as nixos module services continues and is near ready for merge. The cardano-node new tracing system related PRs currently listed in the "Repository Work In Progress" section are planned for merge prior to node 10.6.0 tagging.

Repository Work -- Merged​

Cardano-node​

  • Bumps iohkNix flake pin and updates cardano configurations to match for ci. cardano-node-pr-commit

  • Updates the nixos cardano-node service so that targetNumber* module option parameters inherit from ouroboros-network by default. cardano-node-pr-commit

Cardano-perf​

  • Upgrades the performances and tracing team's infra cluster to nixpkgs 25.05, latest colmena, fixes a wireguard startup race condition, and converts the cluster to usage of ephemeral NVMe volumes for LMDB capable benchmark testing on fast block devices. Details in the PR header. cardano-perf-pr-4

Iohk-nix​

  • Use ouroboros-network defaults for deadline and sync target related config. Update peer snapshots and useLedgerAfterSlot per env. iohk-nix-pr-601

Ouroboros-network-ops​

Repository Work In Progress -- PRs and Branches​