Skip to main content

72 posts tagged with "sre"

View All Tags

SRE Team Update

· 3 min read
John Lotoski
John Lotoski
Service Reliability Engineer

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Ouroboros-network-ops cluster was upgraded to the latest cardano-parts release with ssh over ssm usage.

  • The performance and tracing benchmarking cluster was upgraded to utilize ephemeral NVMe block devices for use in benchmarking ledger LMDB backend.

  • Blockperf, a software for measuring block propagation times in cardano networks, was updated to support the new tracing system.

  • Work to utilize the new tracing system as default for shipped configs such as in release artifacts, docker oci and cardano book as well as nixos module services continues and is near ready for merge. The cardano-node new tracing system related PRs currently listed in the "Repository Work In Progress" section are planned for merge prior to node 10.6.0 tagging.

Repository Work -- Merged​

Cardano-node​

  • Bumps iohkNix flake pin and updates cardano configurations to match for ci. cardano-node-pr-commit

  • Updates the nixos cardano-node service so that targetNumber* module option parameters inherit from ouroboros-network by default. cardano-node-pr-commit

Cardano-perf​

  • Upgrades the performances and tracing team's infra cluster to nixpkgs 25.05, latest colmena, fixes a wireguard startup race condition, and converts the cluster to usage of ephemeral NVMe volumes for LMDB capable benchmark testing on fast block devices. Details in the PR header. cardano-perf-pr-4

Iohk-nix​

  • Use ouroboros-network defaults for deadline and sync target related config. Update peer snapshots and useLedgerAfterSlot per env. iohk-nix-pr-601

Ouroboros-network-ops​

Repository Work In Progress -- PRs and Branches​

SRE Team Update

· 3 min read
John Lotoski
John Lotoski
Service Reliability Engineer

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano parts, cardano-playground and cardano-mainnet clusters were updated to nixpkgs 25.05 with nix 2.29.1 and all machines deployed.

  • Cardano-airgap repo now has the ISO available for download via hydra CI server.

  • The cardano-node new tracing system related PRs currently listed in the "Repository Work In Progress" section are planned for merge prior to 10.6.0 tagging. This will make the binary artifacts, docker images and book config default to use of the new tracing system.

Repository Work -- Merged​

Adawallet​

  • An adawallet maintenance PR, bumping flake inputs, repackaging cardano-hw-cli from source, fixing cardano-cli and cardano-hw-cli breaking changes in the new versions, and adding various features and improvements. Details in the PR header. adawallet-pr-21

Cardano-airgap​

  • An airgap maintenance PR, updating nixpkgs, integrating adawallet, fixing misc version breaking change issues, adding the ISO image to hydra and other various features and improvements. Details in the PR header. cardano-airgap-pr-4

Cardano-parts​

  • Sets nixpkgs to 25.05, and nix to 2.29.1. NixosModules and just recipes with breaking changes from those updates were fixed along with other miscellaneous improvements detailed below in the PR/release header. cardano-parts-release-v2025-06-05

Cardano-mainnet​

  • Nixpkgs has been updated to 25.05 and nix to 2.29.1. NixosModules and just recipes with breaking changes from those updates were fixed. Includes various improvements with cardano-parts release v2025-06-05. cardano-mainnet-pr-35

Cardano-node​

  • Bumps iohkNix flake pin to adjust QueryBatchSize declared default from 100 to 100000 to match the Consensus UTXO HD Migration Guide. Adjusts corresponding config files to match iohkNix update and for related CI jobs to pass. cardano-node-pr-6235

Cardano-playground​

  • Nixpkgs has been updated to 25.05 and nix to 2.29.1. NixosModules and just recipes with breaking changes from those updates were fixed. Includes various improvements with cardano-parts release v2025-06-05. cardano-playground-pr-47

Iohk-nix​

  • Adjust the QueryBatchSize declared default from 100 to 100000 to match the Consensus UTXO HD Migration Guide. iohk-nix-pr-600

Repository Work In Progress -- PRs and Branches​

SRE Team Update

· 3 min read
John Lotoski
John Lotoski
Service Reliability Engineer

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node and cardano-cli release versions are now set to 10.4.1 and 10.8.0.0, respectively, in cardano-parts release v2025-05-22.

  • The upcoming node 10.5 [pre-]release is being configured to use genesis mode by default on preview and preprod environments.

  • SRE is collaborating with IT Sec team to further harden our infra and clusters.

Repository Work -- Merged​

Cardano-parts​

  • Sets node release to 10.4.1, cardano-cli release to 10.8.0.0 and mithril to v2517.1. Adds mithril ancillary key support for signed volatile and ledger state. Switches to a blockPerf fork which can be used without publishing metrics to CF, ex: for testnet usage. Includes additional misc fixes and improvements described in the PR/release header. cardano-parts-release-v2025-05-22

Cardano-mainnet​

  • Sets node release to 10.4.1, cardano-cli release to 10.8.0.0 and mithril release v2517.1. Includes various improvements with cardano-parts release v2025-05-22. Deploys all machines to 10.4.1. cardano-mainnet-pr-34

Cardano-node​

  • Bumps iohkNix for 10.5 network param changes, preview/preprod default GenesisMode consensus config, per env useLedgerAfter and peer-snapshot updates. Updates CI for the iohkNix config changes. In the cardano-node nixos service, sets preview and preprod environments to use peer-snapshots. Enables genesis mode to work OOTB for preview/preprod on: docker images, binary artifacts, nixos built machines. cardano-node-pr-6225

Cardano-playground​

  • Sets node release to 10.4.1, cardano-cli release to 10.8.0.0 and mithril release v2517.1. Deploys blockPerf to all testnet machines without publishing metrics to CF. Includes various improvements with cardano-parts release v2025-05-22. Deploys all machines to 10.4.1. cardano-playground-pr-46

Iohk-nix​

  • Adds mithril network ancillary verification keys: the ancillary verification keys are a breaking change required to pull ancillary mithril state (ie: ledger and volatile state) in addition to immutable state. In earlier mithril releases, the ancillary state was bundled together with the immutable state. iohk-nix-pr-598

  • Node 10.5 updates: set GenesisMode for ConsensusMode as default on preview and preprod; tunes TargetNumberOfEstablishedPeers, and SyncTargetNumberOfEstablishedBigLedgerPeers per network team request; updates peer-snapshots per environment; updates useLedgerAfterSlot per environment. iohk-nix-pr-599

Repository Work In Progress -- PRs and Branches​

SRE Team Update

· 3 min read
John Lotoski
John Lotoski
Service Reliability Engineer

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node and cardano-cli pre-release versions are now set to 10.4.1 and 10.8.0.0, respectively, in cardano-parts release v2025-05-01.

  • In cardano-node, support for GHC 8.10 has been removed in master reducing CI eval times to < ~3 min. The next tagged pre-release or release after node 10.4.1 will support only GHC 9.6.

Repository Work -- Merged​

Cardano-parts​

Cardano-node​

  • Auto ghcr latest tagging currently occurs on full releases. Promotion of pre-release to release, however, doesn't trigger a published event, and instead requires a manual workflow dispatch off the newly promoted release tag. This PR adds ghcr latest tagging to the manual workflow dispatch pre-release to release use case. cardano-node-pr-6200

  • Drops support for haskellNix ghc 8.10.x which also reduces eval time to ~3 min with the removal of 249 variant packages. cardano-node-pr-6206

  • Removes the eventlogged package as ghc >= 9.4 includes unconditional -eventlog support in RTS. cardano-node-pr-6208

Cardano-playground​

  • Sets node pre-release to 10.4.1 and cardano-cli pre-release to 10.8.0.0 with cardano-parts release v2025-05-01. Updates and deploys cardano-book configuration for node pre-release 10.4.1. Adds preview1-test-a-1 machine, sops updates and secrets for test forging across legacy and new tracing. cardano-playground-pr-44

Repository Work In Progress -- PRs and Branches​

SRE Team Update

· 3 min read
John Lotoski
John Lotoski
Service Reliability Engineer

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node and cardano-cli versions are now set to 10.3.1 and 10.7.0.0, respectively, in cardano-parts release v2025-04-28.

Repository Work -- Merged​

Cardano-parts​

  • Sets node pre-release (ng) to 10.3.1 and cardano-cli pre-release (ng) to 10.7.0.0 and updates nix jobs and ci to accommodate cardano-cli pre-release breaking changes. Improves a show_current_forging prepared psql query and makes other miscellaneous improvements. cardano-parts-release-v2025-04-17

  • Sets node release to 10.3.1, cardano-cli release to 10.7.0.0. cardano-parts-release-v2025-04-28

Cardano-mainnet​

  • Sets node release to 10.3.1 and cardano-cli release to 10.7.0.0. Removes prometheus key uid in the dashboards, adds trace-verbose debugging for tofu cluster builds, improves license notice attribution, bumps ci nix to 2.28.1. cardano-mainnet-pr-33

Cardano-node​

  • Adds checkpoint and peer snapshot files to the binary artifact context. Bumps node version to 10.3.1. Adds a cardano-node artifact test using nixosTests. cardano-node-pr-6177

  • Adds the consensus snapshot-converter to pkgs and the release bins. Fixes the ci artifact nixosTest for the larger tarball size. Removes a local haskellNix windows override that has been upstreamed. Bumps iohkNix for LedgerDB cfg and make corresponding ci adjustments. cardano-node-pr-6195

Cardano-playground​

  • Sets node pre-release (ng) to 10.3.1 and cardano-cli pre-release (ng) to 10.7.0.0. Improves a show_current_forging prepared psql query and makes other miscellaneous improvements. Improvements made in cardano-parts release v2025-04-17 are included in this PR. cardano-playground-pr-42

  • Sets node release to 10.3.1 and cardano-cli release to 10.7.0.0. Rotates KES for all environments, addresses IPFS occasional OOM, scales preview for RAM, updates the book configuration. cardano-playground-pr-43

Iohk-nix​

  • Adds explicit per environment LedgerDB default config for node 10.4.x. iohk-nix-pr-597

Repository Work In Progress -- PRs and Branches​