Skip to main content

SRE Team Update

· 5 min read
Michael Fellinger

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Preview network was hard forked to Conway era.
  • Cardano-db-sync was updated to 13.4.0.0 across all environments.

Repository Work​

Cardano Airgap​

Commit-compare

  • Update the image to cardano-cli to 9.2.1.0 and credential-manager to HEAD.
  • Finish testing the airgap image using ext4 partitions and add ventoy to the devShell.

Cardano Parts​

Node 9.1.0, Mithril 2430.0, Chang readiness

Overview​

Sets cardano-node (release) and cardano-node-ng (pre-release) versions to 9.1.0 and mithril to 2430.0. Includes nixosModule improvements for the new tracing system, a new template-clone recipe, various recipe improvements and fixes. A Chang readiness query has been added.

Details​

  • Important versioning updates:
    • cardano-node and cardano-node-ng are now 9.1.0
  • Bumps capkgs:
    • For node releases of 9.1.0
    • For mithril 2430.0 and mithril-unstable
  • Improves the profile-cardano-node-new-tracing nixosModule and new tracing system in general by better cleaning residual legacy config items, restructuring the options for more flexibility in config composition, and configuring the new tracing system to log close to parity volume with the legacy tracing system when using UTXO-HD in memory mode.
  • Adds additional default Tcp and TcpExt metrics to the profile-grafana-agent nixosModules metrics scrape list
  • Adds curl and the pre-push script to the default cardano-parts devShell
  • Adds template alert cardano_node_elevated_restarts
  • Adds a new template recipe of template-clone for when downstream users know they simply want to mirror upstream templates rather than diff or patch them
  • Adds a new template sql script with a Chang era readiness sql query
  • Improves template recipe dbsync-pool-analyze and sql query with parameters to query any CTE in the large dbsync-pool-perf sql from cli
  • Improves template recipe dbsync-pool-analyze to handle queries that result in no non-performing pools
  • Improves template recipe dedelegate-pools with a mempool query instead of a fixed time to handle UTxO on-chain settlement
  • Removes template recipes that are now mostly cardano-playground specific
  • Fixes template dashboard for cardano-node legacy and new tracing application metrics to always show the full environment KES period
  • Fixes template recipe apply-bootstrap
  • Fixes outdated service option name and db-sync snapshot schema description

Cardano-mainnet​

Node 9.1.0, Mithril 2430.0, Bp scheduled restart module Schedule restart initial prototyping

Overview​

Sets cardano-node version to 9.1.0 and mithril to 2430.0. Adds block producer scheduled restart capability.

Details:​

  • Bumps cardano-parts for:
    • Important versioning updates:
      • cardano-node and cardano-node-ng are now 9.1.0
    • Capkgs updates:
      • For node releases of 9.1.0
      • For mithril 2430.0 and mithril-unstable
  • KES rotates mainnet block producers
  • Optimizes bootstrap nodes for -N4 RTS usage
  • Adds cardano-node-schedule-restart nixosModule and associated perSystem packages
  • Adds new alerts for cardano_node_elevated_restarts
  • Fixes dashboard for cardano-node legacy and new tracing application metrics to always show the full environment KES period

Cardano-ops (Legacy Mainnet)​

Commit-compare

Over a two week period the legacy relay nodes were scaled down to running only one instance of cardano-node per machine and then the number of running machines was further reduced by 25%.

Cardano-playground​

Node 9.1.0, Mithril 2430.0, Preview hardfork to Conway

Overview:​

Sets cardano-node (release) and cardano-node-ng (pre-release) versions to 9.1.0 and mithril to 2430.0. Hard forks preview network to Conway. Adds recipe and other improvements, including to the pool performance query recipe interface and a Chang readiness query.

Details:​

  • Bumps cardano-parts for:
    • Important versioning updates:
      • cardano-node and cardano-node-ng are now 9.1.0
    • Capkgs updates:
      • For node releases of 9.1.0
      • For mithril 2430.0 and mithril-unstable
  • Adds a new template sql script with a Chang era readiness sql query
  • Adds a babbage-to-conway cost model to the Cardano book
  • Adds a new recipe kes-rotate for easy kes rotation
  • Adds new alerts for cardano_node_elevated_restarts
  • Adds a commit stamp marker for Cardano book updates
  • Adds a new template-clone recipe for mirroring upstream template files when diffing or patching isn't needed
  • Updates the Cardano book environment for cardano-node 9.1.0
  • Updates the explainer docs for kes-rotation, chain-manipulation, new-network
  • Updates the preview faucet for new govtool operations
  • Rotates sanchonet KES, resizes the metadata server
  • Investigates mempool rejections with new tracing system and modified logging
  • Tests a comparison set of machines in mainnet environment for node 9.1.0 and utxo-hd-9.0
  • Tests a new tracing system branch for metrics renaming and KES metrics update calculations
  • Moves some cardano-playground specific recipes to the scripts/recipes-custom.just module
  • Improves template recipe dbsync-pool-analyze and sql query with parameters to query any CTE in the large dbsync-pool-perf sql from cli
  • Improves template recipe dbsync-pool-analyze to handle queries that result in no non-performing pools
  • Improves template recipe dedelegate-pools with a mempool query instead of a fixed time to handle UTxO on-chain settlement
  • Hard forks preview environment to Conway and resizes one relay member of each preview pool group

Iohk-nix​

Add conway config for mainnet/preprod/preview

Devx-ci​

Fix Hydra alerting immediately on no data

  • Migrate from Grafana Cloud to our self-hosted cardano-monitoring stack
    • Do not filter metrics to keep down number of unique series
    • This allows unlimited collection of metrics from our CI machines for better alerts and measurements for ongoing performance tuning.
  • Upgrade disko partition names manually on remaining machines ci{2,3,4,6,7,8} so boot does not break on the next deployment
  • Grafana dashboard: fix memory usage graph
  • Add alerts

Cardano-monitoring​

Commit-compare

  • Add preliminary support for Loki for log collection