Skip to main content

40 posts tagged with "sre"

View All Tags

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • All environments have been upgraded to cardano-node 9.2.1.

  • Cardano-faucet 9.2 is available and deployed which is compatible with node 9.2.x and has fixed ipv6 functionality.

  • All deployed machines now default to nix 2.24-maint after an upstream bug causing a hash miscalculation in submodules has been fixed. Nix 2.21 forward required some rework of the colmena deployment recipes as dirty git trees now force an impure colmena deployment.

  • To ease the process of upgrading cardano-parts, releases will now be made instead of only PR merges with migration notes.

Repository Work​

Cardano-faucet​

  • Binds ipv6 interface in addition to ipv4, parses and logs all ips to a unified ipv6 format and applies hlint and fmt updates. cardano-faucet-pull-14

  • Makes required changes for cardano-api 9.2.0.0 and 9.3.0.0, removes void type sig constraints, bumps haskellNix, CHaP, cardano-api -> 9.3.0.0 for node 9.2.x compatibility. Disables mingw32 builds until alex in current haskellNix pin is updated. cardano-faucet-pull-15

Cardano-parts​

  • Sets cardano-node to 9.2.1, cardano-faucet to 9.2. Bumps nix to 2.24-maint and adds ipv6 and nix versioning fixes and other improvements. Begins cardano-parts date-based releases for an improved upgrade process. Adds misc fixes and improvements. More detail is available in the release notes: cardano-parts-release-v2024-10-07

Cardano-playground​

  • Deploys cardano-node to 9.2.1, cardano-faucet to 9.2. Bumps nix to 2.24-maint and adds ipv6 and nix versioning fixes and other improvements. Adds a wip node pparams api server. More detail is available in the PR description: cardano-playground-pull-33

Cardano-mainnet​

  • Deploys cardano-node to 9.2.1, bumps nix to 2.24-maint and adds ipv6 and nix versioning fixes and other improvements. Converts bootstraps to a new cached-index-patch branch and upgrades CF canary sql queries. More detail is available in the PR description: cardano-mainnet-pull-23

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • All environments have been upgraded to cardano-node 9.2.0.

  • All IOE run cardano-parts clusters (ie: sanchonet, preview, preprod, etc testnets, mainnet and network-team clusters) have been upgraded to support ipv4/ipv6 dual stack operations. This includes each cardano network's respective public access or backbone DNS, now offering AAAA records for ipv6 connections.

Repository Work​

Cardano-parts​

  • Sets cardano-node to 9.2.0. Adds ipv6 tf, module and recipe support for ipv4/ipv6 dual stack operations. Updates alerts and dashboards for the new tracing system to reflect metrics name changes and legacy metric prefix normalization. Adds misc fixes and improvements. More detail is available in the PR description: cardano-parts-pull-48

Cardano-playground​

  • Deploys cardano-node to 9.2.0. Converts all relevant cluster resources and machines to ipv4/6 dual-stack operations. Updates alerts and dashboards for the new tracing system to reflect metrics name changes and legacy metric prefix normalization. More detail is available in the PR description: cardano-playground-pull-32

Cardano-mainnet​

  • Deploys cardano-node to 9.2.0. Converts all relevant cluster resources and machines to ipv4/6 dual-stack operations. Adds new bootstrap scaling machine startup and shutdown recipes. Updates alerts and dashboards for the new tracing system to reflect metrics name changes and legacy metric prefix normalization. More detail is available in the PR description: cardano-mainnet-pull-22

Ouroborous-network-ops​

  • Deploys cardano-node to 9.2.0. Converts all relevant cluster resources and machines to ipv4/6 dual-stack operations. Updates alerts and dashboards for the new tracing system to reflect metrics name changes and legacy metric prefix normalization. More detail is available in the PR description: ouroborous-network-ops-18

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Mainnet was hard forked to Conway era!

  • Legacy mainnet relays from cardano-ops cluster were stopped and retired.

  • Legacy cardano-explorer hosted at explorer.cardano.org was retired with landing page and beta explorer services now provided by Cardano Foundation.

  • Cardano-smash production load was cutover from the legacy cardano-world cluster to the replacement cardano-mainnet cluster. Remaining cardano-world resources will be retired in the near future.

  • Cardano-faucet was updated for cardano-node 9.1.x level compatibility.

Repository Work​

Cardano Faucet​

  • Brings faucet up to cardano-api and cardano-cli level of cardano-node 9.1: bumps relevant flake pins, updates CHaP indexes, applies fixes for upstream breaking changes, removes cardano-addresses srp, adjusts ghc options, fixes ming32 CI builds, applies most hlint and fourmolu style and config suggestions respectively: cardano-faucet-pull-12

Cardano Parts​

  • Sets cardano-node to 9.1.1, cardano-db-sync to 13.5.0.2, cardano-faucet to 9.1. Adds alerts, dashboard fixes, nixos iowait optimization, smash and blockperf nixosModule improvements. More detail is available in the PR description: cardano-parts-pull-47

Cardano-mainnet​

  • Deploys cardano node to 9.1.1, cardano-db-sync to 13.5.0.2. Improves smash deployments and backup role for production load handling. Improvements made in cardano-parts PR#47 are included in this PR. More detail is available in the PR description: cardano-mainnet-pull-21

Cardano-playground​

  • Deploys cardano node to 9.1.1, cardano-db-sync to 13.5.0.2, cardano-faucet to 9.1. Tests RTS parameter optimization and tracing system changes on preview network machines, tests utxo-hd-9.1.1 on mainnet edge nodes. Improvements made in cardano-parts PR#47 are included in this PR. More detail is available in the PR description: cardano-playground-pull-31

Cardano-world​

  • Destroy retired legacy explorer metal machines and disable alerting: commit-compare

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • The preprod network was hard forked to Conway era.

  • The nixosModule profile-blockperf in cardano-parts now includes prometheus metrics, automatically scraped with grafana-agent along with a dashboard.

  • A nixosModule profile-tcpdump in cardano-parts is now available to push on-going pcaps to s3 for historical reference.

  • Old dev environments were cleaned up and retired after the completion of the ouroboros-network-ops cluster migration to the cardano-parts stack.

  • Causes of blockperf indicated mainnet relay delayed block headers were investigated and improved with adjustments to RTS parameters and machine class.

  • Conway-era mempool log volume increase was investigated and resolved with ouroboros-network improvements.

  • Scaling capability was added to the cardano-mainnet bootstrap cluster.

Repository Work​

Cardano Parts​

  • Sets cardano-db-sync (release) to 13.4.0.0. Includes nixosModule improvements to cardano-db-sync snapshots module with a manual trigger, blockperf module new prom metrics, grafana-agent module with auto-blockperf scrape config and a new tcpdump module for persistent pcaps to s3. Recipe improvements for configuration consistency checking and openTofu improved AMI and DNS filtering have been made. The AWS machine reference spec has been updated and one alert tuned for better sensitivity. More detail is available in the PR description: cardano-parts-pull-46

Cardano-mainnet​

  • Deploys cardano-db-sync (release) to 13.4.0.0. Deploys nixosModule improvements for cardano-db-sync snapshots module with a manual trigger, blockperf module with new prom metrics, grafana-agent module with auto-blockperf scrape config and a new tcpdump module for persistent pcaps to s3. Recipes improvements for configuration consistency checking and openTofu improved AMI and DNS filtering have been made. Makes changes to pool group relays to eliminate or reduce delayed block headers. Tests additional dev patches for missingBlock errors. Adds bootstrap cluster scaling capability and a bootstrap cluster dashboard. Improvements made in cardano-parts PR#46 are included in this PR. More detail is available in the PR description: cardano-mainnet-pull-20

Cardano-ops (Legacy Mainnet)​

  • Over a two week period the legacy relay nodes were scaled down 50% further from the recent machine quantity peak. commit-compare

Cardano-playground​

  • Preprod was hard-forked to Conway. Deploys cardano-db-sync to 13.4.0.0. Recipe improvements for configuration consistency checking and openTofu improved AMI and DNS filtering have been made. Improvements made in cardano-parts PR#46 are included in this PR. More detail is available in the PR description: cardano-playground-pull-30

Cardano-world​

  • Updates openssh to 9.8p1 on remaining cardano-world (soon-to-be-retired) cluster machines commit

· 5 min read
Michael Fellinger

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Preview network was hard forked to Conway era.
  • Cardano-db-sync was updated to 13.4.0.0 across all environments.

Repository Work​

Cardano Airgap​

Commit-compare

  • Update the image to cardano-cli to 9.2.1.0 and credential-manager to HEAD.
  • Finish testing the airgap image using ext4 partitions and add ventoy to the devShell.

Cardano Parts​

Node 9.1.0, Mithril 2430.0, Chang readiness

Overview​

Sets cardano-node (release) and cardano-node-ng (pre-release) versions to 9.1.0 and mithril to 2430.0. Includes nixosModule improvements for the new tracing system, a new template-clone recipe, various recipe improvements and fixes. A Chang readiness query has been added.

Details​

  • Important versioning updates:
    • cardano-node and cardano-node-ng are now 9.1.0
  • Bumps capkgs:
    • For node releases of 9.1.0
    • For mithril 2430.0 and mithril-unstable
  • Improves the profile-cardano-node-new-tracing nixosModule and new tracing system in general by better cleaning residual legacy config items, restructuring the options for more flexibility in config composition, and configuring the new tracing system to log close to parity volume with the legacy tracing system when using UTXO-HD in memory mode.
  • Adds additional default Tcp and TcpExt metrics to the profile-grafana-agent nixosModules metrics scrape list
  • Adds curl and the pre-push script to the default cardano-parts devShell
  • Adds template alert cardano_node_elevated_restarts
  • Adds a new template recipe of template-clone for when downstream users know they simply want to mirror upstream templates rather than diff or patch them
  • Adds a new template sql script with a Chang era readiness sql query
  • Improves template recipe dbsync-pool-analyze and sql query with parameters to query any CTE in the large dbsync-pool-perf sql from cli
  • Improves template recipe dbsync-pool-analyze to handle queries that result in no non-performing pools
  • Improves template recipe dedelegate-pools with a mempool query instead of a fixed time to handle UTxO on-chain settlement
  • Removes template recipes that are now mostly cardano-playground specific
  • Fixes template dashboard for cardano-node legacy and new tracing application metrics to always show the full environment KES period
  • Fixes template recipe apply-bootstrap
  • Fixes outdated service option name and db-sync snapshot schema description

Cardano-mainnet​

Node 9.1.0, Mithril 2430.0, Bp scheduled restart module Schedule restart initial prototyping

Overview​

Sets cardano-node version to 9.1.0 and mithril to 2430.0. Adds block producer scheduled restart capability.

Details:​

  • Bumps cardano-parts for:
    • Important versioning updates:
      • cardano-node and cardano-node-ng are now 9.1.0
    • Capkgs updates:
      • For node releases of 9.1.0
      • For mithril 2430.0 and mithril-unstable
  • KES rotates mainnet block producers
  • Optimizes bootstrap nodes for -N4 RTS usage
  • Adds cardano-node-schedule-restart nixosModule and associated perSystem packages
  • Adds new alerts for cardano_node_elevated_restarts
  • Fixes dashboard for cardano-node legacy and new tracing application metrics to always show the full environment KES period

Cardano-ops (Legacy Mainnet)​

Commit-compare

Over a two week period the legacy relay nodes were scaled down to running only one instance of cardano-node per machine and then the number of running machines was further reduced by 25%.

Cardano-playground​

Node 9.1.0, Mithril 2430.0, Preview hardfork to Conway

Overview:​

Sets cardano-node (release) and cardano-node-ng (pre-release) versions to 9.1.0 and mithril to 2430.0. Hard forks preview network to Conway. Adds recipe and other improvements, including to the pool performance query recipe interface and a Chang readiness query.

Details:​

  • Bumps cardano-parts for:
    • Important versioning updates:
      • cardano-node and cardano-node-ng are now 9.1.0
    • Capkgs updates:
      • For node releases of 9.1.0
      • For mithril 2430.0 and mithril-unstable
  • Adds a new template sql script with a Chang era readiness sql query
  • Adds a babbage-to-conway cost model to the Cardano book
  • Adds a new recipe kes-rotate for easy kes rotation
  • Adds new alerts for cardano_node_elevated_restarts
  • Adds a commit stamp marker for Cardano book updates
  • Adds a new template-clone recipe for mirroring upstream template files when diffing or patching isn't needed
  • Updates the Cardano book environment for cardano-node 9.1.0
  • Updates the explainer docs for kes-rotation, chain-manipulation, new-network
  • Updates the preview faucet for new govtool operations
  • Rotates sanchonet KES, resizes the metadata server
  • Investigates mempool rejections with new tracing system and modified logging
  • Tests a comparison set of machines in mainnet environment for node 9.1.0 and utxo-hd-9.0
  • Tests a new tracing system branch for metrics renaming and KES metrics update calculations
  • Moves some cardano-playground specific recipes to the scripts/recipes-custom.just module
  • Improves template recipe dbsync-pool-analyze and sql query with parameters to query any CTE in the large dbsync-pool-perf sql from cli
  • Improves template recipe dbsync-pool-analyze to handle queries that result in no non-performing pools
  • Improves template recipe dedelegate-pools with a mempool query instead of a fixed time to handle UTxO on-chain settlement
  • Hard forks preview environment to Conway and resizes one relay member of each preview pool group

Iohk-nix​

Add conway config for mainnet/preprod/preview

Devx-ci​

Fix Hydra alerting immediately on no data

  • Migrate from Grafana Cloud to our self-hosted cardano-monitoring stack
    • Do not filter metrics to keep down number of unique series
    • This allows unlimited collection of metrics from our CI machines for better alerts and measurements for ongoing performance tuning.
  • Upgrade disko partition names manually on remaining machines ci{2,3,4,6,7,8} so boot does not break on the next deployment
  • Grafana dashboard: fix memory usage graph
  • Add alerts

Cardano-monitoring​

Commit-compare

  • Add preliminary support for Loki for log collection