Skip to main content

39 posts tagged with "sre"

View All Tags

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node 9.0.0 is now deployed to mainnet, preprod, preview, private and shelley-qa environments. The last several weeks have been very busy with pre-release and release activity and environment upgrades involving cardano-node versions 8.9.3, 8.9.4, 8.12.0-pre, 8.12.1, 8.12.2 and now 9.0.0 as of this update.

  • Sanchonet environment remains pinned at cardano-node version 8.11.0-pre until the next respin which will support 9.0.0 or greater.

  • Ogmios service and package options were added to cardano-parts.

  • Four documents were added to cardano-playground to better explain some operational procedures: debugging of peer-to-peer connections; governance voting with the playground stakepools; faucet setup; faucet pool de-delegation. Found at: docs/explain

  • One document was added to cardano-mainnet to explain cardano-snapshot operations. Found at: docs/explain

  • Private chain was stopped and re-spun with 2 hr epochs for testing.

  • Hydra and performance cluster machines had their configuration updated to be more robust to transient nix store caches outages which may re-occur in the future.

  • All machines in cardano-playground and cardano-mainnet clusters were updated to nixpkgs 24.05.

Lower level summary​

Cardano-mainnet​

  • Sets cardano-node to 8.12.2 as well as usage of a custom gc delay parameter branch for bootstrap nodes. Updates all machines to nixpkgs to 24.05 with openssh 9.8p1. Adds one new explainer readme document, new alerts and various script, recipe, and other improvements. See the PR description for more details: cardano-mainnet-pull-16

Cardano-ops​

  • Bumps to cardano-node 9.0.0, adds coredump metrics, adds OOM/coredump alerting, adjusts systemd stop timeout to avoid some unneccesary chain replays: cardano-ops-compare

Cardano-parts​

  • Sets cardano-node (release) and cardano-node-ng (pre-release) versions to 8.12.2 and cardano-db-sync-ng to sancho-5-0-0. Updates nixpkgs to 24.05. Includes nixosModule, dashboard, metric, alert and recipe improvements and new features. More detail is available in the PR description: cardano-parts-pull-43

Cardano-perf​

  • Adjusts nix config to avoid R2 500 errors on transient cache problems and adds explorer to perf class: cardano-perf-compare

Cardano-playground​

  • Sets cardano-node (release) and cardano-node-ng (pre-release) versions to 8.12.2 and cardano-db-sync-ng to sancho-5-0-0. Updates all machines to nixpkgs to 24.05 with openssh 9.8p1. Respins private chain and KES rotates multiple chains. Adds four new explainer readme documents, new alerts and various script, recipe, and other improvements. See the PR description for more details: cardano-playground-pull-27

Iohk-nix​

Ops-lib​

  • Updates deployers with recent nixpkgs, nix, refactors to preserve legacy nixops usage, adds starship and fzf: ops-lib-pull-134

  • Bumps openssh to 9.8p1 ops-lib-pull-135

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Sanchonet was respun for cardano-node 8.11.0-pre

  • Private chain was respun twice for pre-sancho respin testing and short epoch testing with cardano-node 8.11.0-pre

  • Shelley-qa, two-thirds of preview and one-third of preprod networks were deployed to cardano-node 8.11.0-pre

  • Sanchonet, private chain and shelley-qa networks had dbsync sancho-4-3-0 deployed

  • A dbsync show_current_forging prepared statement was added to the cardano-parts profile-cardano-postgres nixosModule to aid with debugging chain quality issues

  • Three documents were added to cardano-playground to better explain some operations procedures: KES rotation, chain quality debugging and new network creation. Found at: docs/explain

  • A new mithril dashboard template is available in cardano-parts

Lower level summary​

Capkgs:​

  • Avoid git API rate limit errors on update github action via netrc usage and corresponding secret: capkgs-commit

Cardano-parts​

  • Sets cardano-node-ng to 8.11.0-pre and cardano-db-sync-ng to sancho-4-3-0. Adds a dbsync prepared statement, mithril dashboard template, updates the node application dashboard template, improves justfile recipe templates and tunes some systemd dependencies. Iohk-nix-ng was updated for sanchonet and private chain respins. More detail is available in the PR description: cardano-parts-pull-41

Cardano-mainnet​

  • Rotates KES, pins iogp4 as -ng, adds a mithril dashboard, updates the node application dashboard, improves justfile recipes and tunes systemd node and mithril services to avoid some edge case errors. See the PR description for more details: cardano-mainnet-pull-15

Cardano-ogmios​

Cardano-playground​

  • Respins sancho and private chains and deploys cardano-node 8.11.0-pre and cardano-db-sync sancho-4-3-0 to appropriate envs and machines. Adds a mithril dashboard template, updates the node application dashboard template, improves justfile recipe templates. Adds three new explainer readme documents. See the PR description for more details: cardano-playground-pull-24

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • A preview of the new cardano tracing system is available by using the profile-cardano-node-new-tracing new nixosModule from cardano-parts and is complemented with new tracing system dashboards and alerts

  • Custom cardano metrics can be pushed to a statsd server and then scraped to a monitoring solution using the profile-cardano-custom-metrics new nixosModule from cardano-parts

  • A new cardano-parts job for spinning up clusters using a recent cardano-cli command, create-testnet-data, was implemented: .#job-gen-custom-node-config-data

Lower level summary​

Cardano-faucet​

Cardano-parts​

  • Sets cardano-faucet-ng to 8.10, and mithril[-ng] to 2418-1. A preview nixosModule for cardano-node's new tracing system has been added as well as a cardano custom-metrics module. Dashboards have been added for the new tracing system. Cardano-node alerts have been refactored into several rulesets which include some new alerts as well as new tracing system alerts. A job for spinning a cardano cluster up using a new underlying cardano-cli mechanism, create-testnet-data, was added. More detail is available in the PR description: cardano-parts-pull-40

Cardano-mainnet​

  • Sets mithril to 2418-1. Cardano-node alerts have been refactored into several rulesets which include some new alerts as well as new tracing system alerts. See the PR description for more details: cardano-mainnet-pull-14

Cardano-perf​

Cardano-playground​

  • Sets cardano-faucet-ng to 8.10 which fixes a sanchonet faucet instability, and mithril[-ng] to 2418-1. Adds a new cardano-node tracing system preview to some machines with corresponding dashboards. Cardano-node alerts have been refactored into several rulesets which include some new alerts as well as new tracing system alerts. A job for spinning a cardano cluster up using a new underlying cardano-cli mechanism, create-testnet-data, was added and integrated with the Justfile start-demo recipe. Includes other misc tuning and improvements. See the PR description for more details: cardano-playground-pull-23

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node 8.9.2 is now deployed to mainnet, preprod, preview and shelley-qa environments.

  • Cardano-node 8.10.1-pre is now deployed to sanchonet and also to one-third of IOGs preprod environment nodes and two-thirds of IOGs preview environment nodes.

  • Private chain for Voltaire team was paused with plans for a future respin.

  • The network team's ouroboros-network-ops machine cluster was re-written using the cardano-parts stack to upgrade from the nixops/terraform/niv stack it was previously using.

Lower level summary​

Cardano-parts​

  • Sets cardano-node to 8.9.2, dbsync-ng to sancho-4.2.0; mithril to 2412.0, iohk-nix to include new peerSharing defaults and introduce a new block producer config. Adds a new truncate-chain recipe and improves mithril related services. More detail is available in the PR description: cardano-parts-pull-38

  • Sets cardano-node pre (-ng) to 8.10.1, dbsync to 13.2.0.2; mithril pre (-ng) to unstable, iohk-nix and iohk-nix-ng pin includes new Cardano Foundation bootstrap relays. Adds new aws machine management and other recipes, metadata job support for pool creation, misc fixes and improvements. More detail is available in the PR description: cardano-parts-pull-39

Cardano-mainnet​

  • Sets cardano-node to 8.9.2, mithril to 2412.0, iohk-nix to include new peerSharing defaults and colmena.nix cluster refactor for peerSharing adjustments and implements all updates in cardano-parts PR#38. See the PR description for more details: cardano-mainnet-pull-12

  • Sets dbsync to 13.2.0.2, iohk-nix and iohk-nix-ng to include new CF relays, adds new aws machine management recipes and implements all updates in cardano-parts PR#39. See the PR description for more details: cardano-mainnet-pull-13

Cardano-node​

  • Default peerSharing true and add block producer config to release binaries. See the PR description for more details: cardano-node-pull-5789

Cardano-ops​

Cardano-perf​

  • Adds a caddy webserver for run reviews and tunes the NVME FS mounts for performance: cardano-perf-compare

Cardano-playground​

  • Sets cardano-node to 8.9.2, mithril to 2412.0, iohk-nix to include new peerSharing defaults, refactors mdbook out of docs dir, adds example chain manipulation doc and implements all updates in cardano-parts PR#38. See the PR description for more details: cardano-playground-pull-21

  • Sets cardano-node pre (-ng) to 8.10.1, dbsync to 13.2.0.2, mithril pre (-ng) to unstable, iohk-nix and iohk-nix-ng to include new CF relays, adds a public TLS dbsync user connection option, stops private chain cluster and implements all updates in cardano-parts PR#39. See the PR description for more details: cardano-playground-pull-22

Iohk-nix​

Ouroboros-network-ops​

Sanchonet​

Sanchonet-demo​

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Private chain for Voltaire team was respun for cardano-node 8.10.0-pre pre-release and re-hard forked into Conway era.

  • Sanchonet was respun for cardano-node 8.10.0-pre pre-release and re-hard forked into Conway era.

  • Cardano-node 8.10.0-pre was also deployed to one-third of IOGs preprod environment nodes and two-thirds of IOGs preview environment nodes.

Lower level summary​

Cardano-parts​

  • Sets cardano-node|cli-ng to 8.10.0-pre, dbsync-ng to sancho-4.1.0; bumps nixpkgs (23.11) and nixpkgs-unstable; improves, adds new and fixes a number of just recipes; allows repo custom recipes to be kept as a separate import for easy justfile maintenance; refactors ip module checking into its own nixosModule and adds an extra abort option for safety. Much more detail is available in the PR description: cardano-parts-pull-36

Cardano-mainnet​

  • Bumps cardano-parts for a number of recipe additions, improvements, fixes. Fixes blockperf dnsmasq looksups and adjusts colmena topology code to account for intra-cluster localRoots now defaulting to trustable true and implements all updates in cardano-parts PR#36. See the PR description for more details: cardano-mainnet-pull-11

Cardano-perf​

  • Add a new perf-ssd machine class and deployment for ssd related performance testing and benchmarking. Add new just recipes and cluster resource tagging. cardano-perf-compare

Cardano-playground​

  • Bumps cardano-parts for a number of recipe additions, improvements, fixes, and node 8.10.0-pre. Respins private chain and sanchonet for node 8.10.0-pre with corresponding book updates and implements all updates in cardano-parts PR#36. See the PR description for more details: cardano-playground-pull-20

Iohk-nix​

  • WIP: Prepares block producer configurations to accomodate upcoming peerSharing default change: iohk-nix-pull-575

  • Provides iohk-nix updates for node 8.10: iohk-nix-pull-576