Skip to main content

45 posts tagged with "sre"

View All Tags

· 4 min read
Michael Fellinger

High level summary​

The SRE team continues work on cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Our new baseline version of Cardano Node is 9.1.0 and all environments have been updated. The main change from node 9.0.0 is that node 9.1.0 requires a Conway genesis file at startup, where the genesis file was optional in node 9.0.0.

  • Sanchonet had another respin for node 9.0.0 with new Conway genesis parameters and has since been upgraded to node 9.1.0.

  • The cardano-monitoring cluster received a lot of documentation and improvements and now also serves as the home for devx-ci metrics.

  • Hydra CI performance was improved once again by changes to our custom Nix evaluator. We also found further ways to improve the resource usage of waiting for IFDs.

Cardano Airgap​

A new project that provides a completely airgapped environment for constitution members to sign proposals and transactions. It consists of an image for USB sticks and ensures all private data is stored securely with strong encryption.

We'll keep it updated as required with the latest Cardano versions.

cardano-airgap

Cardano Parts​

  • cardano-node and cardano-node-ng are now at version 9.0.0
  • cardano-db-sync and cardano-db-sync-ng are now at version 13.3.0.0
  • Several NixOS modules and recipes have been fixed and improved.
  • Bump dependency of capkgs for node, db-sync, mithril, and cardano-wallet updates.
  • Update profile-cardano-db-sync-snapshots for schema 13.3 docs and with script edge case fixes
  • Update profile-cardano-node-group to use a SIGINT instead of SIGTERM for systemd stop
  • Update profile-common to deploy atd service
  • Update template recipe dbsync-prep to match faucet script defaults
  • Update template recipe update-ips to fix a nushell breaking change
  • Update .envrc with a newer direnv version and allows for symlinks on .envrc.local and ~/.age/credentials

PR#44

Cardano Playground​

  • All networks are now running cardano-node 9.1.0 in preparation of the Chang hard-fork.
  • Also upgraded db-sync to 13.3.0.0
  • Added the cardano-ipfs module and a derivation for pinata-go-cli that is used to store and distribute documents that can be referenced on chain.
  • Some updates to the Cardano Operations Book about:
    • UseLedgerPeerAfter updates
    • Sanchonet respins configs
    • Dbsync EnableFutureGenesis flag
  • Add a block header block producer readiness test
  • Respin of sanchonet for node 9.0.0, then upgraded to 9.1.0
  • Tune webserver size and Varnish RAM to improve caching efficiency
  • Updates govtool module for multi-nginx module compatibility
  • Updates update-ips recipe for nushell breaking change in nixpkgs 24.05
  • Updates direnv version, allow symlinks config files used by direnv
  • Update .envrc with a newer direnv version, allow symlinks on direnv used config files

PR#28

Cardano Mainnet​

  • Upgraded Cardano Node to 9.0.0
  • Upgraded Cardano DB Sync to 13.3.0.0
  • Bump capkgs dependency
  • Investigate bootstrap missingBlock error and deploy fixes for it.
  • Update scripts to be compatible with latest nushell version

PR#17

Cardano Monitoring​

  • Write comprehensive documentation for all the Nix code, as well as detailed instructions for usage and deployment.
  • Overhaul most Just tasks to bring them more in sync with the other repositories
  • Upgrade all machines to NixOS 24.05
  • Upgrade auth-keys-hub to prevent lockout in case SOPS is unable to decrypt
  • Fix SOPS decryption failure on boot because of missing network.
  • Limit bootloader entries to 5 since the /boot partition is tiny
  • Additionally add fallback SSH keys for emergency use

PR#1

IOHK Nix​

  • Update ledger peers to be after a more recent epoch boundary to improve bootstrapping and fix a private chain p2p delayed sync config issue.
  • Update sanchonet conway-genesis for respin
    • DRep voting thresholds both need to be 65%
    • Set govActionLifetime above the guardrail because of the short epochs.
    • Set minCommitteeSize to 5 (from recommended 7) because only 5 ICC members were able to provide keys for the respin.

PR#584 PR#585

Cardano Ops​

  • tweak the stop timeout and change Cardano Node killsignal to SIGINT for clean restarts.

Diff

CAPkgs​

Added following packages:

  • cardano-node 9.0.0 and 9.1.0
  • For cardano-db-sync releases of sancho-5.1.0 and 13.3.0.0
  • For mithril 2428.0 and mithril-unstable
  • For cardano-wallet v2024-07-19

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node 9.0.0 is now deployed to mainnet, preprod, preview, private and shelley-qa environments. The last several weeks have been very busy with pre-release and release activity and environment upgrades involving cardano-node versions 8.9.3, 8.9.4, 8.12.0-pre, 8.12.1, 8.12.2 and now 9.0.0 as of this update.

  • Sanchonet environment remains pinned at cardano-node version 8.11.0-pre until the next respin which will support 9.0.0 or greater.

  • Ogmios service and package options were added to cardano-parts.

  • Four documents were added to cardano-playground to better explain some operational procedures: debugging of peer-to-peer connections; governance voting with the playground stakepools; faucet setup; faucet pool de-delegation. Found at: docs/explain

  • One document was added to cardano-mainnet to explain cardano-snapshot operations. Found at: docs/explain

  • Private chain was stopped and re-spun with 2 hr epochs for testing.

  • Hydra and performance cluster machines had their configuration updated to be more robust to transient nix store caches outages which may re-occur in the future.

  • All machines in cardano-playground and cardano-mainnet clusters were updated to nixpkgs 24.05.

Lower level summary​

Cardano-mainnet​

  • Sets cardano-node to 8.12.2 as well as usage of a custom gc delay parameter branch for bootstrap nodes. Updates all machines to nixpkgs to 24.05 with openssh 9.8p1. Adds one new explainer readme document, new alerts and various script, recipe, and other improvements. See the PR description for more details: cardano-mainnet-pull-16

Cardano-ops​

  • Bumps to cardano-node 9.0.0, adds coredump metrics, adds OOM/coredump alerting, adjusts systemd stop timeout to avoid some unneccesary chain replays: cardano-ops-compare

Cardano-parts​

  • Sets cardano-node (release) and cardano-node-ng (pre-release) versions to 8.12.2 and cardano-db-sync-ng to sancho-5-0-0. Updates nixpkgs to 24.05. Includes nixosModule, dashboard, metric, alert and recipe improvements and new features. More detail is available in the PR description: cardano-parts-pull-43

Cardano-perf​

  • Adjusts nix config to avoid R2 500 errors on transient cache problems and adds explorer to perf class: cardano-perf-compare

Cardano-playground​

  • Sets cardano-node (release) and cardano-node-ng (pre-release) versions to 8.12.2 and cardano-db-sync-ng to sancho-5-0-0. Updates all machines to nixpkgs to 24.05 with openssh 9.8p1. Respins private chain and KES rotates multiple chains. Adds four new explainer readme documents, new alerts and various script, recipe, and other improvements. See the PR description for more details: cardano-playground-pull-27

Iohk-nix​

Ops-lib​

  • Updates deployers with recent nixpkgs, nix, refactors to preserve legacy nixops usage, adds starship and fzf: ops-lib-pull-134

  • Bumps openssh to 9.8p1 ops-lib-pull-135

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Sanchonet was respun for cardano-node 8.11.0-pre

  • Private chain was respun twice for pre-sancho respin testing and short epoch testing with cardano-node 8.11.0-pre

  • Shelley-qa, two-thirds of preview and one-third of preprod networks were deployed to cardano-node 8.11.0-pre

  • Sanchonet, private chain and shelley-qa networks had dbsync sancho-4-3-0 deployed

  • A dbsync show_current_forging prepared statement was added to the cardano-parts profile-cardano-postgres nixosModule to aid with debugging chain quality issues

  • Three documents were added to cardano-playground to better explain some operations procedures: KES rotation, chain quality debugging and new network creation. Found at: docs/explain

  • A new mithril dashboard template is available in cardano-parts

Lower level summary​

Capkgs:​

  • Avoid git API rate limit errors on update github action via netrc usage and corresponding secret: capkgs-commit

Cardano-parts​

  • Sets cardano-node-ng to 8.11.0-pre and cardano-db-sync-ng to sancho-4-3-0. Adds a dbsync prepared statement, mithril dashboard template, updates the node application dashboard template, improves justfile recipe templates and tunes some systemd dependencies. Iohk-nix-ng was updated for sanchonet and private chain respins. More detail is available in the PR description: cardano-parts-pull-41

Cardano-mainnet​

  • Rotates KES, pins iogp4 as -ng, adds a mithril dashboard, updates the node application dashboard, improves justfile recipes and tunes systemd node and mithril services to avoid some edge case errors. See the PR description for more details: cardano-mainnet-pull-15

Cardano-ogmios​

Cardano-playground​

  • Respins sancho and private chains and deploys cardano-node 8.11.0-pre and cardano-db-sync sancho-4-3-0 to appropriate envs and machines. Adds a mithril dashboard template, updates the node application dashboard template, improves justfile recipe templates. Adds three new explainer readme documents. See the PR description for more details: cardano-playground-pull-24

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • A preview of the new cardano tracing system is available by using the profile-cardano-node-new-tracing new nixosModule from cardano-parts and is complemented with new tracing system dashboards and alerts

  • Custom cardano metrics can be pushed to a statsd server and then scraped to a monitoring solution using the profile-cardano-custom-metrics new nixosModule from cardano-parts

  • A new cardano-parts job for spinning up clusters using a recent cardano-cli command, create-testnet-data, was implemented: .#job-gen-custom-node-config-data

Lower level summary​

Cardano-faucet​

Cardano-parts​

  • Sets cardano-faucet-ng to 8.10, and mithril[-ng] to 2418-1. A preview nixosModule for cardano-node's new tracing system has been added as well as a cardano custom-metrics module. Dashboards have been added for the new tracing system. Cardano-node alerts have been refactored into several rulesets which include some new alerts as well as new tracing system alerts. A job for spinning a cardano cluster up using a new underlying cardano-cli mechanism, create-testnet-data, was added. More detail is available in the PR description: cardano-parts-pull-40

Cardano-mainnet​

  • Sets mithril to 2418-1. Cardano-node alerts have been refactored into several rulesets which include some new alerts as well as new tracing system alerts. See the PR description for more details: cardano-mainnet-pull-14

Cardano-perf​

Cardano-playground​

  • Sets cardano-faucet-ng to 8.10 which fixes a sanchonet faucet instability, and mithril[-ng] to 2418-1. Adds a new cardano-node tracing system preview to some machines with corresponding dashboards. Cardano-node alerts have been refactored into several rulesets which include some new alerts as well as new tracing system alerts. A job for spinning a cardano cluster up using a new underlying cardano-cli mechanism, create-testnet-data, was added and integrated with the Justfile start-demo recipe. Includes other misc tuning and improvements. See the PR description for more details: cardano-playground-pull-23

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node 8.9.2 is now deployed to mainnet, preprod, preview and shelley-qa environments.

  • Cardano-node 8.10.1-pre is now deployed to sanchonet and also to one-third of IOGs preprod environment nodes and two-thirds of IOGs preview environment nodes.

  • Private chain for Voltaire team was paused with plans for a future respin.

  • The network team's ouroboros-network-ops machine cluster was re-written using the cardano-parts stack to upgrade from the nixops/terraform/niv stack it was previously using.

Lower level summary​

Cardano-parts​

  • Sets cardano-node to 8.9.2, dbsync-ng to sancho-4.2.0; mithril to 2412.0, iohk-nix to include new peerSharing defaults and introduce a new block producer config. Adds a new truncate-chain recipe and improves mithril related services. More detail is available in the PR description: cardano-parts-pull-38

  • Sets cardano-node pre (-ng) to 8.10.1, dbsync to 13.2.0.2; mithril pre (-ng) to unstable, iohk-nix and iohk-nix-ng pin includes new Cardano Foundation bootstrap relays. Adds new aws machine management and other recipes, metadata job support for pool creation, misc fixes and improvements. More detail is available in the PR description: cardano-parts-pull-39

Cardano-mainnet​

  • Sets cardano-node to 8.9.2, mithril to 2412.0, iohk-nix to include new peerSharing defaults and colmena.nix cluster refactor for peerSharing adjustments and implements all updates in cardano-parts PR#38. See the PR description for more details: cardano-mainnet-pull-12

  • Sets dbsync to 13.2.0.2, iohk-nix and iohk-nix-ng to include new CF relays, adds new aws machine management recipes and implements all updates in cardano-parts PR#39. See the PR description for more details: cardano-mainnet-pull-13

Cardano-node​

  • Default peerSharing true and add block producer config to release binaries. See the PR description for more details: cardano-node-pull-5789

Cardano-ops​

Cardano-perf​

  • Adds a caddy webserver for run reviews and tunes the NVME FS mounts for performance: cardano-perf-compare

Cardano-playground​

  • Sets cardano-node to 8.9.2, mithril to 2412.0, iohk-nix to include new peerSharing defaults, refactors mdbook out of docs dir, adds example chain manipulation doc and implements all updates in cardano-parts PR#38. See the PR description for more details: cardano-playground-pull-21

  • Sets cardano-node pre (-ng) to 8.10.1, dbsync to 13.2.0.2, mithril pre (-ng) to unstable, iohk-nix and iohk-nix-ng to include new CF relays, adds a public TLS dbsync user connection option, stops private chain cluster and implements all updates in cardano-parts PR#39. See the PR description for more details: cardano-playground-pull-22

Iohk-nix​

Ouroboros-network-ops​

Sanchonet​

Sanchonet-demo​