33 posts tagged with "sre"

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

• Cardano-world testnets of preprod, preview, sanchonet and some private test chains are in the process of being migrated to the cardano-playground stack

Lower level summary​

Capkgs​

• Adds offchain-metadata-tools, dbsync sanchonet updates: capkgs-compare

Cardano-parts​

• General package updates, modules improvements and template recipes to support network migration from world to playground cardano-parts-pull-17
  • Bumps cardano-db-sync-ng to sancho-2-0-0 tag
  • Bumps iohk-nix-ng to mig-sancho branch for sanchonet pool migration from world to play
  • Adds more machine system bins and devShell bins for scripting and debug purposes
  • Adds cardano-show-kes-period alias on any node machine importing profile-cardano-node-group module
  • Adds profile-cardano-node-topology module for a simplified interface to most common topology needs
  • Adds a job-delegate-rewards-stake-key job as an optional follow on to pool creation and registration jobs
  • Adds a topology function to filter self from group machines with an allowList for matching infixes
  • Adds metadata-server and related offchain-metadata-tools bins from capkgs
  • Updates justfile template with:
    • a new query-all recipe for getting status of multiple concurrent running environments
    • a new set-default-cardano-env recipe for fast switching between environments
    • a new start-demo recipe for forking a custom env into conway
    • a new start-node recipe for generic environment start
    • a new stop-node recipe for generic environment stop
    • updated list-machines recipe for handling of empty nixos machine config and empty ssh_config conditions
    • updated query-tip recipe to a generic query tip compatible with each environment

Cardano-playground​

• Migrates sanchonet from world, prepares further migration and support automation: cardano-playground-pull-4

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

• Sanchonet environment was updated to 8.5.0-pre.
• Cardano-parts now supports cardano-db-sync, cardano-smash, cardano-faucet and grafana cloud monitoring

Lower level summary​

Capkgs​

• Various improvements and fixes including:
  • Add GHA cron schedule
  • Add nix auto-gc to avoid running out of storage during large package set builds
  • Add new packages to capkgs
  • Reduce runner storage requirement leaving more room for builds
  • Restructure capkgs attribute names to avoid package name collisions
  • Return to non-musl builds for cardano packages to retain journald compatability
  • Update cache usage to from file level to folder level to reduce network and latency overhead
  • Commit diff: capkgs-compare

Cardano-parts​

• Updates cardano-node-ng to 8.5.0-pre and adds a Conway era automation job: cardano-parts-pull-16
• Dbsync, smash, faucet and more: cardano-parts-pull-15
  • Adds cardano-db-sync, cardano-faucet, cardano-postgres, cardano-smash, profiles and/or services and related changes
  • Adds nginx vhost metrics exporter profile
  • Adds smash registered-relay-dump service and exporter for use until legacy relay nodes are scaled down
  • Adds bash *-ng autocompletion compatible wrappers
  • Adds a list-machines just recipe using nushell dataframe outer joins and scj ssh_config parser for fast cluster evals of machine state overview
  • Adds downstream grafana cloud dashboard as templates
  • Adds downstream grafana cloud alerts as templates
  • Updates grafana-agent profile with new exporter scrape hooks: cardano-db-sync, cardano-faucet, nginx-vts, varnish
  • Updates the basic profile with IOG cache and commonly used bins
  • Updates the pre-release profile to support cardano-db-sync, cardano-faucet, cardano-smash *-ng versioning
  • Updates flakeModule jobs with new conway era automation and additional IO encryption shimming and file type checks
  • Updates .sops.yaml template for supporting faucet secrets, workbench secrets, state-demo secrets
  • Updates the Justfile template with terraform fixes for workspace switching and provider auto-reconfiguration
  • Updates the cloudFormation terraformState template with stack modifications to preserve all resources in case of deletion
  • Updates the colmena template with dbsync, smash, faucet machines profiles and roles
  • Improves prior cardano-postgres modules to now automatically tune pg parameters based on machine cpuCount, memMiB and desired conns
  • Bumps capkgs node-ng to 8.5.0-pre

Cardano-playground​

• Dbsync, smash, faucet and more: cardano-playground-pull-3
  • Adds a list-machines just recipe using nushell dataframe outer joins and scj ssh_config parser for fast cluster evals of machine state overview
  • Adds dbsync, smash, faucet machines and corresponding metrics exporters, dashboards and alerts
  • Moves the flake.cardano-parts.cluster.group attrSet name to groups to accurately reflect the plurality and the upstream corresponding change
  • Optimizes machine sizes
  • Updates .sops.yaml for supporting faucet secrets, workbench secrets, state-demo secrets
  • Updates the cloudFormation terraformState file with stack modifications to preserve all resources in case of deletion
  • Updates the cluster isNg definition to support cardano-db-sync, cardano-faucet, cardano-smash *-ng versioning
  • Updates the Justfile with terraform fixes for workspace switching and provider auto-reconfiguration

Cardano-world​

• Update sanchonet network to cardano-node 8.5.0: cardano-world-commit

Sanchonet-demo​

• Update for cardano-node 8.5.0, conway job recipes and cardano-parts interface changes: sanchonet-demo-commit

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

• Sanchonet environment was re-spun starting from slot 7171200 and updated to cardano-node 8.4.0-pre.
• The use of cardano-node docker hub will be deprecated in preference of GHCR

Lower level summary​

Capkgs​

• Refactor parsing scripts, add github action automation, various bugfixes and cleanup: capkgs-compare

Cardano-parts​

• Updates secrets layout scheme, adds sops enc/dec for jobs, adds cloud monitoring profile, updates flake templates and other improvements/fixes: cardano-parts-pull-8

Cardano-playground​

• Updates for new cardano-parts secrets handling and layout, TF workspace handling, group multivalue DNS support, grafana cloud monitoring and other improvements: cardano-playground

Cardano-world​

• Update sanchonet network to cardano-node 8.4.0: cardano-world-compare
• Rotate KES on shelley-qa: cardano-world-commit

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent updates or improvements include:

• Sanchonet environment was updated to 8.3.0-pre.
• Cardano-parts now enables downstream consumers to spin up multiple cardano-networks per repository, an example of which is here.

Lower level summary​

Capkgs​

• Add a readme, optimize flake inputs, improve nu and ruby parsing scripts: capkgs-compare

Cardano-ops​

• Adjust mainnet p2p relays to ~100%, with a legacy canary: cardano-ops-commit

Cardano-parts​

• Utilize content addressed binaries to significantly improve eval, build and devShell performance: cardano-parts-pull-8
• Add an aws flakeModule for obtaining ec2 specs, fix node.socket devShell error: cardano-parts-pull-10
• Fix warns thrown during some builds and devShell usage: cardano-parts-pull-11
• Add flakeModules and nixosModules in preparation for cardano-node grouped deployments: cardano-parts-pull-12
• Enable group deployments by: adding several module, profile and role nixosModules; updating flakeModules; adding sops age secrets handling: cardano-parts-pull-13

Cardano-perf​

• Modify the wireguard network from mesh to star, enable ICMP, enable Nomad raw_exec and add hydraJobs: cardano-perf-compare

Cardano-playground​

• Define preliminary cardano-world networks on cardano-playground: cardano-playground

Cardano-world​

• Rotate KES on preview and preprod, purge unused subnets, adjust explorer rate limits: cardano-world-compare
• Update sanchonet network to 8.3.0: cardano-world-compare

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent updates or improvements include:

• Work on cardano-parts repo continues and was utilized by the sanchonet-demo repository for demonstration purposes
• Work on a new capkgs repository started -- "Content Addressed Packages"
  • This repository provides common Cardano release and pre-release binaries in a pure content addressed nix package type via flake output
  • These packages don't have the associated IFD, long eval times and large source dependencies that the originating source repositories often do
  • Consumers of such packages, such as devShells or nixos machines, can benefit from the improved performance this offers

Lower level summary​

Capkgs​

• Content addressed packages -- new repo: capkgs

Cardano-ops​

• Cardano p2p relay updates, dbSync snapshots server update, monitoring adjustments: cardano-ops-compare

Cardano-parts​

• Flake devShell module improvements: cardano-parts-pull-4
• Cardano-world entrypoints migrated to flake parts module: cardano-parts-pull-5
• Cardano-world jobs migrated to flake parts module: cardano-parts-pull-6
• Drep delegation fixups: cardano-parts-pull-7
• Utilize content addressed binaries, WIP: cardano-parts-pull-8
• Cardano-node module preparation branch, WIP: cardano-parts-compare

Cardano-perf​

• Availability zone pinning and explorer mods: cardano-perf-compare

Cardano-world​

• Remove performance nodes for migration to cardano-perf: cardano-world-commit

Sanchonet-demo​

• Sanchonet demonstration repository: sanchonet-demo