75 posts tagged with "sre"

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

• Cardano explorer is being sunset and will soon be replaced by a new Cardano Foundation explorer
• IOG mainnet pools are now participating in propagation time reporting using Cardano Foundation's blockperf tool
• Cardano-parts node nixosConfigurations, node entrypoint and process-compose node jobs now utilize mithril client automatically on preview and preprod networks for faster synchronization of first start up. Mainnet mithril client use as well as signer modules will be available soon.

Lower level summary​

Cardano-mainnet​

• Adds a blockperf reporting module to the bootstrap nodes and block producer relays: cardano-mainnet-pull-7

Cardano-parts​

• Adds a block performance module which utilizes cardano-foundation's blockperf for aggregate block propagation reporting: cardano-parts-pull-29
• Adds a mithril client for nixosConfigurations, process-compose cardano-node processes and cardano-node entrypoint: cardano-parts-pull-30

Iohk-nix​

• Adds required mithril client environment information for preview, preprod, mainnet: iohk-nix-pull-570

Cardano-world​

• Add the sunset banner to explorer: cardano-world-commit

Upstream Contributions​

Blockperf​

• Add a nix flake with blockperf package and ci action: blockperf-pull-22

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

• Cardano-node 8.7.3 is now generally deployed to all testnet and mainnet environments managed by coretech SRE

• Dbsync and node stack process-compose jobs are now available from cardano-parts for users running nix >= 2.17.0 and nix experimental-features = nix-command flakes fetch-closure

• These can be run with:

   nix run github:input-output-hk/cardano-parts#run-process-compose-node-stack
   nix run github:input-output-hk/cardano-parts#run-process-compose-dbsync-$NETWORK
  

• For more details, see the merged cardano-parts process-compose PR

Lower level summary​

Cardano-monitoring​

• Adds code documentation and some nix refactoring: cardano-monitoring-pull-1

Cardano-mainnet​

• Adds a readme, switches to nonmoving gc for producers, plus misc improvements from cardano-parts: cardano-mainnet-pull-6

Cardano-ops​

• Merged a long standing branch converting legacy mainnet relays to p2p, node -> 8.7.2, db-sync snapshots -> 13.1.1.3, and other improvements: cardano-ops-pull-417

Cardano-parts​

• Adds a readme, provides misc improvements, service optimizations, alert tuning, sql pool performance analysis fix, package updates: cardano-parts-pull-27
• Adds process-compose dbsync and node stacks: cardano-parts-pull-28

Cardano-playground​

• Updates the readme, adds mainnet missedSlot testing, rotates KES on 4 networks, plus misc improvements from cardano-parts: cardano-playground-pull-13
• Adds process-compose dbsync and node stacks from cardano-parts flakeModule: cardano-playground-pull-14

Upstream Contributions​

• Contributions to upstream process-compose related repos were made in order to complete the process-compose dbsync and node stacks in cardano-parts, including the following:

Process-compose-flake​

• Adds apiServer process-compose cli option: process-compose-flake-pull-54
• Add a pre- and post- hook for the process-compose wrapper: process-compose-flake-pull-56

Services-flake​

• Adds socketDir option for postgres service: services-flake-pull-77
• Ensure socketDir mktemp init succeeds for postgres service: services-flake-pull-81

High level summary​

During the lightly staffed holiday period for node SRE, the emphasis was on maintaining environment stability, tuning and resolving any noisey alerts.

Investigation into and testing around the following two topics also started during this period:

• Ledger snapshots causing a small number of missed slots for forgers on mainnet: ouroboros-consensus-issue-868

• A cardano-node rare file descriptor leak, with a more detailed description here

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

• A new repository was created which enables agile deployment of EC2 monitoring servers, compatible with OpenTofu grafana and mimir providers: cardano-monitoring
• The govtool backend swagger interface was nix flake packaged and deployed for Voltaire private chain testing usage
• Grafana cloud monitoring stacks were migrated to new EC2 cardano-monitoring servers
• Cardano-db-sync state snapshots now support client range requests, details here
• In addition to monitoring server centralized grafana metrics, sysstat collected system metrics are now available locally on all cluster machines at high time resolution
• Code changes required due to repository migrations to IntersectMBO have largely been completed

Lower level summary​

Auth-keys-hub​

• Add extra nixos module assertions and warnings to address edge cases: auth-keys-hub-pull-6

Cardano-monitoring​

• A new repository enabling agile deployment of EC2 monitoring servers, compatible with OpenTofu grafana and mimir providers: cardano-monitoring

Cardano-parts​

• Migrate from grafana cloud monitoring to ec2 monitoring, add resource tagging support, declarative route53 CNAME list, and additional improvements and fixes: cardano-parts-pull-25
• Improve ssh key handling and edge cases, resolve misc issues, add IOPS and throughput gp3 openTofu support: cardano-parts-pull-26

Cardano-playground​

• Migrate from grafana cloud monitoring to ec2 monitoring, add resource tagging support, declarative route53 CNAME list, and additional improvements and fixes: cardano-playground-pull-10
• Update auth-keys-hub config to address edge cases: cardano-playground-pull-11
• Add a voltaire govtool backend swagger UI for private chain, misc fixes and updates: cardano-playground-pull-12

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

• Sanchonet was respun to cardano-node 8.7.0-pre, and upgraded to cardano-node 8.7.1-pre shortly afterwards
• Cardano-node 8.7.2 was released and all environments were then upgraded to 8.7.2
• Cardano-parts deployed machines were upgraded to nixpkgs 23.11 and nix 2.19.3
• Cardano-parts resource deployed environments were switched from the use of Terraform to OpenTofu

Lower level summary​

Capkgs​

• Updated for cardano-node 8.7.2 and process-compose packages: capkgs-compare

Cardano-parts​

• Update scripts for 8.6.0-pre: cardano-parts-pull-21
• Sanchonet respin with additional lib refactor and postgres prepared statement update: cardano-parts-pull-22
• Nixpkgs 23.11 update with migration from Terraform to OpenTofu: cardano-parts-pull-23
• Node 8.7.2 release update and process-compose flakeModule addition: cardano-parts-pull-24

Cardano-ops​

• Mainnet high load relays were updated to cardano-node 8.7.2 cardano-ops-compare

Cardano-playground​

• Sancho respin plus related book updates and network debug reference code: cardano-playground-pull-7
• Nixpkgs 23.11 update with required module changes, lint updates and tofu utilization.: cardano-playground-pull-8
• Node 8.7.2 release and process-compose addition, misc fixes: cardano-playground-pull-9

Cardano-world​

• Sanchonet update PR: cardano-world-pull-111
  • Merge the long running sanchonet-updated branch
  • Migrate explorers from ziti to wireguard tunnel usage
  • Remove remaining ziti code and provisioned resources
  • Retire remaining nomad jobs in preference of the cardano-playground environments
  • Downsize the cluster in preference of the cardano-playground environments

Iohk-nix​

• Update mainnet p2p config: iohk-nix-pull-566
• Update mainnet default mempool tracing: iohk-nix-pull-567