Skip to main content

45 posts tagged with "sre"

View All Tags

· One min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano explorer is being sunset and will soon be replaced by a new Cardano Foundation explorer
  • IOG mainnet pools are now participating in propagation time reporting using Cardano Foundation's blockperf tool
  • Cardano-parts node nixosConfigurations, node entrypoint and process-compose node jobs now utilize mithril client automatically on preview and preprod networks for faster synchronization of first start up. Mainnet mithril client use as well as signer modules will be available soon.

Lower level summary​

Cardano-mainnet​

Cardano-parts​

  • Adds a block performance module which utilizes cardano-foundation's blockperf for aggregate block propagation reporting: cardano-parts-pull-29
  • Adds a mithril client for nixosConfigurations, process-compose cardano-node processes and cardano-node entrypoint: cardano-parts-pull-30

Iohk-nix​

  • Adds required mithril client environment information for preview, preprod, mainnet: iohk-nix-pull-570

Cardano-world​

Upstream Contributions​

Blockperf​

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node 8.7.3 is now generally deployed to all testnet and mainnet environments managed by coretech SRE

  • Dbsync and node stack process-compose jobs are now available from cardano-parts for users running nix >= 2.17.0 and nix experimental-features = nix-command flakes fetch-closure

  • These can be run with:

     nix run github:input-output-hk/cardano-parts#run-process-compose-node-stack
     nix run github:input-output-hk/cardano-parts#run-process-compose-dbsync-$NETWORK

  • For more details, see the merged cardano-parts process-compose PR

Lower level summary​

Cardano-monitoring​

Cardano-mainnet​

  • Adds a readme, switches to nonmoving gc for producers, plus misc improvements from cardano-parts: cardano-mainnet-pull-6

Cardano-ops​

  • Merged a long standing branch converting legacy mainnet relays to p2p, node -> 8.7.2, db-sync snapshots -> 13.1.1.3, and other improvements: cardano-ops-pull-417

Cardano-parts​

  • Adds a readme, provides misc improvements, service optimizations, alert tuning, sql pool performance analysis fix, package updates: cardano-parts-pull-27
  • Adds process-compose dbsync and node stacks: cardano-parts-pull-28

Cardano-playground​

Upstream Contributions​

  • Contributions to upstream process-compose related repos were made in order to complete the process-compose dbsync and node stacks in cardano-parts, including the following:

Process-compose-flake​

Services-flake​

· One min read
John Lotoski

High level summary​

During the lightly staffed holiday period for node SRE, the emphasis was on maintaining environment stability, tuning and resolving any noisey alerts.

Investigation into and testing around the following two topics also started during this period:

  • Ledger snapshots causing a small number of missed slots for forgers on mainnet: ouroboros-consensus-issue-868

  • A cardano-node rare file descriptor leak, with a more detailed description here

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • A new repository was created which enables agile deployment of EC2 monitoring servers, compatible with OpenTofu grafana and mimir providers: cardano-monitoring
  • The govtool backend swagger interface was nix flake packaged and deployed for Voltaire private chain testing usage
  • Grafana cloud monitoring stacks were migrated to new EC2 cardano-monitoring servers
  • Cardano-db-sync state snapshots now support client range requests, details here
  • In addition to monitoring server centralized grafana metrics, sysstat collected system metrics are now available locally on all cluster machines at high time resolution
  • Code changes required due to repository migrations to IntersectMBO have largely been completed

Lower level summary​

Auth-keys-hub​

Cardano-monitoring​

  • A new repository enabling agile deployment of EC2 monitoring servers, compatible with OpenTofu grafana and mimir providers: cardano-monitoring

Cardano-parts​

  • Migrate from grafana cloud monitoring to ec2 monitoring, add resource tagging support, declarative route53 CNAME list, and additional improvements and fixes: cardano-parts-pull-25
  • Improve ssh key handling and edge cases, resolve misc issues, add IOPS and throughput gp3 openTofu support: cardano-parts-pull-26

Cardano-playground​

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Sanchonet was respun to cardano-node 8.7.0-pre, and upgraded to cardano-node 8.7.1-pre shortly afterwards
  • Cardano-node 8.7.2 was released and all environments were then upgraded to 8.7.2
  • Cardano-parts deployed machines were upgraded to nixpkgs 23.11 and nix 2.19.3
  • Cardano-parts resource deployed environments were switched from the use of Terraform to OpenTofu

Lower level summary​

Capkgs​

  • Updated for cardano-node 8.7.2 and process-compose packages: capkgs-compare

Cardano-parts​

Cardano-ops​

Cardano-playground​

Cardano-world​

  • Sanchonet update PR: cardano-world-pull-111
    • Merge the long running sanchonet-updated branch
    • Migrate explorers from ziti to wireguard tunnel usage
    • Remove remaining ziti code and provisioned resources
    • Retire remaining nomad jobs in preference of the cardano-playground environments
    • Downsize the cluster in preference of the cardano-playground environments

Iohk-nix​