Skip to main content

51 posts tagged with "sre"

View All Tags

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Sanchonet state was respun from slot 14255995 for cardano-node 8.8.0-pre
  • Cardano node 8.8.0-pre is now deployed to sanchonet, shelley-qa, two-thirds of preprod and one-third of preview networks
  • IOG pool block producers are now signing mithril certificates for preview, preprod and mainnet networks
  • A cardano performance dashboard is now available for both playground and cardano-mainnet deployments

Lower level summary​

Cardano-mainnet​

  • Adds mithril signers and perf dashboard to the mainnet cluster; improves network docs, tunes alerts, plus misc fixes: cardano-mainnet-pull-8

Cardano-parts​

  • Adds mithril signer capability to role-block-producer nixosModule, adds a profile-mithril-relay nixosModule and includes misc other improvements and fixes: cardano-parts-pull-31

  • Updates packages for sanchonet respin, adds a node performance dashboard template and fixes misc issues: cardano-parts-pull-32

Cardano-parts​

  • Adds mithril signers and mithril relays to preview and preprod networks and includes misc other improvements and fixes: cardano-playground-pull-15

  • Updates machines and pin for sanchonet respin, adds a node performance dashboard and fixes misc issues: cardano-playground-pull-16

Iohk-nix​

· One min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano explorer is being sunset and will soon be replaced by a new Cardano Foundation explorer
  • IOG mainnet pools are now participating in propagation time reporting using Cardano Foundation's blockperf tool
  • Cardano-parts node nixosConfigurations, node entrypoint and process-compose node jobs now utilize mithril client automatically on preview and preprod networks for faster synchronization of first start up. Mainnet mithril client use as well as signer modules will be available soon.

Lower level summary​

Cardano-mainnet​

Cardano-parts​

  • Adds a block performance module which utilizes cardano-foundation's blockperf for aggregate block propagation reporting: cardano-parts-pull-29
  • Adds a mithril client for nixosConfigurations, process-compose cardano-node processes and cardano-node entrypoint: cardano-parts-pull-30

Iohk-nix​

  • Adds required mithril client environment information for preview, preprod, mainnet: iohk-nix-pull-570

Cardano-world​

Upstream Contributions​

Blockperf​

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node 8.7.3 is now generally deployed to all testnet and mainnet environments managed by coretech SRE

  • Dbsync and node stack process-compose jobs are now available from cardano-parts for users running nix >= 2.17.0 and nix experimental-features = nix-command flakes fetch-closure

  • These can be run with:

     nix run github:input-output-hk/cardano-parts#run-process-compose-node-stack
     nix run github:input-output-hk/cardano-parts#run-process-compose-dbsync-$NETWORK

  • For more details, see the merged cardano-parts process-compose PR

Lower level summary​

Cardano-monitoring​

Cardano-mainnet​

  • Adds a readme, switches to nonmoving gc for producers, plus misc improvements from cardano-parts: cardano-mainnet-pull-6

Cardano-ops​

  • Merged a long standing branch converting legacy mainnet relays to p2p, node -> 8.7.2, db-sync snapshots -> 13.1.1.3, and other improvements: cardano-ops-pull-417

Cardano-parts​

  • Adds a readme, provides misc improvements, service optimizations, alert tuning, sql pool performance analysis fix, package updates: cardano-parts-pull-27
  • Adds process-compose dbsync and node stacks: cardano-parts-pull-28

Cardano-playground​

Upstream Contributions​

  • Contributions to upstream process-compose related repos were made in order to complete the process-compose dbsync and node stacks in cardano-parts, including the following:

Process-compose-flake​

Services-flake​

· One min read
John Lotoski

High level summary​

During the lightly staffed holiday period for node SRE, the emphasis was on maintaining environment stability, tuning and resolving any noisey alerts.

Investigation into and testing around the following two topics also started during this period:

  • Ledger snapshots causing a small number of missed slots for forgers on mainnet: ouroboros-consensus-issue-868

  • A cardano-node rare file descriptor leak, with a more detailed description here

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • A new repository was created which enables agile deployment of EC2 monitoring servers, compatible with OpenTofu grafana and mimir providers: cardano-monitoring
  • The govtool backend swagger interface was nix flake packaged and deployed for Voltaire private chain testing usage
  • Grafana cloud monitoring stacks were migrated to new EC2 cardano-monitoring servers
  • Cardano-db-sync state snapshots now support client range requests, details here
  • In addition to monitoring server centralized grafana metrics, sysstat collected system metrics are now available locally on all cluster machines at high time resolution
  • Code changes required due to repository migrations to IntersectMBO have largely been completed

Lower level summary​

Auth-keys-hub​

Cardano-monitoring​

  • A new repository enabling agile deployment of EC2 monitoring servers, compatible with OpenTofu grafana and mimir providers: cardano-monitoring

Cardano-parts​

  • Migrate from grafana cloud monitoring to ec2 monitoring, add resource tagging support, declarative route53 CNAME list, and additional improvements and fixes: cardano-parts-pull-25
  • Improve ssh key handling and edge cases, resolve misc issues, add IOPS and throughput gp3 openTofu support: cardano-parts-pull-26

Cardano-playground​