Skip to main content

42 posts tagged with "sre"

View All Tags

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node 8.8.1-pre is now deployed to sanchonet, two-thirds of preprod, one-third of preview and also voltaire private chain networks.

  • Catalyst Voltaire private chain was respun and an updated db-sync and govtool backend deployed after hard fork to Conway era.

  • IOG sanchonet pool block producers are now also signing mithril certificates.

  • A mithril-signer-verifier service now runs on all IOG SRE mithril signers and generates alerts if no recent mithril certificates have been signed.

  • All cardano-parts mithril clients, whether via nixosConfiguration, process-compose jobs or the nix cardano-node entrypoint, will now automatically verify that a trusted IOG pool has signed a mithril certificate prior to using the associated mithril state. If no trusted IOG pool has signed a recent mithril certificate, mithril state usage with be skipped and a classic sync from genesis to tip will be performed.

  • A significant amount of effort was applied towards the upcoming cardano-node 8.9.x-pre release in order to align nixos service module code between cardano-node nixos services, iohk-nix topology generation, cardano-ops legacy code, and cardano-parts module compatibility.

Lower level summary​

Work in progress​

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Sanchonet state was respun from slot 14255995 for cardano-node 8.8.0-pre
  • Cardano node 8.8.0-pre is now deployed to sanchonet, shelley-qa, two-thirds of preprod and one-third of preview networks
  • IOG pool block producers are now signing mithril certificates for preview, preprod and mainnet networks
  • A cardano performance dashboard is now available for both playground and cardano-mainnet deployments

Lower level summary​

Cardano-mainnet​

  • Adds mithril signers and perf dashboard to the mainnet cluster; improves network docs, tunes alerts, plus misc fixes: cardano-mainnet-pull-8

Cardano-parts​

  • Adds mithril signer capability to role-block-producer nixosModule, adds a profile-mithril-relay nixosModule and includes misc other improvements and fixes: cardano-parts-pull-31

  • Updates packages for sanchonet respin, adds a node performance dashboard template and fixes misc issues: cardano-parts-pull-32

Cardano-parts​

  • Adds mithril signers and mithril relays to preview and preprod networks and includes misc other improvements and fixes: cardano-playground-pull-15

  • Updates machines and pin for sanchonet respin, adds a node performance dashboard and fixes misc issues: cardano-playground-pull-16

Iohk-nix​

· One min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano explorer is being sunset and will soon be replaced by a new Cardano Foundation explorer
  • IOG mainnet pools are now participating in propagation time reporting using Cardano Foundation's blockperf tool
  • Cardano-parts node nixosConfigurations, node entrypoint and process-compose node jobs now utilize mithril client automatically on preview and preprod networks for faster synchronization of first start up. Mainnet mithril client use as well as signer modules will be available soon.

Lower level summary​

Cardano-mainnet​

Cardano-parts​

  • Adds a block performance module which utilizes cardano-foundation's blockperf for aggregate block propagation reporting: cardano-parts-pull-29
  • Adds a mithril client for nixosConfigurations, process-compose cardano-node processes and cardano-node entrypoint: cardano-parts-pull-30

Iohk-nix​

  • Adds required mithril client environment information for preview, preprod, mainnet: iohk-nix-pull-570

Cardano-world​

Upstream Contributions​

Blockperf​

· 2 min read
John Lotoski

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent changes, updates or improvements include:

  • Cardano-node 8.7.3 is now generally deployed to all testnet and mainnet environments managed by coretech SRE

  • Dbsync and node stack process-compose jobs are now available from cardano-parts for users running nix >= 2.17.0 and nix experimental-features = nix-command flakes fetch-closure

  • These can be run with:

     nix run github:input-output-hk/cardano-parts#run-process-compose-node-stack
     nix run github:input-output-hk/cardano-parts#run-process-compose-dbsync-$NETWORK

  • For more details, see the merged cardano-parts process-compose PR

Lower level summary​

Cardano-monitoring​

Cardano-mainnet​

  • Adds a readme, switches to nonmoving gc for producers, plus misc improvements from cardano-parts: cardano-mainnet-pull-6

Cardano-ops​

  • Merged a long standing branch converting legacy mainnet relays to p2p, node -> 8.7.2, db-sync snapshots -> 13.1.1.3, and other improvements: cardano-ops-pull-417

Cardano-parts​

  • Adds a readme, provides misc improvements, service optimizations, alert tuning, sql pool performance analysis fix, package updates: cardano-parts-pull-27
  • Adds process-compose dbsync and node stacks: cardano-parts-pull-28

Cardano-playground​

Upstream Contributions​

  • Contributions to upstream process-compose related repos were made in order to complete the process-compose dbsync and node stacks in cardano-parts, including the following:

Process-compose-flake​

Services-flake​

· One min read
John Lotoski

High level summary​

During the lightly staffed holiday period for node SRE, the emphasis was on maintaining environment stability, tuning and resolving any noisey alerts.

Investigation into and testing around the following two topics also started during this period:

  • Ledger snapshots causing a small number of missed slots for forgers on mainnet: ouroboros-consensus-issue-868

  • A cardano-node rare file descriptor leak, with a more detailed description here