56 posts tagged with "sre"

High level summary​

The SRE team continues work on cardano environment improvements and general environment maintenance.

Some notable recent improvements include:

• Re-spin sanchonet for testing a new node version followed by fork to Conway
• Deploying a sanchonet compatible faucet to sanchonet
• Migrate shelley qa legacy environment to cardano-world

Lower level summary​

Cardano-node​

• Bump iohk-nix for sanchonet and shelley qa environment updates: cardano-node-pull-5406

Cardano-ops​

• Mainnet relays are 50% migrated to p2p topology and 1 relay per region dedicated for ledger peers: cardano-ops-compare

Cardano-world​

• Cardano-node p2p all local roots entrypoint option was added: cardano-world-pull-102
• Migrate shelley qa legacy environment to cardano-world: cardano-world-pull-103
• Create a sanchonet compatible faucet: cardano-world-pull-91

Iohk-nix​

• Migrate shelley qa legacy environment to cardano-world: iohk-nix-pull-549
• Drop legacy byron environment parameters: iohk-nix-pull-550

High level summary​

The SRE team continues work on cardano environment improvements, supporting CI improvements, and general environment maintenance.

Some notable recent improvements include:

• Improved cardano-node entrypoint handling to allow for clean restarts in the Nomad environment
• Explorer component updates
• Spongix caching server rewrite for R2 Cloudflare support and various performance improvements

Lower level summary​

Cardano-node​

• Nixos service update to enable non-systemd socket activation multi-node relays: cardano-node-pull-5360

Cardano-ogmios​

• Cardano ogmios bump to 5.6.0 for nixos service: cardano-ogmios-pull-6

Cardano-ops​

• Improve storage resize script, workaround systemd node signal handling, rescale p2p relays: cardano-ops-compare

Cardano-rosetta​

• Cardano-rosetta nixos modifications for rosetta 2.1.0 service: cardano-rosetta-pull-532

Cardano-world​

• Cardano-node entrypoint improvements: cardano-world-pull-99
• Further automate pool performance SQL query: cardano-world-pull-100
• Performance and benchmark job related tweaks: cardano-world-commit
• Explorer related component version bumps and traefik definition handline: cardano-world-pull-101

Spongix​

• Spongix caching server rewrite for Cloudflare R2 backend support plus misc improvements: spongix-simple-branch

High level summary​

The SRE team continues work on CI and cardano environment improvements. Some notable recent improvements include:

• A devx-ci cluster containing a Hydra build server and Linux build farm was stood up and is intended to replace Cicero functionality
• Cardano Sanchonet environment was stood up to test Conway era functionality
• Cardano-node nixos service was updated to support the latest p2p topology format and non-systemd socket activation use case
• Cardano-node 8.1.1 was deployed to preview, preprod and mainnet environments

Lower level summary​

Cardano-node​

• Update cardano-node nixos service for updated p2p topology and non-systemd socket activation: cardano-node-pull-5318

Cardano-ops​

• Updates required for node 8.1.1 nixos service: cardano-ops-pull-414
• Work in progress continues on mixed legacy and p2p topology clusters in cardano-ops: cardano-ops-ops-local, cardano-ops-mn-mixed-p2p

Cardano-world​

• Add Sanchonet environment: cardano-world-pull-96

Ci-ops​

• Add devx-ci cluster integration for legacy darwin ci-ops-compare

Ci-world​

• Add devx-ci cluster integration for caching, monitoring during cicero to devx-ci transition: ci-world-compare

Devx-ci​

• A CI cluster with a hydra build server was stood up and is intended to replace usage of Cicero: devx-ci-repo

Iohk-nix​

• Add Sanchonet environment: iohk-nix-pull-545

High level summary​

The SRE team continues work on CI and cardano environment improvements. Some notable recent improvements include: expanding the darwin CI cluster and providing new aarch64 builder support; adding bare metal bitte cluster capability with network overlay for high IOPS workload performance, such as explorer.

Lower level summary​

Bitte​

• Equinix bare metal capability was added to bitte: bitte-pull-194
• Update bitte nixpkgs, nix version, nomad driver, equinix lifecycle, misc bug fixes: bitte-pull-201

Bitte-cells​

• Improvements made for patroni cluster recovery of large databases: bitte-cells-branch

Cardano-graphql​

• Update cardano-graphql and nixos service: cardano-graphql-pull-815

Cardano-node​

• Update cardano-node to use the updated iohk-nix environments: cardano-node-pull-5241

Cardano-ops​

• Update cardano-graphql, adds monitoring improvements and misc bug fixes: cardano-ops-pull-411
• Work in progress on mixed legacy and p2p topology clusters in cardano-ops: cardano-ops-branch

Cardano-world​

• Add a declarative cardano-faucet dashboard: cardano-world-commit
• Remove a requirement for root user in Nomad docker images: cardano-world-pull-68
• Enable explorer in cardano-world for high IOPS metal access: cardano-world-pull-69
• Migrate explorer cluster to cardano-world: cardano-world-pull-75
• Migrate testnet metadata server to cardano-world, cleanup envs: cardano-world-pull-77
• Add a faucet restore delegation script to cardano-world for non-functioning delegated pools: cardano-world-pull-79
• Integrate node 8.x into cardano-world: cardano-world-pull-84
• Update cardano-world to use iohk-nix for node environments: cardano-world-pull-87
• Create a stakepool analysis query for cluster pool performance overview: cardano-world-commit
• Added an iptables fixup script for broken bridged networking nomad jobs: cardano-world-commit
• Improve cardano-world job automation with additional parameterization: cardano-world-pr-92
• Update preview network experimental config defaults: cardano-world-pr-93

Ci-ops​

• Update legacy darwin builders and buildkite agent for ci-world network overlay and monitoring: ci-ops-pull-108

Ci-world​

• Resolve ci-world linux buildkite agent cleanup bug: ci-world-commit
• Integrate new darwin builders into ci-world: ci-world-pull-22
• Migrate patroni HA db to it's own namespace in ci-world: ci-world-pull-24

Cicero​

• Implement a cicero webhook backoff with exponential decay plus jitter: cicero-pull-79

Iohk-nix​

• Update iohk-nix to be source of truth for node environments: iohk-nix-pull-534
• Update preview network experimental config defaults: iohk-nix-pull-544

Openziti​

• Add darwin intel and arm support to ziti-edge-tunnel nix modules: openziti-bins-pull-1

High level summary​

The SRE team continues work on Cicero, Tullia, and Bitte, as well as providing support for cardano-world.

Lower level summary​

Cicero​

• Fixed various race conditions around transformers.
• Brought our CI up to date.
• Migrated to the Nomad exec driver with Nix support for many actions.
• Moved Nix builds to the Nomad clients for much better cache locality.
• Ongoing work on vastly improving the action matching and evaluation speed.

Tullia​

• Made it easier to support cloning from a PR's fork
• Update to latest std
• Add workaround for cgroup issue: nomad#12877
• github preset: add github.ci.remote and (read|get)Repository functions
• Fix various issues around CUE handling

Bitte​

• Upgrade to NixOS 22.11
• Prototype usage of Colmena for deploys instead of deploy-rs
• Finalized work on Equinix Metal support
• Prototype better secrets management with ragenix instead of sops-nix
• Improve CI and bring it up to date

cardano-world​

• Fixd various OOM issues on preview and preprod
• Rotated KES keys on preview and preprod
• Optimize mainnet db-sync to cope with higher load
• Fix an issue where PostgreSQL would fail after a reboot

bitte-world​

• Updated to NixOS 22.11

ci-world​

• Updated to NixOS 22.11
• Added Equnix cluster
• Improve caching of Nix builds