Skip to main content

SRE Team Update

· 5 min read
John Lotoski
Service Reliability Engineer

High level summary

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • Trace dispatcher was migrated out of cardano-node and into its own repo: hermod-tracing

  • A number of CI improvements to Darwin builder configuration and general CI monitoring and alerting were merged.

  • The cardano-node 10.7.0 pre-release SRE contribution work was completed.

  • Dijkstra network had the van Rossem cost model for PV11 preparation submitted, ratified and enacted.

  • Some cloud resources were relocated to more stable areas after disruption due to conflict in the Middle East.

Repository Work -- Merged

Cardano-monitoring

cardano-monitoring PR#6:

  • Enables Loki alert rule evaluation, and makes the alerts visible in Grafana

Cardano-node

cardano-node PR#6478

  • Bumps iohkNix, updates MinNodeVersion to 10.7.0 and refreshes mainnet-peer-snapshot.json and other ci files.
  • Adds independent lsmDatabasePath NixOS option with uniqueness assertion and mutual-exclusion check against LMDB per instance.
  • Adds kes-agent/kes-agent-control (Linux only) and dmq-node (all platforms) to release binaries.
  • Adds cardano-node-dbtools NixOS test covering db-synthesizer, db-analyser, db-truncater, and the GHC-asserted synthesizer binary against a cardano-testnet create-env environment.
  • Adds --shelley-kes-agent-socket support to run-node and cardano-node-service.nix. Expands the KES assertion to cover three valid forging configurations: relay (none), direct KES key, and KES agent socket.
  • Adds CARDANO_TRACER_SOCKET_NETWORK_{ACCEPT,CONNECT} tracer socket options to run-node.
  • Hardens all node and tracer entrypoint/launch scripts with set -euo pipefail, safe ${VAR:-} expansion throughout, pre-flight file existence checks, and exec for clean process replacement.
  • Consolidates separate relay/block-producer run functions into a single runNode. Derives GENESIS_JSON from CARDANO_CONFIG directory to support non-mainnet deployments.
  • Renames runCommandNoCCLocalrunCommandLocal for nixpkgs 25.11.

Devx-ci

devx-ci PR#145:

  • Upgrades darwin CI infrastructure with version bumps, guest VM lifecycle management, and maintenance improvements.
  • Darwin related infrastructure upgrades:
    • Nix 2.322.33-maintenance (hosts and guests); nix.package now set explicitly
    • nix-darwin 25.0525.11 (guests)
    • UTM versioned per architecture:
      • aarch64-darwin: 4.5.45.0.2
      • x86_64-darwin: pinned to 4.6.5 as UTM > 4.6.5 breaks display driver compatibility with macOS Sequoia+ on Intel
    • ca-derivations experimental feature enabled on hosts and guests
    • Guest bootstrap nix version bumped 2.28.32.32.5
    • Adds a small C binary at /usr/local/bin/nix-daemon-launcher to work around macOS launchd sandbox blocking .dylib loads from the APFS /nix volume
    • darwin.sh gains --system/-s (default: aarch64-darwin) flags; argbash upgraded 2.10.02.11.0
    • Auth-keys-hub is now utilized by the guests and legacy ops-lib usage has been removed
    • A percentage-based threshold garbage collection has been implemented, deriving thresholds from disk size rather than fixed values
  • Bumped nixpkgs-gh-runners v2.330.0v2.332.0
  • See the PR description for additional details

devx-ci PR#146:

  • Deploy hydra-tools hydra-github-bridge 0.2.1.0

devx-ci PR#147:

  • Accommodate the nixos hydra-github-bridge module with an extra secrets file

devx-ci PR#148:

  • Pause the new hydra-github-bridge usage until after pre-release of 10.7.0
  • Remove zramSwap to make more physical RAM available to Hydra
  • Reduce max concurrent evals/jobs down to 4 to keep throughput manageable

devx-ci PR#149:

  • Rekey all secrets to accommodate contributions from another SRE

devx-ci PR#150:

  • Improves alerting infrastructure using Mimir Alertmanager and Loki ruler.
  • Sets up Dead Man's Snitch.
  • Infrastructure changes:
    • Moved OpenTofu configuration from perSystem/packages/opentofuConfig/ to flake/opentofu/ for consistency with other SRE repos
    • Added Mimir provider for Prometheus-style alerting rules
    • Added Loki provider for log-based alerting rules
    • Configured Mimir Alertmanager to route all alerts to PagerDuty
    • Added various metrics based and log based alerts
  • See the PR description for additional details

devx-ci PR#151:

  • Alert on every OOM and setup annotations to show Alertmanager alerts.

Hermod-tracing

hermod-tracing PR#2

  • Adds hydraJobs to flake top level attrs
  • Adds aarch64-linux
  • Tests hydra integration
  • Makes explicit required and nonrequired aggregate jobs
  • Moves default pkgs to trace-dispatcher

Iohk-nix

iohk-nix PR#610:

  • Include updated configs from respun dijkstra net from 2026-02-19
  • Re-add sanchonet back to the available environments since it is persisting as a long lived community test network
  • Update per-environment useLedgerAfterSlot values
  • Update per-environment peer-snapshot.json files
  • Update MinNodeVersion to 10.7.0 as the peer-snapshot files made a version breaking change

Usdcx-infra

usdcx-infra PR#5:

  • Adds missing series eval resolution targets

Repository Work In Progress -- PRs and Branches

Hydra Team Update

· One min read
Noon van der Silk
Software Engineering Lead

What did the team achieve?

  • Continued progress on partial fanout #1667, #2538
  • Final touches on the "Directly Open Heads" feature #1329
  • Investigating a blocker when upgrading to cardano-api 10.21 #2520
  • Working on more resilient snapshots #2551

What's next?

  • Merge approach for resilient snapshots #2551
  • Merge "Directly Open Heads" #1329
  • Get more structure in our benchmarks #2439
  • Release 2.0.0!

Consensus Team Update

· One min read
Damian Nadales
Consensus Team Lead

High level summary

  • Peras protocol development (Treasury Funding Initiative 17: Maintenance and Support):
    • Reviewed and merged the implementation of a model of the committee selection scheme, which defines how voters are chosen for Peras's fast-finality voting rounds (#1839).
    • Reviewed and merged the implementation of the node tracking Peras certificate progress on-chain, needed for the protocol's chain-selection rule (#1864).
  • Node improvements (Treasury Funding Initiative 17: Maintenance and Support):
    • Released ouroboros-consensus-1.0.0.0 (#1926).
    • Reworked internal resource management for ledger state access, improving node robustness (#1910).
    • Fixed a bug in the immutable DB's chunk enumeration logic which only affected test coverage (#1923).
    • Integrated the Networking team's ouroboros-network 1.0 release (#1918, #1927, #1929).

Mithril Team Update

· 4 min read
Jean-Philippe Raynaud
Mithril Tech Lead

High level overview

This week, the Mithril team completed several SNARK-related milestones: they finished implementing the Halo2 circuit in the STM library, completed work on SNARK aggregation primitives for creating and verifying SNARK proofs, and finalized circuit refactoring, including modularity enhancements and a switch to a transcript hash function. They also completed the technical report for the recursive Halo2 circuit. The team continued work on the full review of the recursive SNARK circuit prototype, the impact assessment of SNARK on Mithril protocol security, wiring the SNARK proof into the aggregate signature, and activating the SNARK prover in the dev networks.

They also completed implementing the new prover for Cardano blocks and transactions, the client library, and enhanced the signed entity type configuration. They continued progressing on the client CLI implementation, partial block range support, and usage examples for Cardano blocks and transactions. Additionally, they kept working on the release process improvements, the removal of the legacy Cardano database backend in the client, and the enhancement of the protocol security page on the website.

Finally, they completed static builds of Mithril nodes in CI, finalized testing of the DMQ node 0.3.0 pre-release, and updated the Midnight ZK library audit status.

Low level overview

Features

  • Completed the issue Implement Halo2 circuit in STM library #2895
  • Completed the issue Refactor SNARK circuit - Modularity enhancement with gadgets #3039
  • Completed the issue SNARK aggregation primitives: Create SNARK proof with circuit #3040
  • Completed the issue SNARK aggregation primitives: Verify SNARK proof #3041
  • Completed the issue Switch transcript hash function of circuit #3067
  • Completed the issue Prepare technical report for recursive Halo2 circuit #2981
  • Completed the issue Implement new prover for Cardano Blocks and Transactions #2987
  • Completed the issue Implement Cardano Blocks and Transactions in client library #3031
  • Completed the issue Enhance the support for signed entity types with configuration #3030
  • Worked on the issue Full review of recursive SNARK circuit prototype #2982
  • Worked on the issue Impact of SNARK on Mithril protocol security #2803
  • Worked on the issue SNARK aggregation primitives: Wire SNARK proof in aggregate signature #3042
  • Worked on the issue Activate SNARK prover in dev network #3104
  • Worked on the issue Implement Cardano Blocks and Transactions in client CLI #3032
  • Worked on the issue Support partial block range in Cardano blocks and transactions #3099
  • Worked on the issue Implement examples for Cardano Blocks and Transactions #3100

Protocol maintenance

  • Completed the issue Implement static build of Mithril nodes in CI #2989
  • Completed the issue Test DMQ node 0.3.0 pre-release in Mithril nodes #3053
  • Completed the issue Update Midnight ZK library audit status #2983
  • Worked on the issue Remove v1 backend for Cardano database in client library and CLI #3080
  • Worked on the issue Update release process to anticipate on unreleased Cardano node #3070
  • Worked on the issue Enhance protocol security page on website #2703

Network Team Update

· 4 min read
Marcin Szamotulski
Network Team Lead

Overview of sprints: 108 and 109.

Cardano Node 10.7.0

We relased

  • ouroboros-network-1.0.0.0 (later refined with ouroboros-network-1.1.0.0
  • cardano-diffusion-1.1.0.0 which are integrated with the ongoing cardano-node-10.7 release.

See release board for all PRs included in these releases.

Leios

PR / IssueStatus
Mux egress prioritisationin progress
Performance improvement for tx-submission v2in progress
TxSubmissionV2 demoin progress
Expand network sectionin review

Ouroboros-Network

PR / IssueStatusNotes
tx-submission robustness and testing enhancementsmerged
RNG handling in SRV lookups for public root peersclosedcontribution by Saviour Uzoukwu
cardano ping implemented with ouroboros-networkblocked
mux: refactored ReadBuffer & socket API changesmerged
ledger-peers: introduced SignLedgerPeersKindmerged
cardano-ping: output valid JSON when no pongs are receivedmerged
Add {To,From}JSON instances for TxSubmissionLogicVersionmerged
[Cardano Topology: module exports][pr#5339]merged
Haddock improvementsmerged

DMQ-Node

PR / IssueStatusnotes
SigSubmission v2merged
Integration with cardano-tracerin progress
[Churn test][#5303]merged
Updated dependencies of dmq-nodemerged
Improved documentationmerged
Fixed number of unacknowledged sigIds on the outbound sidemerged
[Added pattern synonyms for EraIndex (CardanoEras c)][pr#1919]mergedouroboros-consensus change needed for Ledger peers
Ledger peersmerged
[dmq-node-0.3.0.0 pre-release][pr#37]merged
Haddocks, docs & release scriptsmerged

IOSim

PR / IssueStatus
Added flushEventLog to MonadEventlogmerged
io-sim improvementsmerged
PR review commentsmerged
ghc-9.14 supportmerged
io-sim: io-classes constraintmerged