Skip to main content

· 3 min read
John Lotoski

High level summary​

The SRE team continues work on Cardano environment improvements and general maintenance.

Some notable recent changes, updates or improvements include:

  • With the exception of a few canary machines still running the legacy cardano-node tracing system, the majority of IOE playground and mainnet cardano-node machines are now running the new tracing system.

  • Adawallet received a feature upgrade to sign messages, enabling it to complete glacier drop claims for adawallet accounts. The cardano-airgap image was updated to include this capability.

Repository Work -- Merged​

Adawallet​

  • Bumps to node 10.5.1 for cardano-cli 10.11.0.0, adds cardano-signer v1.29.0 to the devShell, adds a yarn devShell, improves the cardano-hw-cli package by switching to default nodejs pkg and greatly simplifies the build. adawallet-pr-22

  • Add an adawallet sign-msg feature which signs with either payment or stake key, enabling glacier drop claims on adawallet accounts. Cleans up some more legacy cardano-hw-cli packaging and adds back bash auto-completion. adawallet-pr-23

Cardano-airgap​

  • Updates adawallet for sign-msg support for glacier drop and adds cardano-signer and misc support packages and services to the devShell and iso. cardano-airgap-pr-6

Cardano-mainnet​

  • This PR primarily upgrades all machines to the cardano-node new tracing system. It provides alert, dashboard and nix module upgrades for compatibility with the new tracing system. This PR includes improvements from cardano-parts release v2025-08-05. Additional details can be found in the PR description. cardano-mainnet-pr-38

Cardano-node​

  • Adds the snapshot-converter binary to the nix overlay and the node OCI container. Adds documentation on how to use the snapshot-converter within the image for changing ledger state type. cardano-node-pr-6299

Cardano-parts​

  • This cardano-parts release changes the default tracing system from legacy to the new cardano-node tracing system for deployed machines. See the release notes for details. cardano-parts-release-v2025-08-05

  • Updates cardano-signer to v1.29.0 which allows for Byron era address claims for Midnight Glacier drop. Bumps mithril unstable and adds some flakeModule cluster options for more service granularity. cardano-parts-release-v2025-08-14

Cardano-playground​

  • This PR primarily upgrades all playground testnet machines with a few canary exceptions to the cardano-node new tracing system. It provides alert, dashboard and nix module upgrades for compatibility with the new tracing system. This PR includes improvements from cardano-parts release v2025-08-05. Additional details can be found in the PR description. cardano-playground-pr-45

Cardano-signer (nix packaged)​

  • The nix packaging for upstream cardano-signer was updated for release 1.29.0 for byron address glacier drop compatibility and a GHA for ci build testing was added. cardano-signer-pr-2

Repository Work In Progress -- PRs and Branches​

· One min read
Alexey Kuleshevich

High level summary​

This period we've started work on CIP-112 - Observe script type or as we call it in Ledger "Required Guards". We've also started work on refactoring the stake pool state, in order to accommodate future features and a few performance optimizations. Besides that we've also put some effort into writing documentation and organizing some of our tests.

Low level summary​

Features​

  • PR-5160 - Provide a document that describes the Reward Calculation
  • PR-5217 - Conway/Dijkstra CDDL: OSet for certs and proposals
  • PR-5196 - Introduce StakePoolState to be used instead of PoolParams
  • PR-5215 - Add guards field to DijkstraTxBody

Testing​

  • PR-5183 - Conway:DELEG: Predfailures for deposits & refunds
  • PR-5166 - Reorganize Consensus Examples
  • PR-5216 - Fix the flaky nightly tests
  • PR-5218 - Ts fix neil nightly take2
  • PR-5187 - Separate conformance testing and constrained generators
  • PR-5223 - Run Imp tests for every version of an era

· 4 min read
Michael Karg

High level summary​

  • Benchmarking: Release benchmarks for 10.5; LMDB (UTxOs on-disk) benchmarks.
  • Development: New memory-constrained benchmark families, targeting UTxOs on-disk.
  • Infrastructure: Migration of bench cluster completed.
  • Tracing: Deprecation of legacy system; TCP forwarding merged; improved self-documentation.
  • Meetup: Performance & Tracing meetup held in Cardiff, Wales.

Low level overview​

Benchmarking​

We've performed and analysed release benchmarks for Node 10.5. The pre-release turned on peer sharing by default; our benchmarks indicated a negative performance impact when enabling this on block producers. The current release 10.5.1 does not enable peer sharing for block producers; the published final results can be found in the Performance report for 10.5.

Additionally, we've achieved meaningful benchmarks for UTxOs-on-disk, which use the LMDB backend. Our new profiles support seamless scaling of RAM pressure on the backend, forcing disk I/O to a varying degree. We're currently analysing the observations made, and gathering more data using different scaling factors if needed; the goal is a reliable assessmemt of LMDB's viability for block producing nodes.

Development​

Developing RAM-constrained benchmarks that would put tunable pressure on an UTxOs-on-disk backend posed a unique challenge.

First and foremost, limiting memory for past in-memory benchmarks has never been a requirement at all. A consistent approach to do so given the existing deployment had to be built, along with pertinent diagnostic tooling. Second, the LMDB backend is not managed by Haskell's GHC runtime, but comes with its own memory management - which required us to develop a double-pronged approach to selectively apply RAM limits. Lastly, other parts of the Node's code didn't support executing in tightly limited, constant space and would lead to the OS terminating the Node for running out of memory.

The interaction of various cgroup limits on Linux kernels, memory settings in our Nomad deployment and GHC RTS options let us create a stable runtime behavior over the course of the benchmark - a hard requirement, as system metrics taken at the beginning of a run must be comparable to those towards the end. A blocker for initializing the Node for benchmarks was resolved in cardano-node PR#6295: Using mmap allowed us to use the OS's virtual memory subsystem for on-demand loading instead of it being managed by the Haskell runtime - which significantly brought down the heap size required for the task.

Infrastructure​

The migration to the new benchmarking cluster is finalized. For representative performance measurements of UTxOs-on-disk, we require direct SSD storage on each cloud machine instance. Along with deploying the new hardware came a full upgrade of OS and nixpgks software. Validating the migration was extra effort: A seemingly innocent cloud provider service (which was newly added to their NixOS image) did nothing more than a heartbeat request every 5 min to some central infrastructure server. Yet, it caused the standard deviation of some of our network related metrics to double - thus reducing confidence in those metrics.

After validation, we performed a complete re-run of existing performance baselines on the new hardware.

Tracing​

Work on the new tracing system has yielded various improvements. Trace forwarding over TCP is now fully functional and merged to master. This will make setting up forwarding to remote hosts much easier than by using UNIX domain sockets / Windows named pipes. However, it's recommended for secure and isolated environments only (cardano-node PR#6241).

The auto-documentation feature of the new system has improved user experience; the generated documents are now structed in a more accessible way, and contain necessary metadata as to which Node version / commit is being documented by them (cardano-node PR#6283).

Another refactoring targeted the new system's core library, trace-dispatcher. It now comes with a minimal dependency footprint and fully equipped with all necessary type definitions to be built using Haskell's community packages (Hackage) exclusively. This greatly enhances the ease to use it for other applications than the Node as well - Cardano or non-Cardano (cardano-node PR#6268). Increasing the dependency footprint is only required for additional features, like trace forwarding.

With the upcoming Node 10.6 release, we plan to officially deprecate the legacy tracing system. This means it will enter a grace period of ~3 months, where both systems coexist in the same Node build; then, it will be decommissioned and actively removed from the code base.

Meetup​

We organized 2025's in-person team meetup in Cardiff, Wales. We had highly insightful and productive days - I would like to thank all team members who contributed, and extend my special thanks to guests from outside of the team: Your presence and contributions were greatly appreciated, Can Huzmeli and Neil Davies.

· 2 min read
Noon van der Silk

High-level summary​

We have been extremely busy these last few weeks supporting the glacier drop. We continue to focus on this, as well as starting to make some broader changes.

What did the team achieve?​

  • Released several patch versions; latest being 0.22.4

    • On this branch we have made many improvements:
    • Fix Hydra API not correctly interpreting events after event log rotation #2170
    • Ignore snapshot signatures of already confirmed snapshots; reduces load when running mirror nodes #2146
    • Fixed internal persistent queue blocking when it reached capacity #2147
    • Address etcd blocking on writes when under load #2171
    • Fixed log rotation id consistency #2173
    • Fix etcd lease keep-alive logic when the lease expired #2148
    • Fix InvalidTx causing stuck head #2134
    • Identified mirror nodes as a solution to node crashing and failing to restart #2117

  • On master

    • Added HTTP endpoint for submitting a transaction #2107
    • Allow partial Ada commit during incremental commits #2140
    • Hydra API server responds with content-type application/json #2094
    • Greetings message contains more information #2087
    • Fixed bug around fanning out with decommit/increment in progress
    • Hydra node rejects UTxO if it's below the limit #2106
    • Fixed log rotation id consistency #2173
    • Fix InvalidTx causing stuck head #2134

What's next?​

  • "Forward-port" the changes from 0.22.4 to master
  • Resolve the TUI being out of sync due to event-log rotation #2156
  • Start to investigate implementing partial fanout
  • Enable recovering a deposit when the head is closed #1812
  • Working on CI for Mac #2139
  • Working on a log viewer #2047
  • Present hydra doom at RareEvo
  • Continue to support the glacier drop

Thanks​

Big thanks to Juan and Jonathan for big contributions to these recent releases!

· One min read
Damian Nadales

High level summary​

During the past couple of weeks, the team has continued working across multiple workstreams: LSM-tree integration, Peras implementation, KES agent integration, and Leios prototyping. Current efforts include:

  • Implementing a streaming conversion for ledger snapshots.
  • Drafting a set of mini-protocols to cooperatively implement the Linear Leios specification. Particular focus is being placed on ensuring adequate concurrency and resource usage bounds, which are less explicitly defined in the high-level spec.
  • Releasing a stubbed version of kes-agent, and integrating it into the Consensus version targeted for Cardano Node 10.6. While the KES code will be included in this release, functionality will remain inactive pending a code audit.
  • Merging Peras-related optimizations into both the main and peras-staging branches.