Skip to main content

Performance & Tracing Update

· 4 min read
Michael Karg
Performance and Tracing Team Lead

High level summary​

  • Benchmarking: Node 11.1.0 release benchmarks.
  • Development: beacon benchmark: new metric capturing disk access time for UTxO lookup; replaced Cairo rendering with gnuplot to reduce bloat and increase portability.
  • Tracing: Alert manager feature for cardano-tracer now in implementation phase.
  • Leios: On-disk LedgerDB tx validation times benchmark delivered; distributable, self-contained benchmark package in the works.
  • Organizational: Decentralization of delivery: The entire Performance & Tracing team successfully moved from IOG to ICAN Group.

Low level overview​

Benchmarking​

The P&T team has performed and analyzed benchmarks, as well as pre-release bencharks, for the Node 11.1.0 release. While we did measure an increased resident set size, we also saw a clear reduction in process CPU usage. The memory increase has already prompted an investigation and changes, which will make their way into a 11.1.1 patch release.

Development​

beacon's comparison and variance reports now break totalTime outliers down into ledger-tick and table-read cost, and commit to a single totalTime metric instead of quietly double-counting table-read time into it — table-read time being what actually captures UTxO lookup cost against the on-disk backing store (beacon PR#16, second of three parts split off from #13, still under review).

We've also dropped beacon's Cairo-based plotting backend entirely in favour of a small gnuplot wrapper (beacon PR#21). The old Chart/Chart-cairo dependency pulled in a full X11 client stack for a tool that only ever renders to PNG files; the switch shrinks the binary's shared-library footprint from 25 down to 5, and let us finally add macOS to CI.

Tracing​

Following on from the alarm-system design work reported last time, we've moved into implementation (cardano-node PR#6664). The dispatch payload sent to alarm consumers now carries the source, rule ID, severity and a summary, instead of just the name of the consumer being notified; on the timeseries side, alarm rules are now backed by a small, pure, edge-triggered state machine — cycling through Inactive/Pending/Active states with configurable for/repeatEvery timing and a deterministic series key — while the evaluator thread and value decoder driving it are left for a follow-up PR.

Native support for nested spans in Hermod remains ongoing work, with no PR created yet.

Leios​

The on-disk LedgerDB tx validation benchmark work mentioned in our last two posts has been delivered: beacon run can now force the LSM backend to genuinely exercise disk I/O via a new --lsm-no-cache flag, bypassing the OS page cache through O_DIRECT, with --mem-limit/--heap-limit cgroup caps as a fallback for older db-analyser builds; every run also now reports peak RSS, block I/O counts, and a true wall-clock totalTime alongside the previous mutator-only timing (beacon PR#12). Concrete measurements from this on-disk backend are being published under Leios issue#553, the original ask behind this benchmarking effort.

In parallel, we're packaging this benchmark into a self-contained, distributable executable that SPOs can run on their own hardware without nix or network access (beacon PR#22, based on #16). glue bundles glibc and its own curl/unzip, patches its own ELF interpreter at extraction time, and drives measurement, hardware profiling and report assembly through auditable shell scripts rather than Haskell. It's currently under review.

Organizational​

As of this month, the entire Performance & Tracing team has moved from IOG to ICAN Group. The team has been warmly welcomed, and the move comes with a genuine sense of a fresh start — new structure, same people, same focus. Delivery on the Cardano Maintenance & Support proposal has continued without disruption throughout the transition; everything reported above was produced across the move.