66 posts tagged with "performance-tracing"

High level summary​

• Benchmarking: Preliminary benchmarks on the Leios prototype and the Leios Week42 build; refinement of performance analysis.
• Development: In preparation for Node 11.2, we focused on fixing issues and addressing reports from the community.
• Tracing: Switch to Hermod Tracing - preparation phase concluded; Optimization pass for trace filtering; Alert manager and Native spans ongoing.
• Infrastructure: New Dijkstra-scoped benchmarks such as nested transactions and reference script measurements.
• Leios: Tx validation benchmark on SPO hardware analyzed.

Low level overview​

Benchmarking​

The Performance & Tracing team has done several cluster benchmarks on the Leios prototype and the subsequent Leios Week42 build. There were several focus areas we watched out for, such as the ratio of certified EBs (Endorser Blocks). With voting traffic and timing constraints, and physical bandwidth being known limiting factors, we wanted to measure both high- and low-confidence parameter sets for the protocol in that regard. Additionally, we compared RBs' (Ranking Blocks') performance properties to a Praos-only baseline - how do block production, diffusion and adoption metrics fare? Then, we quantified mempool fragmentation and potential improvements due to the TxCache. Last but not least, as usual, our eye was on resource usage of a fully integrated build.

Our surgical windows into Leios performance are still being adjusted, based on the observations we make and asks from the Leios team; we're continuously refining coverage of our performance reports for Leios. All of the findings have been shared with the Leios team; however, they're not a public item as they do not describe an official release candidate and already contain known bottlenecks that are actively being worked on.

Development​

In preparation for Node 11.2, the team worked through a handful of community-reported bugs. A counter tracking slots missed while block forging was intentionally disabled used to jump by the whole disabled interval the moment forging resumed, booking time that was never actually missed; this is now fixed (cardano-node PR#6726). On Windows - and on Linux under a non-UTF-8 locale - the node could crash outright on a non-ASCII host or node name, or emit trace lines that weren't valid UTF-8; both cardano-node and cardano-tracer now force UTF-8 console encoding on startup regardless of the host's locale settings (cardano-node PR#6731). On the Hermod Tracing side, the Prometheus metrics endpoint now exposes its label sets with full conformance to the Prometheus text format, fixing a parsing issue some scrapers ran into (hermod-tracing PR#20).

Tracing​

As reported in earlier posts, the plan is to replace the node's current tracing system (trace-dispatcher) with Hermod Tracing once it's ready, bringing a cleaner API/core split, live reconfiguration of trace filter rules, and a single source of truth for defining tracers used across emission, documentation and consistency checks. Hermod Tracing's packages have now reached version 1.1.0, completing the API rework needed so the node's own dependencies - ouroboros-network, cardano-diffusion and ouroboros-consensus - can build against Hermod's lightweight public interface alone, rather than pulling in the full implementation package (hermod-tracing PR#21). With this now tested together with the existing node codebase, the preparation phase for the tracing system switch is concluded.

Separately, the team has been making the cost of a filtered-out trace message as cheap as possible, since most emitted traces are dropped by a severity or namespace filter before they're ever written anywhere. A series of changes to Hermod's internal configuration and message-combination logic - avoiding repeated allocations, building configured tracers once instead of per message, and specialising hot code paths - cut the cost of discarding such a message by roughly 7x, with a similar drop in memory allocated per message (hermod-tracing feature branch). The work isn't in a PR yet, pending full validation.

The monitoring/alarm system (cardano-node PR#6664) as well as library support of tracing spans (hermod-tracing PR#19) is ongoing work, with the latter being finalized and under review currently.

Infrastructure​

In preparation for the Node 11.2 release, the team extended our performance workbench (the benchmarking automation framework) with new Dijkstra-scoped profiles: nested transactions - transactions that themselves carry sub-transactions - and reference-script-based Plutus calls, benchmarked alongside the existing witness-script path (cardano-node feature branch). Getting representative results for the nested-transaction profiles also required lowering how long a node waits before requesting transactions from a newly connected peer, from the default 60 seconds down to 1, so the new profiles' load reaches the whole cluster fast enough to produce meaningful numbers.

Leios​

Validating that Leios's ledger-side assumptions hold up on a real implementation is an effort we've reported on over several past posts: can the ledger keep up with the transaction throughput Leios is designed for, and - since that depends heavily on disk I/O once the UTxO set no longer fits in memory - how much does storage hardware matter (Leios issue#553)? Last month we turned the resulting transaction-validation benchmark into a standalone executable and shipped it to SPOs so it could be run directly on their own hardware, without a nix or Haskell/cabal stack.

We've now collected and analysed submissions from a broad sample of real SPO machines. Raw read latency per transaction varied by about 32x across the field; once that's adjusted for each machine's raw CPU speed (which accounts for much of the spread on its own), the gap narrows to roughly 7-8x, driven mainly by one factor: whether a disk's write cache is write-back or write-through. Write-through disks were close to 6x slower than write-back ones before adjusting for CPU, and still more than 3x slower after - unless they sit behind a battery-backed RAID controller, which closes most of the gap. The full breakdown is available for anyone who wants to dig deeper (Benchmark results).

High level summary​

• Benchmarking: Node 11.1.1 release benchmarks; LSM-trees (on-disk LedgerDB) benchmarks.
• Development: Benchmarking workload submission full Dijkstra support - as well as on-disk serialization for db-synthesizer.
• Tracing: Full switch to Hermod Tracing in planning; Alert manager for cardano-tracer is ongoing work; Native span tracing for Hermod.
• Infrastructure: cardano-smartbench: A detached performance workbench - feasibility.
• Leios: On-disk LedgerDB tx validation benchmark shipped to SPOs. First Leios tests on the performance cluster, including tooling updates.
• Organizational: Performance & Tracing Team Meetup in Helsinki.

Low level overview​

Benchmarking​

The P&T team has performed, analyzed and published benchmarks for the Node 11.1.1 release. While the CPU usage improvement persists, and the memory increase is smaller than with 11.1.0, it's still measurable. It was mentioned as a known issue for this release. It seems to correlate with cyclic events such as ledger snapshotting, and while it has been observed on Mainnet as well, it did not represent an operational risk to the network.

Furthermore, we re-ran the 11.0 baseline, as network latencies between cluster machines had changed since May 2026 and were confounding network-related performance metrics. We additionally ran benchmarks with a new Ledger feature that loads huge datasets injected into ledger state in a streaming fashion, and we could validate it works as intended. This, however, is only relevant for testnets; the feature is blocked for Mainnet, as no data may be injected at all.

Last but not least, for Node 11.1 we repeated benchmarks of on-disk LedgerDB stressing the UTxO set - both with and without artificial memory constraints on the process. Those benchmarks are currently under analysis.

Development​

tx-generator, the workload-submission tool behind our benchmarking profiles, gained two new pieces of functionality (cardano-node PR#6695). It can now dump its generated transaction stream to disk - as plain text or as raw CBOR - instead of always submitting it live against a running cluster, so other tools can inspect or validate a workload without needing a full cluster to generate against. At the same time, its era-specific code paths were extended to cover the Dijkstra era, so we can generate and submit Dijkstra-era transactions for benchmarking purposes.

The longer-term goal of writing an entire transaction workload to disk is having db-synthesizer as a consumer. It enables creation of fully valid Praos or Leios chain fragments in a short time - and those chain fragments will exhibit all the properties defined in our benchmarking profiles.

Tracing​

In one of the upcoming node versions after the Dijkstra hard fork, we plan on replacing the new tracing system (trace-dispatcher) with Hermod Tracing. While Hermod currently is at feature and API parity with trace-dispatcher, it makes a sensible split between API and core packages, allowing for improved development in the future. But more importantly, it will enable some new tracing system features which have been deferred up to now, such as live reconfiguration of trace filter rules, and single-source definition of tracers for all purposes - consistency checks, auto-documentation and of course trace emission. We're currently integrating Hermod into the Haskell node component stack and preparing for a smooth transition when the time comes.

The implementation of the monitoring/alarm system is progressing (cardano-node PR#6664). Currently we're working on a robust test harness; a feature like this requires extremely rigorous coverage. Additionally, we're working on interfacing the system with escalation channels, such as pagers, pager app APIs, email and so forth.

Native support for nested spans in Hermod, which we've mentioned as exploratory work in earlier posts, has taken its first concrete shape: a new module implements Loki-style begin/end span tracing, pairing an opening and a closing trace message under a shared span identifier (hermod-tracing PR#19). The PR is still fresh and under review.

Infrastructure​

We've begun a feasibility study for detaching our performance workbench (the benchmarking automation framework) from the main Haskell node project - working title cardano-smartbench. It would let the workbench pick custom inputs, tooling and analysis targets rather than being tied to the Haskell node project's own layout. This addresses issues for benchmarking branches that have diverged a long distance from the project master (such as Leios), and would also generalize the workbench to work with clients other than the Haskell node in the future. The current challenge is keeping the existing CI guarantees alive on the Haskell node's master branch, without risking all downstream tooling silently going out of sync. As it's a feasibility study, there's no PR for it (yet).

Leios​

The transaction validation time benchmark has been turned into a self-contained, distributable executable and shipped to SPOs. It can run on their own hardware without any nix or Haskell/cabal stack. We're specifically interested in the on-disk measurements for LedgerDB on a wide variety of configurations - as being I/O bound (vs. CPU and RAM only) implies being dependent on a much larger array of parameters, such as kernel version, SSD hardware and connection, file system choice and even mount options.

We're currently testing our benchmarking automation on the performance cluster with the Leios prototype. While the numbers obtained from that won't be representative, we're preparing for the Leios Release Candidate: our infrastructure needs to support everything Leios adds - from config values and protocol parameters to understanding Leios-specific timestamped events for performance analysis.

Organizational​

This month, the Performance & Tracing team had a three-day in-person meetup in Helsinki. We had excellent discussions on focus shifts that come with our transition to ICAN Group. Additionally, we ideated means of making the team's delivery more accessible to the wider Cardano community, especially non-technical people - as our direct audience is no longer only IOG engineers. Thank you to everyone who contributed their views and ideas!

High level summary​

• Benchmarking: Node 11.1.0 release benchmarks.
• Development: beacon benchmark: new metric capturing disk access time for UTxO lookup; replaced Cairo rendering with gnuplot to reduce bloat and increase portability.
• Tracing: Alert manager feature for cardano-tracer now in implementation phase.
• Leios: On-disk LedgerDB tx validation times benchmark delivered; distributable, self-contained benchmark package in the works.
• Organizational: Decentralization of delivery: The entire Performance & Tracing team successfully moved from IOG to ICAN Group.

Low level overview​

Benchmarking​

The P&T team has performed and analyzed benchmarks, as well as pre-release bencharks, for the Node 11.1.0 release. While we did measure an increased resident set size, we also saw a clear reduction in process CPU usage. The memory increase has already prompted an investigation and changes, which will make their way into a 11.1.1 patch release.

Development​

beacon's comparison and variance reports now break totalTime outliers down into ledger-tick and table-read cost, and commit to a single totalTime metric instead of quietly double-counting table-read time into it — table-read time being what actually captures UTxO lookup cost against the on-disk backing store (beacon PR#16, second of three parts split off from #13, still under review).

We've also dropped beacon's Cairo-based plotting backend entirely in favour of a small gnuplot wrapper (beacon PR#21). The old Chart/Chart-cairo dependency pulled in a full X11 client stack for a tool that only ever renders to PNG files; the switch shrinks the binary's shared-library footprint from 25 down to 5, and let us finally add macOS to CI.

Tracing​

Following on from the alarm-system design work reported last time, we've moved into implementation (cardano-node PR#6664). The dispatch payload sent to alarm consumers now carries the source, rule ID, severity and a summary, instead of just the name of the consumer being notified; on the timeseries side, alarm rules are now backed by a small, pure, edge-triggered state machine — cycling through Inactive/Pending/Active states with configurable for/repeatEvery timing and a deterministic series key — while the evaluator thread and value decoder driving it are left for a follow-up PR.

Native support for nested spans in Hermod remains ongoing work, with no PR created yet.

Leios​

The on-disk LedgerDB tx validation benchmark work mentioned in our last two posts has been delivered: beacon run can now force the LSM backend to genuinely exercise disk I/O via a new --lsm-no-cache flag, bypassing the OS page cache through O_DIRECT, with --mem-limit/--heap-limit cgroup caps as a fallback for older db-analyser builds; every run also now reports peak RSS, block I/O counts, and a true wall-clock totalTime alongside the previous mutator-only timing (beacon PR#12). Concrete measurements from this on-disk backend are being published under Leios issue#553, the original ask behind this benchmarking effort.

In parallel, we're packaging this benchmark into a self-contained, distributable executable that SPOs can run on their own hardware without nix or network access (beacon PR#22, based on #16). glue bundles glibc and its own curl/unzip, patches its own ELF interpreter at extraction time, and drives measurement, hardware profiling and report assembly through auditable shell scripts rather than Haskell. It's currently under review.

Organizational​

As of this month, the entire Performance & Tracing team has moved from IOG to ICAN Group. The team has been warmly welcomed, and the move comes with a genuine sense of a fresh start — new structure, same people, same focus. Delivery on the Cardano Maintenance & Support proposal has continued without disruption throughout the transition; everything reported above was produced across the move.

High level summary​

• Benchmarking: Van Rossem HF: Created and performed benchmarks for new Plutus execution budgets and internals; PlutusV3 baseline shift.
• Development: Hermod package restructre completed; cardano-config support in trace-dispatcher.
• Infrastructure: Genesis cache overhaul / Protocol Version 11 overlay have landed.
• Tracing: Alert manager feature for cardano-tracer in design phase; native support for spans in Hermod.
• Leios: Major new beacon feature to impose memory limits; on-disk LedgerDB tx validation times benchmark in the works.

Low level overview​

Benchmarking​

The P&T team has been safeguarding the now enacted Van Rossem hard fork, which brings Protocol Version 11 with new Plutus internals and a bump to the execution budgets to mainnet. Our benchmarking automation framework, the performance workbench, gained new chain-fragment-creation profiles as well as new plutuscall-* profiles, targeting those new Plutus features specifically (cardano-node PR#6627 and cardano-node PR#6440). The former PR also adds a wb chain-fragment export op, which packages a profile run's node database, genesis files and a path-patched node config into a chain fragment consumable by beacon tool — the tooling behind the on-disk benchmarks described under Leios below.

The benchmarks performed using those new profiles have shown that the execution budget raise leads to a reasonable, predictable and expected increase in Block Adoption times. For the new PlutusV3 internals, they've also shown that they're consistently priced relative to the existing internals, and that, even when executed up to full budget exhaustion, there were no performance regressions in their implementation.

To enable those benchmarks, work has been well underway on extending plutus-scripts-bench with PlutusV3 variants of the existing scripts, and adding two new PV11 scripts targeting the new execution internals: MultiScalarMulG1 (BLS12-381 multi-scalar multiplication based on CIP-0133), and ExpModInteger, exercising the new expModInteger builtin (cardano-node PR#6440). The PR wires these through to tx-generator, and gives our analysis tool locli a new PlutusBudgetSummary module to output Plutus workload calibration data as part of comparison reports. The PR is currently under review.

Development​

The long-planned restructuring of trace-dispatcher into separate Hermod packages, which we reported as underway last month, is now complete (hermod-tracing PR#16). The stable API surface — trace types, LogFormatting/MetaTrace instances, and the structural/annotation combinators — now lives in a lightweight hermod-tracing-api package, itself split into an internal and a public sublibrary, so that packages which only need to instrument their own code no longer pull in EKG, Prometheus, CBOR forwarding or the YAML config parser as dependencies. The full backend stack remains in hermod-tracing-core, while the Prometheus TCP server has been carved out into its own optional hermod-tracing-prometheus package. All modules and identifiers were renamed from Cardano.Logging.*/TraceDispatcherMessage and friends to their Hermod.Tracing.* equivalents, and trace-dispatcher itself has been kept around as a backward-compatible mirror so existing pinned consumers keep building unchanged until they migrate.

Last but not least, we've added compatibility with the cardano-config library to trace-dispatcher (hermod-tracing PR#18).

Infrastructure​

The genesis cache overhaul and full Protocol Version 11 overlay for our performance workbench, close to merging as of last month, have now landed (cardano-node PR#6544). Beyond the split-cache "ripper" backend and the restoration of non-default Plutus cost models via the Alonzo genesis extraConfig field reported previously, the merged PR also standardises genesis creation on cardano-cli's create-testnet-data (retiring the old create-staked path and the separate byron genesis step), makes the era an explicit, validated input to wb start and wb run allocate via a new --era-name flag, and simplifies the workbench CLI to named flags only. A handful of breaking changes ship alongside: WB_BACKEND is renamed to WB_BACKEND_NAME, --relays and the genesis "spec" sub-command are removed, and non-default Plutus cost models must now be supplied through extraConfig. Analysis of older runs is unaffected, as wb analyse falls back to the previous genesis layout when the new one is absent.

Tracing​

We've begun designing a central alarm system for cardano-tracer (alarm system concept). Today, operators typically rely on third-party tooling for this — something built on top of Grafana, the Prometheus Alertmanager, or Loki log analysis — but there's no solution native to the tracing and metrics stack, and none that takes a unified approach across trace messages and metrics. The idea is to instead give cardano-tracer a single place that collects alarms from multiple sources — our hermod-recon verifier flagging a violated property, a rule evaluated against the metrics cardano-tracer already stores, or simply a trace message severe enough to warrant attention — and forward them to wherever operators want to be notified — typically a paging service's webhook API, but custom webhooks, email, and plain logging are supported too — while keeping a queryable history of past alarms. It deliberately stays a one-shot notification layer rather than a full incident tracker: there's no acknowledgement or resolution workflow, that's left to whatever tooling operators already use for that. This is still an early design and not yet reflected in any implementation work.

On the Hermod side, we're also exploring native support for nested spans, in the spirit of tracing systems like OpenTelemetry. A span represents a bounded unit of work — say, processing a single transaction or forging a block — and the point is to let automated log analysis reconstruct not just individual events, but how long an operation took and how it decomposes into sub-operations, by pairing up an opening and a closing trace message that share a unique identifier. The tricky part is Haskell's laziness: without care, an expression's evaluation, and the trace events it triggers, can end up happening after its enclosing span has already been closed, so a fair amount of the design work goes into guaranteeing that no trace event ever escapes the span it logically belongs to. There's no branch to point to yet — this is still exploratory work.

Leios​

Building on the chain-fragment export tooling introduced above, beacon — the ledger ops benchmarking tool in ouroboros-consensus-tools — gained a memory-limiting feature aimed squarely at tx validation times using an on-disk LedgerDB backend and forcing disk I/O at the same time (beacon PR#12, currently under review).

A new flag forces the LSM (on-disk) backend to genuinely exercise disk I/O, rather than being served invisibly out of the OS page cache; --mem-limit/--heap-limit cgroup caps serve as a fallback for past db-analyser builds without support for that new flag. Every run now records peak RSS and block I/O counts. A representative "sweet spot" using those new mechanisms has been determined empirically for the chain fragments used in this year's April measurements. This finally gives us concrete on-disk LedgerDB metrics by determining the overhead of I/O-backed validation times vs. pure ones, closing a gap in our Leios throughput benchmarking. A new beacon/docs/METHODOLOGY.md documents the overall approach; benchmarks are currently being performed on our benchmarking cluster, to obtain measurements on the same hardware as we did in April for the in-memory backend — but now with LSM backend forced to perform disk I/O.

High level summary​

• Benchmarking: -O2 build optimization benchmarks.
• Development: Legacy iohk-monitoring tracing backend removed from cardano-node; RTView removed from cardano-tracer; trace-dispatcher maintenance and ongoing restructuring into Hermod packages.
• Infrastructure: Genesis cache overhaul with full Protocol Version 11 overlay close to merging.
• Tracing: ReCon framework and trace-resources moved to Hermod project; cardano-tracer timeseries HTTP API and Grafana datasource merged; ASCII operator spellings for cardano-recon-framework.

Low level overview​

Benchmarking​

The P&T team has performed benchmarks of a highly optimized build (-O2) vs. a build using standard optimization (-O1), the latter being the current default for our releases. The benchmarks demonstrated that under low to near-idle submission load, no significant gains in resource usage were achieved. Only under full saturation over extended periods of time was a ~17% reduction in Process CPU usage measured, whereas RAM usage was stable. However, in both cases, we observed small increases (3% - 6%) in block adoption times across the network. For the current Praos Node, we still recommend the default. When using another Consensus mode with a significantly higher idle baseline, it's worth noting there's a potential trade-off that can be made.

Development​

Last month we reported that the removal of the legacy iohk-monitoring-based tracing backend from cardano-node was complete but awaiting final verification. That PR has now been merged (cardano-node PR#6580). The change eliminates approximately 11,000 lines of dead code across 15 modules and removes 9 transitive build dependencies, leaving trace-dispatcher as the sole tracing backend. With this out of the way, the remaining components of the new tracing system can be moved into the self-contained Hermod Tracing System project repository without coupling their release cycle to cardano-node. Please refer to the PR description for a detailed breakdown of now obsolete Node config keys.

Alongside that, RTView has been removed from cardano-tracer (cardano-node PR#6607). RTView was an experimental, opt-in browser dashboard hidden behind a build flag since version 0.2.4. Its functionality is now fully covered by the Grafana datasource and the timeseries server built into cardano-tracer, making the component redundant. The removal cleans up 35 source files and several build dependencies.

The P&T team has also completed a focused maintenance pass over trace-dispatcher (hermod-tracing PR#12). Additionally, work is underway to restructure trace-dispatcher into two distinct packages — hermod-tracing-api, carrying types and combinators, and hermod-tracing-core, carrying the full backend stack including EKG and Prometheus (hermod-tracing PR#16). This split will allow downstream packages that only need to instrument their own code to avoid pulling in the full backend stack as a dependency.

Infrastructure​

The genesis cache overhaul for our performance workbench (the benchmarking automation framework) is completed and close to merging (cardano-node PR#6544). As reported last month, the new approach splits a genesis into two independently-cached halves — the heavy dataset (keys, UTxO entries, delegators, DReps) and the per-era protocol parameters — which are then reassembled on demand. This greatly widens the range of parameters a benchmarking profile can modify without causing a full cache miss. The PR additionally restores non-default Plutus cost models via the Alonzo genesis extraConfig field, and includes a full Protocol Version 11 overlay with the updated cost models and execution budgets that have been submitted as a governance action on mainnet.

Tracing​

With the legacy tracing removal merged, the cardano-recon-framework and the trace-resources package have both been moved from cardano-node into the Hermod Tracing System project. The ReCon framework has been accommodated under hermod-tracing (cardano-node PR#6598, hermod-tracing PR#11); trace-resources — which provides cross-platform OS-level resource sampling (CPU, memory, network I/O, filesystem I/O, thread count) — has been renamed hermod-trace-resources (hermod-tracing PR#14), with its module namespace updated from Cardano.Logging.Resources.* to Hermod.Tracing.Resources.*. All logic, platform-specific C bindings, and tests are preserved unchanged.

The cardano-tracer timeseries HTTP API and its accompanying Grafana datasource, which we reported as being in testing last month, have now been merged (cardano-node PR#6562). The API provides PromQL-like queries over metrics timeseries stored by cardano-tracer, aligned closely with the Prometheus HTTP API schema so that existing Grafana integrations and community-built tooling can reuse familiar glue code. The Grafana datasource shipped alongside it replaces the now-removed RTView and includes a reference dashboard for operators to build from.

The cardano-recon-framework formula parsers now accept ASCII alternatives to their Unicode operators (hermod-tracing PR#15). Spellings such as \globally, \finally, && and => are now valid alongside the original Unicode forms, as are numeric indices in parentheses (e.g. \globallyN(2)). This makes authoring LTL properties more accessible in environments where Unicode input is inconvenient, and ships as part of v1.4.0.