55 posts tagged with "performance-tracing"

High level summary​

• Benchmarking: Node 10.6 (pre-)release benchmarks; compiler benchmarks: GHC9.12.
• Development: Ongoing work on queryable timeseries prototype and message-based cardano-tracer library API.
• Infrastructure: Optimitazation of trace forwarding.
• Tracing: New quality-of-life features like Prometheus service discovery; significant tech debt removal.
• Leios: Process specification for Leios conformance and performance testing / benchmarks; simulation result re-evaluation.
• Node diversity: New framework for system conformance testing based on linear temporal logic.

Low level overview​

Benchmarking​

We've performed and analysed (pre-)release benchmarks for Node 10.6.0. Our benchmarks exhibited a clear increase in RAM usage (~15%) for a block producer. We're currently investigating the underlying cause, and whether it's reproducible outside of the benchmark, without the constant, artificial stress we subject the system to. The published final results can be found in the Performance report for 10.6.0-pre.

Furthermore, we've performed compiler benchmarks for the Node built with GHC9.12 instead of our current default GHC9.6. After getting both the 10.5 and the 10.6 branches to build with the new version, we performed cluster benchmarks for both, resulting in a cross-comparison to their respective baselines. We could determine that the GHC9.12 build is on par with the GHC9.6 one for Node 10.5, and even slightly better for Node 10.6 as far as block production, diffusion and adoption metrics are concerned. However, it exhibits unexplained increases in CPU time used (~9%) and Allocations + Minor GCs (~6%). We also found that by disabling a particular optimization on GHC9.12 called 'speculative evaluation of dictionary functions', the increase in CPU time is roughly halved. So while the code base seems to be ready overall to switch to the more recent compiler version, that increase, though not dramatic, is still being investigated.

Development​

Our prototype for aggregating timeseries of Node metrics, and evaluating PromQL-like queries on them, has made significant progress. The query language is nearly complete, and is evaluated with reasonable speed. Our current work focuses on an efficient in-memory representation of timeseries - before considering any on-disk or database backed solution. To that end, we've created a microbenchmark tracking space usage, and will use real-life scraped metrics data from nodes deployed and monitored by SRE.

The clean separation of cardano-tracer into a library and an application part has entered testing. As it is built for highly concurrent operations, we chose a message-passing API where messages are always typed, and guaranteed to be consumed in the order they were emitted. Custom applications will be able to implement handlers in a composable manner that react to certain message types (or values); at the same time, they can emit messages that won't interfere with the library's internal messaging - which is assured by Haskell's type system.

Infrastructure​

We've simplified the trace-forward implementation (a non-breaking change). When forwarding traces, the host application buffers those objects in a bounded queue until they're requested. The implementation used to switch between queue sizes based on congestion, and to mitigate short, temporary service interruptions. After taking a look into the specifics of the queue implementation (in a dependency), we found that there's no overhead for the unused capacity of any queue - meaning switching the queue size to the shorter variant does not yield any advantage. We were able to simplify the implementation based on that insight - which also turned out to be an optimization. Merged: cardano-node PR#6361.

Tracing​

The new tracing system is now the default since Node 10.6 (pre-)release. Based on SRE's internal deployments and feedback, we've been adding several quality-of-life features as well as addressing some tech debt, as to enhance the new system's user experience. This includes a Prometheus HTTP service discovery feature for cardano-tracer, which allows for dynamically registering and scraping connected nodes, a CBOR formatting for trace messages which can be used to implement a binlog in the future (for space efficiency), and a basic Haskell type with parser instances for machine-readable logs (JSON or CBOR), which greatly simplifies building any tooling that consumes trace output / log files.

Additionally, there are many small changes that improve robustness, or code quality; such as safe garbage collection of threaded writers, simplification of the LogFormatting typeclass which is an interface of how traces can be rendered, properly tracing forwarding service interruptions, and fixing delay increments in the retry policy for re-establishing a forwarding connection.

The correspoding cardano-node PR#6377 is currently in draft state.

Leios​

We're finalizing the Leios performance testing plan. This substantiates our approach based on the impact analysis, and formulates concrete steps regarding formal specification of system traces, generalisation of how benchmarking workloads are defined declaratively (and generated), taking advantage of common cases in performance and conformance testing (reducing the necessity for domain-specific tooling), and setting up dedicated microbenchmarks for key components (like e.g. crypto) which are designed to provide comparability over long time.

Node diversity​

For conformance testing across diverse node implementations, we've designed a framework based on linear temporal logic (LTL). This allows for system or protocol properties to be expressed as logical propositions; it does not require any executable specification or similar, and is thus language independent. Evaluation of LTL tends to be very fast and can potentially scale up to routine verifications as part of CI. Even though Haskell was chosen as a language for the project, it can consume trace evidence from Cardano implementations in any language.

The project Cardano Trace LTL is still in prototype stage; it's able to verify a few very basic Praos properties, expressed as LTL propositions, based on our benchmarking environment's trace evidence.

High level summary​

• Benchmarking: Various maintenance to prepare for upcoming Node 10.6 changes; metrics migration guide for SPOs.
• Development: Prototyping a PromQL-based alert system for new tracing.
• Infrastructure: Located a space leak that was interfering with on-disk benchmarks.
• Tracing: Equipping dmq-node with the new tracing system; cardano-tracer library and API ongoing.
• Leios: Impact analysis and tech design; preparation for simulations hand-off.
• Hydra: Kick-off for development of system integration level benchmarks.
• Node diversity: Trace semantics specification ongoing; example test case ready for merging.

Low level overview​

Benchmarking​

We've performed various maintenance tasks to accomodate our automation to several breaking changes that come with Node 10.6. There are new constraints on Plutus cost models in genesis files which make it more difficult to set up a customized testnet, like our benchmarking cluster. Furthermore, we've been assisting with integrating and debugging the upcoming release's components, such as tx validation in the cardano-api, or the implementation of new trace messages the Node emits.

As the new tracing system will be the default on Node 10.6, we've created a tool which, given a side-by-side listing of metrics names from both systems, will automatically generate an exhaustive migration guide for SPOs. As the metrics names have changed slightly, a migration for existing monitoring setups is required as a one-off effort. The migration guide will be published on the Cardano Developer Portal.

Whilst the legacy tracing system is still fully operational in Node 10.6, the release marks the begin of its deprecation period - giving SPOs sufficient time to adjust their setups.

Development​

While cardano-tracer logs forwarded trace messages according to detailed configs, and exposes forwarded metrics, it does not provide any built-in functionality for monitoring and alerts. The experimental RTView component, which still can be built for cardano-tracer and remains fully functional, was an attempt to provide dashboards and alerts out of the box. However, due to its restricted design and low interoperability with existing monitoring standards, it has been discontinued.

Currently, we're taking another stab at this: We're building a prototype that creates timeseries directly from observed metrics, and is able to parse and evaluate PromQL queries referring to them. Based on that prototype, we'll assess resource usage and feasibility of fully building that feature into cardano-tracer. As most monitoring alerts can be (and are) defined as conditions on PromQL query results, and PromQL is an established industry standard, we see a low barrier for adaptation. Furthermore, if built sufficiently modular, it would eliminate the need to operate additional 3rd party services for scraping metrics and monitoring for alert conditions - at least in some usage scenarios.

Infrastructure​

With help and support from the Consensus team (Gracias, Javier!), we were able to locate a space leak that affected on-disk benchmarks. While the current on-disk benchmarks are representative, the space leak prevented us from scaling memory limits for the Node process with finer granularity. This will get merged post 10.6 release, and will be of much use when we do comparative benchmarks of the LMDB and the new lsm-trees on-disk backing stores.

Tracing​

We've been working on equipping the Network team's new dmq-node service with our new tracing system, trace-dispatcher - still a work in progress. Currently, the dmq-node uses plain contravariant tracing. Having a configurable system, with an API abstraction to define metadata and message serializations as well as process metrics, is a necessary step towards production-ready tracing. The added benefit of using trace-dispatcher is reusability of definitions already implemented in cardano-node, and a uniform way of how cardano-node and dmq-node are configured, as well as how the expose trace evidence and metrics.

The work on cardano-tracer as a library, with principled API and intra-process communications, is ongoing. Implementation is almost complete, and will enter testing phase soon.

Leios​

We've contributed to Leios Impact Analysis. The Performance & Tracing section summarizes how implementing, and eventually benchmarking Leios will impact various aspects of our approach. This spans from adding new Leios-specific observables into component code, to deriving submission workloads suitable for Leios, to finding a scaling approach to be able to correlate performance observations to exact changes in Leios protocol parameters.

Additionally, we're currently working on the Leios technical design and implementation plan, which lays out our approach and derives some very high-level steps on how to realize it, based on the impact analysis.

Hydra​

We've kicked off a collaboration with the Hydra team. The goal is to build system integration level benchmarks for Hydra, which can target system pressure points, and which are able to scale operations in various directions - much akin to what we're doing for the Cardano node. Eventually, those benchmarks should provide more than just immediate feedback for Hydra engineers; they should be able to yield realistic figures of what to expect from operating one (or more) Hydra heads, and what the resource requirements are to achieve that. Currently, we're familiarizing ourselves with the system and its built-in observability to assess if it meets the requirements for the benchmarks we have in mind.

Node diversity​

The work on (multi-)node conformance testing is ongoing. We're in the process of creating a specification document for semantics of existing Node traces. While a few of them might be unique to the Haskell implementation, the majority documents how Cardano itself runs and behaves; those traces can implemented accordingly across diverse Node projects.

Our own demonstration test case for conformance testing is fully implemented and ready to be merged after the Node 10.6 release. It validates a metric exposed by a node wrt. trace evidence in its logs and internal events the metric is based on; see cardano-node PR#6322.

High level summary​

• Benchmarking: Feature benchmarks for a new tx submission logic; generalized on-disk benchmark profiles.
• Development: New tracing system for cardano-submit-api; Node conformance testing groundwork, being applied to metrics.
• Infrastructure: Dijkstra era being incorporated into benchmark tooling.
• Tracing: Creating a library package and API from cardano-tracer, facilitating dedicated trace consumer applications.
• Team: Ruslan joins Performance & Tracing, will be aimed at Leios.

Low level overview​

Benchmarking​

We've performed and analysed feature benchmarks of TxSubmission protocol, version 2. It's designed to reduce redundant exchange of transactions between network nodes. To that end we've created a dedicated benchmark which is able to locally reproduce performance observations on the cloud cluster - and hence be executed with GHC profiling for tracking time and space usage.

The on-disk (LMDB) benchmarks for the cluster have been generalized into full-fledged benchmarking profiles that can be scaled for available RAM (and thus the pressure on the on-disk backing store to perform disk I/O) - and be applied to other backing store implementations as well. This will allow for direct performance comparisons with the lsm-trees solution, once it is integrated into the Node.

Development​

There's a microservice, cardano-submit-api, which is run independently from the Node (if desired). It allows for Cardano transactions to be submitted via HTTP POST, instead of Cardano native protocols. It's still using the legacy tracing system, which is why we're currently porting its logging and metrics to the new one.

We're also laying groundwork for (multi-)node conformance testing. This entails creating a specification document for semantics of existing traces. These traces can then be emitted accordingly across diverse node implementations. Given the unified semantics, these points of evidence can then be evaluated against each other, against our Haskell reference implementation, or against a model of specified / expected behaviour, resulting in a quantifiable way to assess conformance across individual implementations. Currently, we're implementing a playground version of this in the tracing system's own test suite, where we assess whether the metrics a node exposes conforms to the trace evidence in its logs, and to the metrics it forwards to cardano-tracer.

Infrastructure​

As a maintenance task, we're integrating the new Dijkstra ledger era into our performance workbench and all benchmarking tools. This will allow us to specify existing profiles in the new era (allowing us to comparatively benchmark its implementation against previous eras) as well as create new benchmarks making use of any Dijkstra-specific feature.

Tracing​

The trace consuming / processing service cardano-tracer had been built as a monolithic application. We're currently redesigning it as a more modular one, splitting it up into a library and an application proper (which hosts all its current high-level functionality). The underlying library will be equipped with an API that meets community standards in the future. For now, we're focusing on making all library components safely restartable and reconfigurable, as well as providing abstract, clean intra-process communications (cardano-tracer is a highly multi-threaded app). These capabilities are also verified in the test suite.

This will facilitate rapid development of custom, specialized applications for trace consumption and processing: The library package will provide all mid-level abstractions, as well as the Cardano native multiplexed forwarding protocol. It will allow any application to focus exclusively on implementing the high-level features it aims to.

Team​

Ruslan joined Performance & Tracing as a Software Engineer. He has a ton of functional programming experience under his belt, in Idris, in Haskell and in Scala. Additionally, he used to work as a perfomance engineer for a large, distributed commercial system. This all makes him an ideal candidate for Cardano performance engineering, and perspectively for all new Leios benchmarks and performance tests. Welcome, Ruslan!

High level summary​

• Benchmarking: Release benchmarks for 10.5; LMDB (UTxOs on-disk) benchmarks.
• Development: New memory-constrained benchmark families, targeting UTxOs on-disk.
• Infrastructure: Migration of bench cluster completed.
• Tracing: Deprecation of legacy system; TCP forwarding merged; improved self-documentation.
• Meetup: Performance & Tracing meetup held in Cardiff, Wales.

Low level overview​

Benchmarking​

We've performed and analysed release benchmarks for Node 10.5. The pre-release turned on peer sharing by default; our benchmarks indicated a negative performance impact when enabling this on block producers. The current release 10.5.1 does not enable peer sharing for block producers; the published final results can be found in the Performance report for 10.5.

Additionally, we've achieved meaningful benchmarks for UTxOs-on-disk, which use the LMDB backend. Our new profiles support seamless scaling of RAM pressure on the backend, forcing disk I/O to a varying degree. We're currently analysing the observations made, and gathering more data using different scaling factors if needed; the goal is a reliable assessmemt of LMDB's viability for block producing nodes.

Development​

Developing RAM-constrained benchmarks that would put tunable pressure on an UTxOs-on-disk backend posed a unique challenge.

First and foremost, limiting memory for past in-memory benchmarks has never been a requirement at all. A consistent approach to do so given the existing deployment had to be built, along with pertinent diagnostic tooling. Second, the LMDB backend is not managed by Haskell's GHC runtime, but comes with its own memory management - which required us to develop a double-pronged approach to selectively apply RAM limits. Lastly, other parts of the Node's code didn't support executing in tightly limited, constant space and would lead to the OS terminating the Node for running out of memory.

The interaction of various cgroup limits on Linux kernels, memory settings in our Nomad deployment and GHC RTS options let us create a stable runtime behavior over the course of the benchmark - a hard requirement, as system metrics taken at the beginning of a run must be comparable to those towards the end. A blocker for initializing the Node for benchmarks was resolved in cardano-node PR#6295: Using mmap allowed us to use the OS's virtual memory subsystem for on-demand loading instead of it being managed by the Haskell runtime - which significantly brought down the heap size required for the task.

Infrastructure​

The migration to the new benchmarking cluster is finalized. For representative performance measurements of UTxOs-on-disk, we require direct SSD storage on each cloud machine instance. Along with deploying the new hardware came a full upgrade of OS and nixpgks software. Validating the migration was extra effort: A seemingly innocent cloud provider service (which was newly added to their NixOS image) did nothing more than a heartbeat request every 5 min to some central infrastructure server. Yet, it caused the standard deviation of some of our network related metrics to double - thus reducing confidence in those metrics.

After validation, we performed a complete re-run of existing performance baselines on the new hardware.

Tracing​

Work on the new tracing system has yielded various improvements. Trace forwarding over TCP is now fully functional and merged to master. This will make setting up forwarding to remote hosts much easier than by using UNIX domain sockets / Windows named pipes. However, it's recommended for secure and isolated environments only (cardano-node PR#6241).

The auto-documentation feature of the new system has improved user experience; the generated documents are now structed in a more accessible way, and contain necessary metadata as to which Node version / commit is being documented by them (cardano-node PR#6283).

Another refactoring targeted the new system's core library, trace-dispatcher. It now comes with a minimal dependency footprint and fully equipped with all necessary type definitions to be built using Haskell's community packages (Hackage) exclusively. This greatly enhances the ease to use it for other applications than the Node as well - Cardano or non-Cardano (cardano-node PR#6268). Increasing the dependency footprint is only required for additional features, like trace forwarding.

With the upcoming Node 10.6 release, we plan to officially deprecate the legacy tracing system. This means it will enter a grace period of ~3 months, where both systems coexist in the same Node build; then, it will be decommissioned and actively removed from the code base.

Meetup​

We organized 2025's in-person team meetup in Cardiff, Wales. We had highly insightful and productive days - I would like to thank all team members who contributed, and extend my special thanks to guests from outside of the team: Your presence and contributions were greatly appreciated, Can Huzmeli and Neil Davies.

High level summary​

• Benchmarking: Feature benchmarks for ledger metrics tracer and InboundGovernor optimizations.
• Development: Ledger metrics merged; 2 hotfixes for old tracing.
• Infrastructure: Migration plan for on-disk benchmarks (LMDB, LSM-tree); initial Leios impact analysis.
• New Tracing: Tracer service now independent of Node; new feature enabling forwarding over TCP.

Low level overview​

Benchmarking​

We've completed two distinct feature benchmarks: The new periodic ledger metrics tracer and InboundGovernor optimizations on the network layer. Both features have shown a positive performance impact; the former improves CPU usage and block production metrics, the latter slightly improves diffusion metrics.

Development​

Having finalized and benchmarked the periodic ledger metrics tracer feature, it was merged to master and will be part of the upcoming 10.5 release. The feature decorrelates obtaining several metrics from the beginning of the forging loop. This avoids competition for synchronization primitives during the "hot phase" of block production. Furthermore, by decoupling those metrics from a forging tracer, we enable exposing those metrics from a relay as well. cardano-node PR#6180

Additionally, we've been vital in creating two hotfixes for the old tracing system:

1. The old tracing system metric utxoSize was missing due to using the pre-UTxO-HD variant of querying the set size. The fix ports the correct solution from the new tracing system to the old one: cardano-node PR#6217
2. On the upcoming Node 10.5 integration branch only, the old tracing system could leak file descriptors. Again, the fix was ported from the new tracing system to the old one - kudos to Karl Knutsson: iohk-monitoring PR#654

Infrastructure​

We've discussed and set up a migration plan for our benchmarking cluster hardware. For fair and representative performance measurements of on-disk backing stores of UTxO-HD, we require direct SSD storage on the machine instance in the cloud; running disk I/O through additional layers to and from some shared SSD device, even in the same data center, would introduce significant confounding factors. The plan includes invalidating as little of our existing performance baselines as possible when migrating to the new hardware. We're looking forward to benchmark the current on-disk backend (LMDB) for block producers - as well as the futuere LSM-tree based one.

We've also discussed an initial Leios impact analysis. To fairly and reliably benchmark a future Leios implementation, our infrastructure and tooling will need to be extended significantly. Several metrics won't have the same weight they currently carry for Praos, due to Leios' later finality; other metrics will need to be introduced for different new Leios block types, adding appropriate observability to the implementation. Finally, creating and submitting a saturation workload for a system which is built for extremely high throughput will be a challenge in itself.

New Tracing​

We've been working on a medium-sized refactoring that eliminates the cardano-node dependency from cardano-tracer. This means, the tracer service can now be built independently of the Node; all shared data types have been moved to some more basic packages of the new tracing system. This also enables us to issue releases of the tracer service independently of the Node's release cycle. cardano-node PR#6125

Last not least, we've kicked off development for a new feature that's been motivated by community feedback: Forwarding observables (trace messages, metrics) over TCP. Forwarding to different hosts currently assumes a UNIX domain socket that connects the Node and the tracer service through an SSH tunnel. This is a portable, versatile, and probably one of the most secure ways to transmit sensitive data. However, in an environment where an operator controls all network port mapping and firewalls, one can argue that forwarding over TCP/IP is equally viable, as it can be properly isolated - and it is much more convenient to set up and configure. The feature aims, when it's completed, to offer both forwarding routes, and let the end user decide.