63 posts tagged with "performance-tracing"

High level summary​

• Benchmarking: Van Rossem HF: Created and performed benchmarks for new Plutus execution budgets and internals; PlutusV3 baseline shift.
• Development: Hermod package restructre completed; cardano-config support in trace-dispatcher.
• Infrastructure: Genesis cache overhaul / Protocol Version 11 overlay have landed.
• Tracing: Alert manager feature for cardano-tracer in design phase; native support for spans in Hermod.
• Leios: Major new beacon feature to impose memory limits; on-disk LedgerDB tx validation times benchmark in the works.

Low level overview​

Benchmarking​

The P&T team has been safeguarding the now enacted Van Rossem hard fork, which brings Protocol Version 11 with new Plutus internals and a bump to the execution budgets to mainnet. Our benchmarking automation framework, the performance workbench, gained new chain-fragment-creation profiles as well as new plutuscall-* profiles, targeting those new Plutus features specifically (cardano-node PR#6627 and cardano-node PR#6440). The former PR also adds a wb chain-fragment export op, which packages a profile run's node database, genesis files and a path-patched node config into a chain fragment consumable by beacon tool — the tooling behind the on-disk benchmarks described under Leios below.

The benchmarks performed using those new profiles have shown that the execution budget raise leads to a reasonable, predictable and expected increase in Block Adoption times. For the new PlutusV3 internals, they've also shown that they're consistently priced relative to the existing internals, and that, even when executed up to full budget exhaustion, there were no performance regressions in their implementation.

To enable those benchmarks, work has been well underway on extending plutus-scripts-bench with PlutusV3 variants of the existing scripts, and adding two new PV11 scripts targeting the new execution internals: MultiScalarMulG1 (BLS12-381 multi-scalar multiplication based on CIP-0133), and ExpModInteger, exercising the new expModInteger builtin (cardano-node PR#6440). The PR wires these through to tx-generator, and gives our analysis tool locli a new PlutusBudgetSummary module to output Plutus workload calibration data as part of comparison reports. The PR is currently under review.

Development​

The long-planned restructuring of trace-dispatcher into separate Hermod packages, which we reported as underway last month, is now complete (hermod-tracing PR#16). The stable API surface — trace types, LogFormatting/MetaTrace instances, and the structural/annotation combinators — now lives in a lightweight hermod-tracing-api package, itself split into an internal and a public sublibrary, so that packages which only need to instrument their own code no longer pull in EKG, Prometheus, CBOR forwarding or the YAML config parser as dependencies. The full backend stack remains in hermod-tracing-core, while the Prometheus TCP server has been carved out into its own optional hermod-tracing-prometheus package. All modules and identifiers were renamed from Cardano.Logging.*/TraceDispatcherMessage and friends to their Hermod.Tracing.* equivalents, and trace-dispatcher itself has been kept around as a backward-compatible mirror so existing pinned consumers keep building unchanged until they migrate.

Last but not least, we've added compatibility with the cardano-config library to trace-dispatcher (hermod-tracing PR#18).

Infrastructure​

The genesis cache overhaul and full Protocol Version 11 overlay for our performance workbench, close to merging as of last month, have now landed (cardano-node PR#6544). Beyond the split-cache "ripper" backend and the restoration of non-default Plutus cost models via the Alonzo genesis extraConfig field reported previously, the merged PR also standardises genesis creation on cardano-cli's create-testnet-data (retiring the old create-staked path and the separate byron genesis step), makes the era an explicit, validated input to wb start and wb run allocate via a new --era-name flag, and simplifies the workbench CLI to named flags only. A handful of breaking changes ship alongside: WB_BACKEND is renamed to WB_BACKEND_NAME, --relays and the genesis "spec" sub-command are removed, and non-default Plutus cost models must now be supplied through extraConfig. Analysis of older runs is unaffected, as wb analyse falls back to the previous genesis layout when the new one is absent.

Tracing​

We've begun designing a central alarm system for cardano-tracer (alarm system concept). Today, operators typically rely on third-party tooling for this — something built on top of Grafana, the Prometheus Alertmanager, or Loki log analysis — but there's no solution native to the tracing and metrics stack, and none that takes a unified approach across trace messages and metrics. The idea is to instead give cardano-tracer a single place that collects alarms from multiple sources — our hermod-recon verifier flagging a violated property, a rule evaluated against the metrics cardano-tracer already stores, or simply a trace message severe enough to warrant attention — and forward them to wherever operators want to be notified — typically a paging service's webhook API, but custom webhooks, email, and plain logging are supported too — while keeping a queryable history of past alarms. It deliberately stays a one-shot notification layer rather than a full incident tracker: there's no acknowledgement or resolution workflow, that's left to whatever tooling operators already use for that. This is still an early design and not yet reflected in any implementation work.

On the Hermod side, we're also exploring native support for nested spans, in the spirit of tracing systems like OpenTelemetry. A span represents a bounded unit of work — say, processing a single transaction or forging a block — and the point is to let automated log analysis reconstruct not just individual events, but how long an operation took and how it decomposes into sub-operations, by pairing up an opening and a closing trace message that share a unique identifier. The tricky part is Haskell's laziness: without care, an expression's evaluation, and the trace events it triggers, can end up happening after its enclosing span has already been closed, so a fair amount of the design work goes into guaranteeing that no trace event ever escapes the span it logically belongs to. There's no branch to point to yet — this is still exploratory work.

Leios​

Building on the chain-fragment export tooling introduced above, beacon — the ledger ops benchmarking tool in ouroboros-consensus-tools — gained a memory-limiting feature aimed squarely at tx validation times using an on-disk LedgerDB backend and forcing disk I/O at the same time (beacon PR#12, currently under review).

A new flag forces the LSM (on-disk) backend to genuinely exercise disk I/O, rather than being served invisibly out of the OS page cache; --mem-limit/--heap-limit cgroup caps serve as a fallback for past db-analyser builds without support for that new flag. Every run now records peak RSS and block I/O counts. A representative "sweet spot" using those new mechanisms has been determined empirically for the chain fragments used in this year's April measurements. This finally gives us concrete on-disk LedgerDB metrics by determining the overhead of I/O-backed validation times vs. pure ones, closing a gap in our Leios throughput benchmarking. A new beacon/docs/METHODOLOGY.md documents the overall approach; benchmarks are currently being performed on our benchmarking cluster, to obtain measurements on the same hardware as we did in April for the in-memory backend — but now with LSM backend forced to perform disk I/O.

High level summary​

• Benchmarking: -O2 build optimization benchmarks.
• Development: Legacy iohk-monitoring tracing backend removed from cardano-node; RTView removed from cardano-tracer; trace-dispatcher maintenance and ongoing restructuring into Hermod packages.
• Infrastructure: Genesis cache overhaul with full Protocol Version 11 overlay close to merging.
• Tracing: ReCon framework and trace-resources moved to Hermod project; cardano-tracer timeseries HTTP API and Grafana datasource merged; ASCII operator spellings for cardano-recon-framework.

Low level overview​

Benchmarking​

The P&T team has performed benchmarks of a highly optimized build (-O2) vs. a build using standard optimization (-O1), the latter being the current default for our releases. The benchmarks demonstrated that under low to near-idle submission load, no significant gains in resource usage were achieved. Only under full saturation over extended periods of time was a ~17% reduction in Process CPU usage measured, whereas RAM usage was stable. However, in both cases, we observed small increases (3% - 6%) in block adoption times across the network. For the current Praos Node, we still recommend the default. When using another Consensus mode with a significantly higher idle baseline, it's worth noting there's a potential trade-off that can be made.

Development​

Last month we reported that the removal of the legacy iohk-monitoring-based tracing backend from cardano-node was complete but awaiting final verification. That PR has now been merged (cardano-node PR#6580). The change eliminates approximately 11,000 lines of dead code across 15 modules and removes 9 transitive build dependencies, leaving trace-dispatcher as the sole tracing backend. With this out of the way, the remaining components of the new tracing system can be moved into the self-contained Hermod Tracing System project repository without coupling their release cycle to cardano-node. Please refer to the PR description for a detailed breakdown of now obsolete Node config keys.

Alongside that, RTView has been removed from cardano-tracer (cardano-node PR#6607). RTView was an experimental, opt-in browser dashboard hidden behind a build flag since version 0.2.4. Its functionality is now fully covered by the Grafana datasource and the timeseries server built into cardano-tracer, making the component redundant. The removal cleans up 35 source files and several build dependencies.

The P&T team has also completed a focused maintenance pass over trace-dispatcher (hermod-tracing PR#12). Additionally, work is underway to restructure trace-dispatcher into two distinct packages — hermod-tracing-api, carrying types and combinators, and hermod-tracing-core, carrying the full backend stack including EKG and Prometheus (hermod-tracing PR#16). This split will allow downstream packages that only need to instrument their own code to avoid pulling in the full backend stack as a dependency.

Infrastructure​

The genesis cache overhaul for our performance workbench (the benchmarking automation framework) is completed and close to merging (cardano-node PR#6544). As reported last month, the new approach splits a genesis into two independently-cached halves — the heavy dataset (keys, UTxO entries, delegators, DReps) and the per-era protocol parameters — which are then reassembled on demand. This greatly widens the range of parameters a benchmarking profile can modify without causing a full cache miss. The PR additionally restores non-default Plutus cost models via the Alonzo genesis extraConfig field, and includes a full Protocol Version 11 overlay with the updated cost models and execution budgets that have been submitted as a governance action on mainnet.

Tracing​

With the legacy tracing removal merged, the cardano-recon-framework and the trace-resources package have both been moved from cardano-node into the Hermod Tracing System project. The ReCon framework has been accommodated under hermod-tracing (cardano-node PR#6598, hermod-tracing PR#11); trace-resources — which provides cross-platform OS-level resource sampling (CPU, memory, network I/O, filesystem I/O, thread count) — has been renamed hermod-trace-resources (hermod-tracing PR#14), with its module namespace updated from Cardano.Logging.Resources.* to Hermod.Tracing.Resources.*. All logic, platform-specific C bindings, and tests are preserved unchanged.

The cardano-tracer timeseries HTTP API and its accompanying Grafana datasource, which we reported as being in testing last month, have now been merged (cardano-node PR#6562). The API provides PromQL-like queries over metrics timeseries stored by cardano-tracer, aligned closely with the Prometheus HTTP API schema so that existing Grafana integrations and community-built tooling can reuse familiar glue code. The Grafana datasource shipped alongside it replaces the now-removed RTView and includes a reference dashboard for operators to build from.

The cardano-recon-framework formula parsers now accept ASCII alternatives to their Unicode operators (hermod-tracing PR#15). Spellings such as \globally, \finally, && and => are now valid alongside the original Unicode forms, as are numeric indices in parentheses (e.g. \globallyN(2)). This makes authoring LTL properties more accessible in environments where Unicode input is inconvenient, and ships as part of v1.4.0.

High level summary​

• Benchmarking: Release benchmarks for 11.0.1; Feature benchmarks for: TxSubmissionLogicV2; Compiler version.
• Development: Removal of legacy tracing completed - not yet merged.
• Infrastructure: Genesis caching and post-processing completed - not yet merged.
• Tracing: cardano-tracer HTTP API for metrics timeseries queries and Grafana datasource - not yet merged.
• Leios: Leios/Mempool benchmarks using tx-centrifuge.
• Node Diversity: Formal trace schema definition merged; Conformance framework to be presented at Porto workshop.

Low level overview​

Benchmarking​

We've performed, analysed and published relase benchmarks for Node version 11.0.1 - the release shows no performance regressions compared to 10.7.1. These benchmarks ran under Protocol Version 11, and were required to ensure there's no performance risk in using this version.

Furthermore, we've run feature benchmarks for a new incarnation of v2 of the tx submission logic. The new logic is an optimization and aims, among other things, to reduce redundancy in tx diffusion. While the feature is experimental, the benchmarks provided valuable measurements and data for the network team to move it forward.

Additionally, we've re-run benchmarks using the GHC9.12 compiler version on the new 11.0.1 baseline; since 10.6.2, there have been many changes in Ledger which impact generated code and compiler optimizations. While there's no fundamental performance blocker to use this more recent compiler on our code base, there are still a few unknowns. The data is currently still under review and discussion.

Development​

With the upcoming 11.1 release, the legacy tracing system 'iohk-monitoring-framework' will finally be removed from the Node. The change extensive, as it involves large differences in project dependencies, in code, in configuration and in test suites. Old and new tracing system have been part of the Node build side-by-side for roughly two years now, with the new tracing system gaining wider adoption the last half year. Removing the need to stay backwards compatible with the legacy system within the same build unblocks several planned features for the new system, as well as finally moving it out into its own self-contained Hermod Tracing System project repository.

While the implemention is complete, the PR cardano-node PR#6580 is currently still in draft state, awaiting full verification and testing.

Infrastructure​

The modularization of our automation's genesis cache is completed. In addition to quickly stitching together a custom genesis with a huge amount of injected staking data, it allows for all protocol-relevant fields of genesis to be freshly generated by cardano-cli - and not taken from the cache. This means, the post-processing has now been reduced to a minimum; that improves confidence in the benchmarking profiles insofar as it eliminates testing of workbench changes still being correctly patched onto potentially very long-lived cache entries on a variety of hosts.

Moreover, this change includes a proper profile overlay for Protocol Version 11, which includes changes to Plutus cost models and execution budgets that have already been submitted as a gov action on Mainnet. The (quite extensive) PR is currently in draft state and under testing: cardano-node PR#6544.

Tracing​

The new version of cardano-tracer will come with an (opt-in via config) HTTP REST API to query metrics timeseries directly. As cardano-tracer can now store metrics of all connected Nodes, it's able to evaulate PromQL-like queries directly. This can be used as an alternative to having Prometheus scrape all of those processes. With the new release, we made the (previously experimental) API as much aligned to what users are accustomed to from Prometheus, so that it has reached reasonable stability.

Moreover, we built - from scratch - a Grafana datasource using that API. This datasource contains a dashboard to replace the deprecated 'RTView' component of cardano-tracer, and is intended to serve as a reference for the community to define their own dashboards and queries according to their monitoring needs.

This PR, too, is fairly extensive, and also contains several improvements and fixes of the underlying cardano-timeseries-io package: cardano-node PR#6562, currently in testing phase.

Leios​

We've created, and performed, full cluster benchmarks for Leios - using our new high-pressure submission tool tx-centrifuge. The point of interest of these benchmarks was observing Mempool behaviour, under various levels of fragmentation, and various configurations as to its capacity. These benchmarks are meant close a gap to the Leios simulations providing evidence, by measuring concrete timings of a concrete Mempool implementation. The benchmarks have shown that a standard Mempool tuned to Praos will likely throttle maximum throughput for Leios. With this benchmark at hand, and Mempool identified as a potential bottleneck, the necessary adjustments or optimizations can always be confirmed and backed up by evidence.

Node Diversity​

The comprehensive formal schema definition of the Node's existing trace messages has been merged (cardano-node PR#6527). This encodes the syntax and semantics of all the observable events that the Haskell Node implementation provides. Thus, it can serve as a reference to what diverse clients may implement - to gain comparability in protocol conformance, network performance, and the reuse of existing tooling relying on those observables.

That being said, the cardano-recon-framework is one such example. We've continuosly improved our Linear Temporal Logic based trace verifier for system behaviour; we've set several interesting properties that can be checked continuously from Node logs. One of our team will attend the Node Diversity workshop in Porto beginning of June, and contribute a presentation and a demo of this framework.

High level summary​

• Benchmarking: Release benchmarks for 10.7.1; LSM-trees benchmarks; Plutus interpreter benchmarks.
• Development: tx-centrifuge - high-pressure tx submission service moved to testing.
• Infrastructure: Optimizing benchmark genesis caching and post-processing to be highly modular.
• Tracing: cardano-timeseries-io now integrated with cardano-tracer, offering HTTP query API.
• Leios: Tx validation benchmarks with beacon; Arithmetic extension of cardano-recon-framework.
• Node Diversity: Formal trace schema definition ready.

Low level overview​

Benchmarking​

The release benchmarks for 10.7.1 have been an iterative process. Various changes on 10.7 caused several performance regressions, which needed to be isolated from each other, located individually and addressed. This led to running and analyzing several full benchmarks, to confirm each change individually as well. Finally, the 10.7.1 release turned out to be a small, but consistent improvement as far as block production, diffusion and adoption metrics are concerned, and a huge improvement in CPU time and usage patterns.

With a healthy baseline in place, we were able to run benchmarks on the LSM-trees on-disk backing store. Initial performance results show the on-disk backend to be on par with the in-memory one given sufficient RAM. This is the optimal outcome, as using LSM-trees instead of in-memory without changing the underlying hardware does not incur a performance penalty. We're currently running benchmarks where the underlying hardware is indeed changed, and where multiple constraints on the Haskell heap and the OS's headroom for page caching force disk I/O under low RAM conditions. We'll then assess the effect on system and network metrics.

Last but not least, we confirmed a change to the Plutus interpreter - which impacts its performance characteristics - to be healthy and ready for integration.

Development​

The new tx-centrifuge project has reached the MVP stage. It's built for massive tx submission pressure and seamless scaling, so it will be able to saturate a network running Leios over extended periods of time. To confirm all its intended properties, and iron out bumps in the pipeline, we're currently running tests on the benchmarking cluster - albeit on Praos nodes, as for those, we exactly know the expected outcome. For details on tx-centrifuge architecture and design, please see cardano-node PR#6494.

Infrastructure​

Our benchmarks require very large genesis files, which are costly (hours, in the worst case) to create for each and every run. This is why our automation uses caching. We're currently reworking the caching mechanism so a genesis can be highly modular. This will lead to much more flexibility in applying a specific benchmarking profile to an existing cache entry, and widen the range of parameters a profile can modify without leading to a genesis cache miss.

Tracing​

Our new Haskell library cardano-timeseries-io, which builds and stores timeseries of metrics from multiple sources, has been integrated with the cardano-tracer service. This now enables arbitrary queries over those timeseries to be submitted via an HTTP API. While this does not replace existing Prometheus endpoints, cardano-tracer can now answer PromQL-like queries directly without the need to run a separate scraper (cardano-node PR#6473).

This is currently an experimental feature; the API is not yet stable. We're working on aligning the the request and response schema closely with the Prometheus HTTP API, such that a Grafana integration, or any potential community-bulit frontend, can reuse much of the existing glue code out there.

Leios​

We've created, and performed, transaction validation benchmarks for the current Cardano Ledger implementation. The benchmarks use beacon as a framework, which means looking at ledger operations only. Of those, anything besides block application using different validation strategies has been factored out. As input data, several synthetic workloads are used varying in tx content (script execution, or just moving ADA), block / tx batch size, or number of tx inputs. In the context of Leios, this allows for confirming protocol assumptions, or determining (and validating) potentially necessary optimizations in the ledger. For a full report and discussion, please see Leios issue#553. Next steps will be scaling different workload properties systematically, as well as forcing tx inputs to be read back from disk.

The cardano-recon-framework, a Linear Temporal Logic based verifier for observed system behaviour, now has better support for existential quantification in its propositions, as well as added support for Presburger arithmetic. This arithmetic extension allows for a wider range of properties to be evaluated, which are of particular interest to Leios. Those features, along with some quality-of-life improvements, are already released on CHaP; relevant PRs are cardano-node PR#6531 and cardano-node PR#6546. Current work is narrowing down the context in the framework's output; in case of a property not being satisfied, this will be highly specific as to which piece of evidence is the root cause for it.

Node Diversity​

The comprehensive formal schema definition of the Node's existing trace messages is being merged (cardano-node PR#6527). Definitions are extracted directly from the actual implementation into a fully validated JSON schema. The extracted data is automatically verified, and can be compared to past data, to capture any changes. The schema can be amended manually with comments or refinement types, and these user-provided annotations will be merged with the extracted data - with a notification if any conflict is discovered. Future work will see hardened verification, as well as rendering a human-readable document, detailing the specification exhaustively.

High level summary​

• Benchmarking: Compiler benchmarks on 10.6.2; Trace evaluation feature benchmarks.
• Development: Started new project tx-centrifuge: A tx submission service generating extremely high, continuous workload.
• Infrastructure: Small maintenance items, such as fixing profiled nix builds for local benchmarking.
• Tracing: New tracing system now its own project: Hermod Tracing; New library cardano-timeseries-io, which accumulates metrics into queryable timeseries, released.
• Leios: cardano-recon-framework (formerly LTL Trace Verifier) integrated and in use.
• Node Diversity: Formal trace schema definition nearing merge; Trace forwarding in native Rust on hiatus.

Low level overview​

Benchmarking​

We've repeated the GHC9.12 compiler benchmarks on Node 10.6.2, which we now know to be completely free of regressions or any space leak. This confirmed our earlier findings that the code generated by GHC9.12 is on par performance-wise as far as block production, diffusion and adoption metrics go, but it exhibits unexplained increases in CPU time used, Allocations & Minor GCs. Several potential suspects for causing this have been identified with a profiled build. However, many of those will be replaced or changed in the 10.7 release, so that this benchmark will have to be re-run on Node 10.7.

The feature for new tracing, which forces a lazy trace value in a controlled section of code, is slated for inclusion in Node 10.7. To that end, we backported it to Node 10.6.2 and performed feature benchmarks for it - to ensure it won't distort the upcoming 10.7 performance baseline. Indeed we found the performance impact of that feature to be negligible in all categories of observed metrics.

Development​

We've started a new project - tx-centrifuge - for transaction submission (i.e. workload generation) during benchmarks and other scenarios. It is meant to be complementary to the existing tx-generator. The latter is tailored very much to our Praos benchmarking use case and the implementation is based on a rather monolithic design. tx-centrifuge's approach however is a different one. It's built for seamless scaling, both horizontally and vertically. This means it will be able to saturate a network running Leios over extended periods of time, due to its massive tx output. Furthermore, it's able to cut down the setup phase (where UTxOs are created for benchmarking) and immediately launch into the benchmark phase. This also enables it to function as a potentially long-running, configurable submission service for scenarios other than benchmarking. The implementation is currently in prototype stage.

Infrastructure​

As far as infrastructure is concerned, we've addressed various small-sized maintenance tasks. This includes fixing profiled nix builds for local benchmarks, migrating benchmarking profiles and configs to the upcoming Node 10.7 release and increasing robustness of the locli analysis tool in dealing with incomplete / partial trace output.

Tracing​

Our new tracing system has been set up as its own project - and named the Hermod Tracing System. As of now, we've only migrated the core package trace-dispatcher. This marks the first step of eventually moving all tracing and metrics related packages out of the cardano-node project, and bundling them with consistent branding, API and documentation. Eventually, the system will be generalized so that it can be used by any Haskell application - not just cardano-node. Seeing that the dmq-node already adopted it, we have reason to assume it might be considered by the broader community as go-to choice to add principled observability to an application.

We've built and released a new Haskell library cardano-timeseries-io (cardano-node PR#6495). The library builds and stores timeseries of metrics from multiple source applications, much like Prometheus. It can process queries over those timeseries in a query language quite similar to PromQL. Integration into cardano-tracer, the trace / metrics processing service, is ongoing work. It will allow for custom monitoring solutions and alerts directly from cardano-tracer, without the need to scrape metrics and maintain them externally. It is not meant to replace existing Prometheus endpoints, rather provide richer functionality out of the box if desired: cardano-node PR#6473.

Leios​

We've released cardano-recon-framework, formerly known as the Linear Temporal Logic (LTL) Trace Verifier (cardano-node PR#6454). It's already seen adoption, and is used productively to verify system properties and conformance exclusively based on live trace output. We've been asked by Formal Methods Engineering to extend the LTL fragment the framework uses, such that a wider range of properties can be expressed; work on that is already ongoing.

Node Diversity​

The comprehensive formal schema definition of all the Node's existing trace messages is nearing integration / merging. The initial version will be able to extract all definitions from the actual implementation into a fully validated JSON schema. Future work will address completing the automated verification suite, adding a mechanism to amend the extracted schema manually (e.g. with comments or refinement types) and a pipeline to facilitate usage, such as automatic derivation of a parser, or rendering of a human-readable specification PDF.

Due to resourcing issues, the trace / metrics forwarding mini-protocol implementation in native Rust, unfortunately, had to be put on hiatus for the forseeable future.