Skip to main content

Benchmarking -- Node 11.0.1

· 4 min read
Michael Karg
Michael Karg
Performance and Tracing Team Lead

Setup​

As part of the release benchmarking cycle, we're comparing benchmarking runs for 2 different versions of cardano-node:

  • 10.7.1 - the current Node 10.7 performance baseline, running Protocol Version 10.
  • 11.0.1 - the latest Node 11.0 release, running Protocol Version 11.

For this benchmark, we're gathering various metrics under 2 different workloads:

  1. value-only: Each txn consumes 2 inputs and creates 2 outputs, changing the UTxO set. Full blocks (> 80kB) exclusively; high submission pressure (TPS > 10).
  2. Plutus: Each txn contains a Plutus script exhausting the per-tx execution budget. Small blocks (< 3kB) exclusively; low submission pressure (TPS < 1).

Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. All runs were performed in the Conway era using the in-memory LedgerDB backend.

Observations​

These benchmarks are about evaluating specific corner cases in a constrained environment that allows for reliable reproduction of results; they're not trying to directly recreate the operational conditions on Mainnet.

Resource Usage​

  1. 11.0.1 exhibits a small increase in Process CPU usage by 6% under saturation (and by a negligible 0.4% under Plutus) workload.
  2. Additionally, Allocation rate and Minor GCs increase slightly by 6% each under saturation (and a negligible 0.5% under Plutus) workload.
  3. Observed CPU 85% span durations show a minor fluctuation, with a 1% decrease under saturation, and a 3% increase under Plutus workload.
  4. Average RAM usage is unchanged, as are Major GC occurrences.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.

Anomaly control​

  1. Under saturation workload only, Height & Slot battles on 11.0.1 occur roughly 2.5x more frequently; however, the sample size of the benchmark is rather limited, which leads to some variance in that metric.

Forging Loop​

  1. There is a small fluctuation in leadership checks -- starting 1ms later under saturation, and 1ms earlier under Plutus workload.
  2. Under Plutus workload, the observed minor changes to block production timings are not significant.
  3. Under saturation workload, there's a minor 1ms improvement in time to Header Announcement, mostly due to a 16% improvement in Ledger Ticking time.

The metric 'Slot start to announced' (see in attachments) is cumulative, and demonstrates how far into a slot the block producing node first announces the new header.

Peer propagation​

  1. Under saturation workload only, Block Fetch duration improves by 8ms or 2%.
  2. Beyond that, there are no significant changes to block diffusion metrics.

End-to-end propagation​

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

  1. Cluster adoption metrics on 11.0.1 exhibit a minor improvement by 1% - 2% across all centiles under saturation workload.
  2. Under Plutus workload, there's a 2% improvement in the 100th centile only.

Conclusion​

  1. 11.0.1 does not contain major component changes compared to 10.7.1 - hence, the performance characteristics of those Node versions are expectedly close.
  2. This implies, the conclusions from the previous report (10.7.1 vs. 10.6.2) largely extend to the 11.0.1 release.
  3. The small increase in CPU time has to be relativized in that larger persepective: There was a massive 46% - 71% reduction in that metric going from 10.6 to 10.7.1.
  4. Thus, 11.0.1 did not exhibit any performance regressions.
  5. Protocol Version 11 can be run without performance penalties.
  6. An eye would have to be kept on Height & Slot battles - as stated before often, the benchmarks are designed to greatly amplify trends (and don't provide a large enough sample size to guarantee very high confidence in that particular metric).

Attachments​

Full comparison for value-only workload, PDF downloadable here.

Full comparison for Plutus workload, PDF downloadable here.