Setup​

As part of the release benchmarking cycle, we're comparing benchmarking runs for 2 different versions of cardano-node:

• 10.7.1 - the current Node 10.7 performance baseline, running Protocol Version 10.
• 11.0.1 - the latest Node 11.0 release, running Protocol Version 11.

For this benchmark, we're gathering various metrics under 2 different workloads:

1. value-only: Each txn consumes 2 inputs and creates 2 outputs, changing the UTxO set. Full blocks (> 80kB) exclusively; high submission pressure (TPS > 10).
2. Plutus: Each txn contains a Plutus script exhausting the per-tx execution budget. Small blocks (< 3kB) exclusively; low submission pressure (TPS < 1).

Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. All runs were performed in the Conway era using the in-memory LedgerDB backend.

Observations​

These benchmarks are about evaluating specific corner cases in a constrained environment that allows for reliable reproduction of results; they're not trying to directly recreate the operational conditions on Mainnet.

Resource Usage​

1. 11.0.1 exhibits a small increase in Process CPU usage by 6% under saturation (and by a negligible 0.4% under Plutus) workload.
2. Additionally, Allocation rate and Minor GCs increase slightly by 6% each under saturation (and a negligible 0.5% under Plutus) workload.
3. Observed CPU 85% span durations show a minor fluctuation, with a 1% decrease under saturation, and a 3% increase under Plutus workload.
4. Average RAM usage is unchanged, as are Major GC occurrences.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.

Anomaly control​

1. Under saturation workload only, Height & Slot battles on 11.0.1 occur roughly 2.5x more frequently; however, the sample size of the benchmark is rather limited, which leads to some variance in that metric.

Forging Loop​

1. There is a small fluctuation in leadership checks -- starting 1ms later under saturation, and 1ms earlier under Plutus workload.
2. Under Plutus workload, the observed minor changes to block production timings are not significant.
3. Under saturation workload, there's a minor 1ms improvement in time to Header Announcement, mostly due to a 16% improvement in Ledger Ticking time.

The metric 'Slot start to announced' (see in attachments) is cumulative, and demonstrates how far into a slot the block producing node first announces the new header.

Peer propagation​

1. Under saturation workload only, Block Fetch duration improves by 8ms or 2%.
2. Beyond that, there are no significant changes to block diffusion metrics.

End-to-end propagation​

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

1. Cluster adoption metrics on 11.0.1 exhibit a minor improvement by 1% - 2% across all centiles under saturation workload.
2. Under Plutus workload, there's a 2% improvement in the 100th centile only.

Conclusion​

1. 11.0.1 does not contain major component changes compared to 10.7.1 - hence, the performance characteristics of those Node versions are expectedly close.
2. This implies, the conclusions from the previous report (10.7.1 vs. 10.6.2) largely extend to the 11.0.1 release.
3. The small increase in CPU time has to be relativized in that larger persepective: There was a massive 46% - 71% reduction in that metric going from 10.6 to 10.7.1.
4. Thus, 11.0.1 did not exhibit any performance regressions.
5. Protocol Version 11 can be run without performance penalties.
6. An eye would have to be kept on Height & Slot battles - as stated before often, the benchmarks are designed to greatly amplify trends (and don't provide a large enough sample size to guarantee very high confidence in that particular metric).

Attachments​

Full comparison for value-only workload, PDF downloadable here.

Full comparison for Plutus workload, PDF downloadable here.

Setup​

As part of the release benchmarking cycle, we're comparing benchmarking runs for 2 different versions of cardano-node:

• 10.6.2 - the current Node 10.6 performance baseline
• 10.7.1 - the latest Node 10.7 release

For this benchmark, we're gathering various metrics under 2 different workloads:

1. value-only: Each txn consumes 2 inputs and creates 2 outputs, changing the UTxO set. Full blocks (> 80kB) exclusively; high submission pressure (TPS > 10).
2. Plutus: Each txn contains a Plutus script exhausting the per-tx execution budget. Small blocks (< 3kB) exclusively; low submission pressure (TPS < 1).

Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. All runs were performed in the Conway era using the in-memory LedgerDB backend.

Observations​

These benchmarks are about evaluating specific corner cases in a constrained environment that allows for reliable reproduction of results; they're not trying to directly recreate the operational conditions on Mainnet.

Resource Usage​

1. 10.7.1 exhibits a massive reduction in Process CPU usage -- 46% under saturation workload, and 71% under Plutus workload.
2. Allocation rate and Minor GCs increase moderately (by 14% - 16%) under saturation workload; however, GC CPU usage still shows a 4% improvement.
3. Under Plutus workload, they both slightly decrease (by 4%), and GC CPU usage improves by 16%.
4. Major GC events occur less frequently, by 16% - 18%, depending on workload.
5. Observed CPU 85% spans exhibit a decrease in duration as well -- ~1.3 - ~1.4 slots depending on workload.
6. Average RAM usage exhibits a slight increase by 2% for either workload.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.

Anomaly control​

1. Under saturation, Height & Slot battles on 10.7.1 occur 12% less frequently.

Forging Loop​

1. Under saturation workload, there are no significant changes in block production metrics.
2. Under Plutus workload, there are various small improvements in Ledger Ticking (4ms), Self Adoption (2ms) and Forged to Sending (1ms).
3. As a result, a block producer is able to announce a new header 3ms or 13% earlier into a slot (Plutus workload only).

The metric 'Slot start to announced' (see in attachments) is cumulative, and demonstrates how far into a slot the block producing node first announces the new header.

Peer propagation​

1. Under saturation workload, Block Fetch duration decreases by 7ms or 2%.
2. Average Adoption times on the peers improve by 2ms / 3ms (3% / 8%), depending on workload.

End-to-end propagation​

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

1. Cluster adoption metrics on 10.7.1 exhibit a small but consistent improvement by 2% - 3% for either workload in the 80th centile and above.

Conclusion​

1. 10.7.1 contains major optimizations in Ledger and Consensus, as well as various smaller ones such as in the metrics system. The large improvement in CPU usage confirms their effectiveness.
2. The benchmarks are designed to amplify trends. Managing expectations, the observed improvements will be somewhat less prominent on Mainnet; furthermore, some of the optimizations target block producers only.
3. A space leak that was present on 10.7.0 is now provably absent from 10.7.1.
4. 10.7.1 is a small, but consistent improvement over 10.6.2 as far as block production, diffusion and adoption metrics are concerned.
5. Conversely, not a single performance regression could be observed on 10.7.1.

Attachments​

Full comparison for value-only workload, PDF downloadable here.

Full comparison for Plutus workload, PDF downloadable here.

Setup​

As part of the release benchmarking cycle, we're comparing benchmarking runs for 2 different versions of cardano-node:

• 10.5.4 - the current Node 10.5 Mainnet release
• 10.6.2 - the current Node 10.6 Mainnet release

For this benchmark, we're gathering various metrics under 2 different workloads:

1. value-only: Each txn consumes 2 inputs and creates 2 outputs, changing the UTxO set. Full blocks (> 80kB) exclusively; high submission pressure (TPS > 10).
2. Plutus: Each txn contains a Plutus script exhausting the per-tx execution budget. Small blocks (< 3kB) exclusively; low submission pressure (TPS < 1).

Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. All runs were performed in the Conway era.

Observations​

These benchmarks are about evaluating specific corner cases in a constrained environment that allows for reliable reproduction of results; they're not trying to directly recreate the operational conditions on Mainnet.

Resource Usage​

1. 10.6.2 exhibits a clear 6% reduction in Process CPU usage under full saturation workload, and a 2% reduction under Plutus workload.
2. Allocation rate and Minor GCs are significantly reduced as well (~59% and 64% depending on workload).
3. Major GC events go up by 27% and 34%, depending on workload.
4. Observed CPU 85% spans exhibit a clear increase in duration -- ~4.1 and ~1.8 slots depending on workload.
5. RAM usage decreases by 19% and 24% depending on workload. HOWEVER: This is a known result of optimizations in the benchmarking setup. From a seperate benchmark with those optimizations applied on top of 10.5, we know that 10.6.2 exhibits a very minor increase (1% - 2%) in average Heap size, with a reduction in maximum Heap size under saturation workload only.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.

Forging Loop​

1. We can observe Ledger Ticking decrease by 3ms - 5ms.
2. Under saturation workload only, Mempool snapshotting and Forged to Sending exhibit small increases by 2ms each.
3. As a result, a block producer is able to announce a new header 2ms - 3ms earlier into a slot (depending on workload).
4. Self adoption on the forging node also decreases by 2ms - 3ms.

The metric 'Slot start to announced' (see in attachments) is cumulative, and demonstrates how far into a slot the block producing node first announces the new header.

Peer propagation​

1. Block Fetch duration decreases by 2% (3ms or 5ms, depending on workload).
2. For small blocks, Adoption times on the peers increase by 2ms, however, for large blocks they decrease by 3ms.

End-to-end propagation​

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

1. Cluster adoption metrics on 10.6.2 exhibit no significant change under high submission / large blocks workload.
2. Under low submission / small blocks workload, there are small improvements of 1% - 3% in the body of the distribution, and a 10% increase in the 100th centile.

Conclusion​

1. 10.6.2 is more efficient in its usage of CPU time. Considered in conjunction with the increase in CPU 85% spans, it points to a redistribution of work which results in less bursts, and more plateaus.
2. Seeing there is no negative impact on block production and diffusion metrics, and considering the overall decrease in CPU usage, the CPU 85% span increase can't be interpreted as a risk to performance or responsiveneess.
3. The RAM increase observed on 10.6.0-pre has been successfully fixed in 10.6.2.
4. 10.6.2 is more efficient wrt. block production and diffusion.
5. Adoption metrics are largely equivalent to 10.5.4; the 10% increase in the Plutus workload's 100th centile is an outlier resulting from the benchmark's very restrictive topology. Unless the increase also manifests in the 98th and 96th centiles, or below, it is not considered a risk.
6. From a performance perspective, we can determine 10.6.2 to be regression-free and attest a clean bill of health.

Attachments​

Full comparison for value-only workload, PDF downloadable here.

Full comparison for Plutus workload, PDF downloadable here.

Setup​

As part of the release benchmarking cycle, we're comparing benchmarking runs for 2 different versions of cardano-node:

• 10.5-baseline - performance baseline from previous Node 10.5 releases
• 10.5.4 - the current Node 10.5.4 release

For this benchmark, we're gathering various metrics under 2 different workloads:

1. value-only: Each transaction consumes 2 inputs and creates 2 outputs, changing the UTxO set. This workload produces full blocks (> 80kB) exclusively.
2. Plutus: Each transaction contains a Plutus script exhausting the per-tx execution budget. This workload produces small blocks (< 3kB) exclusively.

Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. All runs were performed in the Conway era.

Observations​

These benchmarks are about evaluating specific corner cases in a constrained environment that allows for reliable reproduction of results; they're not trying to directly recreate the operational conditions on Mainnet.

Resource Usage​

1. 10.5.4 exhibits a slight 3% increase in Process CPU usage -- regardless of workload type.
2. RAM usage decreases slightly by 3% (0.23GiB - 0.26GiB, depending on workload).
3. Observed CPU 85% span duration exhibits a very faint increase -- between ~0.25 and ~0.35 slots.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.

Forging Loop​

1. For large blocks only, we can observe a small increase in Adoption time by 2ms.
2. Beyond that, there are no significant changes to block production metrics.

The metric 'Slot start to announced' (see in attachments) is cumulative, and demonstrates how far into a slot the block producing node first announces the new header.

Peer propagation​

1. For large blocks, average Block Fetch duration decreases by 2ms, but Adoption times on the peers increase by 2ms.
2. For small blocks, average Block Fetch duration increases by 5ms, but Adoption times on the peers decrease by 3ms.

End-to-end propagation​

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

1. Cluster adoption metrics on 10.5.4 exhibit no significant change under high submission / large blocks workload.
2. Under Plutus workload (low submission / small blocks), there's a minor increase by 2% - 3% between the 80th and 98th centiles.

Conclusion​

1. The small up and down changes in Adoption times and Block fetch duration are well within margin of slack and do not pose any kind of performance risk.
2. Transitively, this applies to observed E2E propagation metrics as well.
3. All in all, 10.5.4 is pretty closely aligned with the existing 10.5 performance baseline.
4. From a performance perspective, we can determine it to be regression-free and attest a clean bill of health.

Attachments​

Full comparison for value-only workload, PDF downloadable here.

Full comparison for Plutus workload, PDF downloadable here.

Setup​

This report compares benchmarking runs for 3 different settings of the Plutus memory execution budget:

• 10.6.1-jan26 - current mainnet memory execution budget
• mem-x1.5 - 1.5 x current mainnet memory execution budget
• mem-x2 - 2 x current mainnet memory execution budget

For this comparison, we gather various metrics under the Plutus workload used in release benchmarks: Each block produced during the benchmark contains 4 identical script transactions calibrated to fully exhaust the memory execution budget. Thus, script execution is constrained by the memory budget limit every case. The workload produces small blocks (< 3kB) exclusively.

Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology.

Identical scaling benchmarks were performed in Q1 2025 on Node 10.2 / GHC8.10 and Node 10.3 / GHC9.6. This comparison aims to ascertain the past observations and conclusions still apply, given the most recent Node version (10.6.1, with patches for 10.6.2 backported) and its recommended compiler version GHC9.6.7.

Observations​

Resource Usage​

1. Scaling the memory budget impacts Allocation Rate and Minor GCs. 1.5 x the budget results in rises of 21% - 22%; for doubling the budget the corresponding rises are 36% - 38%.
2. These increases exhibit a slightly sublinear correlation with raising mem budget; however, in absolute terms they are much steeper (roughly 4x) compared to Node 10.3.
3. The Node process RAM footprint is unaffected by and the effects on Process CPU usage is negligible for either scaling factor.
4. CPU 85% span duration exhibits a slight constant increase (approx. 0.5 slots) when scaling the mem budget, regardless of the factor.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.

Forging Loop​

1. Scaling the memory budget has significant impact on self adoption time only.
2. Scaling by factor 1.5 leads to an 11ms (or 25%) increase, whereas factor 2 leads to 22ms (51%); this is fully congruent with the observations on Node 10.3.
3. This increase is linearly correlated with raising the mem budget.
4. With increased memory budget, counterintuitively, the time from slot start until new header announcement decreases slightly - by 4ms (factor 1.5) and 2ms (factor 2). This was not observed on Node 10.3.

Peer propagation​

1. Same as on the block producer, scaling the memory budget has significant impact on block adoption times only.
2. Scaling by factor 1.5 leads to an 10ms (or 22%) increase, whereas factor 2 leads to 17ms (37%).
3. Again, these increases exhibit a slightly sublinear correlation with raising the mem budget - largely congruent with the absolute increases seen on Node 10.3.

End-to-end propagation​

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

1. 1.5 x the memory budget results in a minor increase of 2ms - 16ms in cluster adoption times (1% - 3%).
2. 2 x the memory budget results in a slight 10ms - 13ms increase (2% - 3%).
3. These increases are clearly lower (very roughly 2.5x) than those observed on Node 10.3.

Conclusion​

These measurements outline the headroom for raising the memory budget, along with the expected performance impact:

1. Block adoption time is the only network metric that's affected significantly, increasing both on the forger and the peers by the same extent.
2. These increases seem to correspond linearly at worst with raising the memory budget. This gives excellent predictability of performance impact up to a hypothetical 100% raise.
3. Expectedly, more allocations and minor GCs take place; however, CPU and RAM usage remain nearly constant.
4. Block diffusion is only marginally affected by changing the execution budget: Due to header pipelining, announcing and (re-)sending a block precedes adoption in most cases.
5. As such, measurements taken with either budget adjustment do not indicate performance risks to the network, but clearly evidence their respective performance cost.
6. With the exception of extra allocations, all measurements point to a recent Node version delivering equal or slightly better performance compared to 10.3 given some memory budget increase.

Attachment​

Full report PDF downloadable here.

Setup​

As part of the release benchmarking cycle, we're comparing benchmarking runs for 2 different versions of cardano-node:

• 10.5 - baseline from the previous Node release
• 10.6.0-pre - the current (pre-)release tag

For this benchmark, we're gathering various metrics under 2 different workloads:

1. value-only: Each transaction consumes 2 inputs and creates 2 outputs, changing the UTxO set. This workload produces full blocks (> 80kB) exclusively.
2. Plutus: Each transaction contains a Plutus script exhausting the per-tx execution budget. This workload produces small blocks (< 3kB) exclusively.

Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. All runs were performed in the Conway era.

Observations​

These benchmarks are about evaluating specific corner cases in a constrained environment that allows for reliable reproduction of results; they're not trying to directly recreate the operational conditions on Mainnet.

Resource Usage​

1. 10.6.0-pre exhibits a slight shift in CPU usage. It consumes 3% less CPU time under saturation, whereas under a low submission workload it consumes 4% more.
2. Allocation rate and Minor GCs impact are significantly reduced - (~45% and ~59% depending on workload). This takes much pressure away from the garbage collector.
3. RAM usage increases significantly by 0.9GiB - 1.1GiB (15% - 17% depending on workload).
4. Observed CPU 85% spans are longer -- ~3.1 slots under value and ~1.6 slots under Plutus workload.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.

Forging Loop​

1. We can observe slight increases in Ledger Ticking and Mempool Snapshotting by 2ms each.
2. This causes a block producer to announce a new header 3ms - 4ms (or 6% - 12%) later into a slot.
3. Additionally, Adoption time on the block producer also increases by 3ms - 4ms.

The metric 'Slot start to announced' (see in attachments) is cumulative, and demonstrates how far into a slot the block producing node first announces the new header.

Peer propagation​

1. Under saturation workload only, Block Fetch duration increases by 7ms (or 2%).
2. Adoption times on the peers increase by 2ms - 3ms.

End-to-end propagation​

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

1. Cluster adoption metrics on 10.6.0-pre exhibit a small 2% - 3% increase across all centiles.
2. Under Plutus workload only, the increase becomes superlinear in 98th-100th centiles, with an extra 89ms (or 18%) required for full cluster adoption.

Conclusion​

1. The small increases in block production, diffusion and adoption metrics do not pose a performance risk to the network.
2. The restricted topology of the benchmark forces a regression in the tail end of the adoption metrics distribution to surface; in a live network, this is mitigated by a much higher number of connected peers / peer sharing.
3. The increase in RAM usage has so far not manifested on relay nodes deployed in a live network; as this is a pre-relase, the precise effect of our benchmark (exposing block producers to high pressure over extended time) is under investigation.

Attachments​

Full comparison for value-only workload, PDF downloadable here.

Full comparison for Plutus workload, PDF downloadable here.

Setup​

As part of the release benchmarking cycle, we're comparing benchmarking runs for 2 different versions of cardano-node:

• 10.4.1 - baseline from the previous Node release
• 10.5.0 - the current (pre-)release tag

For this benchmark, we're gathering various metrics under 2 different workloads:

1. value-only: Each transaction consumes 2 inputs and creates 2 outputs, changing the UTxO set. This workload produces full blocks (> 80kB) exclusively.
2. Plutus: Each transaction contains a Plutus script exhausting the per-tx execution budget. This workload produces small blocks (< 3kB) exclusively.

Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. All runs were performed in the Conway era.

Preliminaries​

1. The feature in 10.5 with major performance impact is periodic ledger metrics. This is exclusive to the new tracing system.
2. 10.5 flips the default config for PeerSharing to true; however, the recommendation is to explicitly set it to false on block producers. If not for privacy issues alone, we also found disadvantageous performance impact on block production when enabled. Hence, our benchmarks do not factor in that overhead.

Observations​

These benchmarks are about evaluating specific corner cases in a constrained environment that allows for reliable reproduction of results; they're not trying to directly recreate the operational conditions on Mainnet.

Resource Usage​

1. 10.5.0 shows a clear reduction in CPU usage - by ~30% regardless of workload type.
2. Furthermore, Allocation rate and GC impact are clearly reduced - by 27%-29% and 24%-25% respectively.
3. Heap size increases very slightly under saturation (by 1%) and decreases very slightly (by 1%) under Plutus workload.
4. CPU 85% spans are slightly shorter (~0.2 slots) under saturation, and slightly longer (~0.26 slots) under Plutus workload.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.

Forging Loop​

1. Block Context Acquisition time (prior to leadership check) is greatly reduced - from ~24ms to under 1ms.
2. Under saturation only, Ledger Ticking and Mempool Snapshotting exhibit very slight upticks (by 3ms and 2ms respectively).
3. Under Plutus workload only, Self Adoption on the forger exhibits a very slight uptick (by 3ms).
4. In summary, a block producer is able to announce a new header 20ms or 21% earlier into the slot (22ms or 43% under Plutus workload).

The metric 'Slot start to announced' (see in attachments) is cumulative, and demonstrates how far into a slot the block producing node first announces the new header.

Peer propagation​

1. Under saturation workload only, Block Fetch duration increases by 14ms (or 4%).
2. Under saturation, block adoption is slightly faster (by 3ms), while under Plutus workload it's slightly slower (by 2ms).

End-to-end propagation​

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

1. Under saturation workload, cluster adoption times on 10.5.0 are identical to those on 10.4.1.
2. Under Plutus workload, they show a moderate 3% - 5% improvement, with 7% in the 50th percentile.

Conclusion​

1. We could not detect any regressions or performance risks to the network on 10.5.0.
2. CPU usage is clearly reduced.
3. The forging loop executes faster, new header announcements happen earlier.
4. Diffusion / adoption metrics exhibit a small overall improvement and indicate 10.5.0 will deliver network performance at least comparable to 10.4.1.
5. All improvements listed above hinge on the ledger metrics feature and will materialize only when using the new tracing system. Using the legacy system, 10.5.0 performance is expected to be almost identical to 10.4.1.

Attachments​

Full comparison for value-only workload, PDF downloadable here.

Full comparison for Plutus workload, PDF downloadable here.

NB. The benchmarks for 10.5.0 extend to a potential 10.5.1 tag, as that won't include any changes with a performance impact; thus, measurements performed on 10.5.0 remain valid.

Setup​

As part of the release benchmarking cycle, we're comparing benchmarking runs for 2 different versions of cardano-node:

• 10.3.1 - baseline from the previous Node release
• 10.4.1 - the current release

For this benchmark, we're gathering various metrics under 2 different workloads:

1. value-only: Each transaction consumes 2 inputs and creates 2 outputs, changing the UTxO set. This workload produces full blocks (> 80kB) exclusively.
2. Plutus: Each transaction contains a Plutus script exhausting the per-tx execution budget. This workload produces small blocks (< 3kB) exclusively.

Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. All runs were performed in the Conway era.

10.4.1 features the UTxO-HD in-memory backing store V2InMemory of LedgerDB, which replaces the in-memory representation of UTxO entries in 10.3 and prior.

Observations​

These benchmarks are about evaluating specific corner cases in a constrained environment that allows for reliable reproduction of results; they're not trying to directly recreate the operational conditions on Mainnet.

Resource Usage​

1. On 10.4.0 under value workload, Heap size increases slightly by 2%, and 5% under Plutus workload. This corresponds to using ~170MiB-390MiB additional RAM.
2. Allocation rate and GC impact are virtually unchanged.
3. Process CPU usage improves slightly by 2% regardless of workload type.
4. CPU 85% spans are slightly (~0.37 slots) longer under value workload, and slightly shorter (~0.33) under Plutus workload.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.

Forging Loop​

1. We can observe a clear improvement in Mempool snapshotting by 9ms or 16% (2ms or 8% under Plutus workload).
2. Self-Adoption time improves by 4ms or 5% (and remains virtually unchanged under Plutus workload).
3. Hence a block producer is able to announce a new header 10ms or 9% earlier into the slot (1ms or 2% under Plutus workload).

The metric 'Slot start to announced' (see in attachments) is cumulative, and demonstrates how far into a slot the block producing node first announces the new header.

Peer propagation​

1. Under value workload, Fetch duration and Fetched to Sending improve slightly by 3ms (1%) and 2ms (4%).
2. Under Plutus workload, Fetched to Sending has a slightly longer delay - 2ms (or 5%).

End-to-end propagation​

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

1. Under value workload, cluster adoption times exhibit a small 1% - 3% improvement across all percentiles.
2. Under Plutus workload, they show a small 1% - 2% increase across all percentiles (except the 80th).

Conclusion​

1. We could not detect any regressions or performance risks to the network on 10.4.1.
2. There is a small and reasonable price to pay in RAM usage for adding the LedgerDB abstraction and thus enable exchangeable backing store implementations.
3. On the other hand, CPU usage is reduced slightly by use of the in-memory backing store.
4. 10.4.1 is beneficial in all cases for block production metrics; specifically, block producers will be able to announce new headers earlier into the slot.
5. Network diffusion and adoption metrics vary only slightly and indicate 10.4.1 will deliver network performance comparable to 10.3.1.

Attachments​

Full comparison for value-only workload, PDF downloadable here.

Full comparison for Plutus workload, PDF downloadable here.

NB. The benchmarks for 10.4.1 were performed on tag 10.4.0. The patch version bump did not include changes relevant to performance; thus, measurements performed on 10.4.0 remain valid. The same holds for 10.3.1 and 10.3.0.

Setup​

As part of the release benchmarking cycle, we're comparing benchmarking runs for 3 different versions of cardano-node:

• 10.2 - baseline from the previous release (bulit with GHC8.10.7)
• 10.3.0-ghc8107 - the current release built with GHC8.10.7
• 10.3.0-ghc965 - the current release built with GHC9.6.5

For this benchmark, we're gathering various metrics under 2 different workloads:

1. value-only: Each transaction consumes 2 inputs and creates 2 outputs, changing the UTxO set. This workload produces full blocks (> 80kB) exclusively.
2. Plutus: Each transaction contains a Plutus script exhausting the per-tx execution budget. This workload produces small blocks (< 3kB) exclusively.

Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. All runs were performed in the Conway era.

10.3.1 supports two compiler versions, which will be taken into account when comparing performance of different builds of that release.

Observations​

These benchmarks are about evaluating specific corner cases in a constrained environment that allows for reliable reproduction of results; they're not trying to directly recreate the operational conditions on Mainnet.

Resource Usage​

1. 10.3.1 exhibits a clear reduction in Process CPU usage, more prominent under value workload:
   • value workload: 10% with GHC8, and 24% with GHC9.
   • Plutus workload: 4% GHC8, and 6% with GHC9.
2. There also is a reduction in RAM usage, more prominent under Plutus workload:
   • value workload: 1% or ~54MiB with GHC8, and 6% or ~574MiB with GHC9.
   • Plutus workload: 14% or ~1.2GiB with GHC9 only.
3. Minor GCs and Allocation rate both drop on 10.3.1, more significantly under value workload:
   • value workload: 11% each with GHC8, and 24% each with GHC9.
   • Plutus workload: 3% and 1% with GHC8; 5% and 4% with GHC9.
4. Under value workload, CPU 85% spans increase by 45% with GHC8, but only by 14% with GHC9.
5. Under Plutus workload, those spans decrease by 5% with GHC8; even by 19% with GHC9.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.

Forging Loop​

1. Under value workload, several block production metrics improve clearly on 10.3.1, most prominently Mempool Snapshotting.
2. With GHC8, the improvement is 23%, with further significant improvements in Adoption time (11%) and Ledger ticking (10%).
3. With GHC9, the improvement is 27%, with further significant improvements in Adoption time (10%) and Ledger ticking (17%).
4. Under value workload, this enables a block producer to announce a header earlier into the slot, namely by 23ms (GHC8) and by 28ms (GHC9).
5. Under Plutus workload, Adoption time increases by 3ms (6%) with GHC8, but decreases by 8ms (15%) with GHC9.
6. Furthermore, there are no significant changes to the header announcement timing.

The metric 'Slot start to announced' (see in attachments) is cumulative, and demonstrates how far into a slot the block producing node first announces the new header.

Peer propagation​

1. Under value-only workload only, we observe an increase in Block Fetch duration: 7ms (2%) with GHC8, and 23ms (7%) with GHC9.
2. Block adoption times on the peers improve clearly: 11ms (12%) with GHC8, and 12ms (14%) with GHC9.
3. Under Plutus workload, however, similarly to the block producer, adoption times increase by 3ms (6%) with GHC8, but decrease by 7ms (13%) with GHC9.

End-to-end propagation​

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

1. Under value workload, cluster adoption times on 10.3.1 are largely unchanged.
2. With GHC8, there are 5% and 3% improvements in the 50th and 100th centiles; with GHC9, there's a small 3% improvement in the 50th centile.
3. Under Plutus workload, with GHC8, there's a moderate 6% increase in cluster adoption times in the 100th centile.
4. With GHC9, however, there's a small 2% improvement in all but the 100th centile.

Conclusion​

1. For 10.3.1 we could not detect any performance risks or regressions.
2. Improving resource usage was a stated goal for the 10.3 release; this could be confirmed via measurements for CPU and RAM usage as well as CPU spikes.
3. 10.3.1 achieves network performance comparable to 10.2.1 using clearly less system resources - for both compiler versions.
4. Several key metrics improve on 10.3.1: Block producers announce a new header sooner into the slot; we observe lower adoption times (GHC9 only).
5. The GHC9.6.5 build has demonstrable performance advantages over the GHC8.10.7 build; especially the Plutus interpreter seems to gain considerably from using GHC9. For those reasons we now recommend GHC9.6.x for production builds.

Attachments​

Full report for value-only workload, PDF downloadable here.

Full report for Plutus workload, PDF downloadable here.

NB. The benchmarks for 10.3.1 were performed on tag 10.3.0. The patch version bump did not include changes relevant to performance; thus, measurements performed on 10.3.0 remain valid.

Setup​

This report compares benchmarking runs for 3 different settings of the Plutus memory execution budget:

• loop-memx1 - current mainnet memory execution budget
• loop-memx1.5 - 1.5 x current mainnet memory execution budget
• loop-memx2 - 2 x current mainnet memory execution budget

For this comparison, we gather various metrics under the Plutus workload used in release benchmarks: Each block produced during the benchmark contains 4 identical script transactions calibrated to fully exhaust the memory execution budget. Thus, script execution is constrained by the memory budget limit every case. The workload produces small blocks (< 3kB) exclusively.

Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. Node version 10.2 was used, built with GHC8.10.7.

Observations​

Resource Usage​

1. Scaling the memory budget impacts Allocation Rate and Minor GCs. 1.5 x the budget results in rises of 5% and 6% respecetively; for doubling the budget the corresponding rises are 10% and 11%.
2. Those increases seem to correlate linearly with raising mem budget.
3. The effect on CPU usage is almost negligible: a 1% (or 3%, for doubling the budget) increase of Process CPU.
4. The Node process RAM footprint is unaffected.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.

Forging Loop​

1. Scaling the memory budget has significant impact on block adoption time only.
2. Scaling by factor 1.5 leads to a 14ms (or 25%) increase, whereas factor 2 leads to 28ms (49%).

Peer propagation​

1. Same as on the block producer, scaling the memory budget has significant impact on block adoption times only.
2. Scaling by factor 1.5 leads to a 15ms (or 26%) increase, whereas factor 2 leads to 28ms (48%).

End-to-end propagation​

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

1. 1.5 x the memory budget results in a slight increase of 19ms - 22ms in cluster adoption times (4% - 5%).
2. 2 x the memory budget results in a moderate 27ms - 34ms increase (5% - 7%, with 9% in the 50th centile).

Conclusion​

These measurements outline the headroom for raising the memory budget, along with the expected performance impact:

1. Block adoption time is the only metric that's affected significantly, increasing both on the forger and the peers by the same extent.
2. These increases seem to correspond linearly with the raising the memory budget. This gives excellent predictability of performance impact.
3. Expectedly, more allocations happen; we can observe the same linear correspondence here as well.
4. It has to be pointed out that block diffusion is only slightly affected by changing the execution budget: Due to pipelining, announcing and (re-)sending a block precedes adoption in most cases.
5. As such, regarding absolute cluster adoption times, measurements taken with either budget adjustment do not exhibit performance risks to the network. They do illustrate, however, the performance cost of those budget adjustments.

Attachment​

Full report PDF downloadable here.