Skip to main content

· 5 min read
Damian Nadales

2023-04 -- 2023-06​

Main achievements​

UTxO HD​

  • We finished a major prototype refactoring, which includes:
    • A better and finer grained DB lock mechanism.
    • Elimination of race conditions.
    • Support for configuring batch query size and flushing rate. This is crucial to allow node users to tweak performance.
    • Architectural simplifications and performance improvements.
  • We implemented a new package to support db-sync integration with UTxO-HD.
  • We ran another set of ad-hoc benchmarks:
    • We uncovered a performance regression on the Network component when using GHC-9.2/9.4.
    • The synchronization and replay speed are as expected.
    • However, we uncovered memory consumption issues (see figure below).
      • The in-memory backend is consuming more memory than the baseline.
      • The LMDB backend shows an unexpected memory usage peak.
      • Investigation on these issues is ongoing.
  • We integrated the latest changes in main branch.
    • This required a re-design of the mempool to include the mempool fairness improvement.

Genesis​

  • The Genesis work for this PI focused on an high-priority issue from the IOG Researchers' feedback on the proposal.
    • This particular question was not anticipated when the Q2 PI was planned.
    • As a result, the chain generators work, the ChainSync Jumping performance work, and the Genesis node prototype work were deprioritized.
    • That work has accordingly been rolled over into the Statement of Work for the first Genesis vendor work package.
  • The IOG Researchers' feedback on the design was very valuable. It had two primary effects.
  • Outcome 1: We re-introduced distinct behaviors when the node is "syncing" versus when it is "caught up".
    • This eliminated a DoS vector introduced by the proposal, instead of having to argue that it was well-mitigated.
    • The additional design complexity is relatively small.
  • Outcome 2: The issue that was unanticipated is whether the Cardano chain is consistently dense enough to rely on Genesis without any checkpointing.
    • The determination so far is that---assuming the adversary never controls more than four of the seven genesis keys---the most vulnerable segment is in the pure Praos era.
    • All the preceding windows are significantly more robust, including the entire Byron and Transitional Praos eras.
    • Thus checkpointing is not necessary for the initial Genesis release, though it still may be a reasonable addition later.
    • The primary invention was a model for bounding how much benefit the adversary's long-range attack could possibly gain from Praos's natural short forks.
  • Relevant questions that the IOG Researchers are still assessing.
    • These do not block the Genesis implementation, but do affect the ultimate values of specific parameters.
    • Question 1: what is the upper bound on the duration of an eclipse that a healthy Praos node will survive?
    • Question 2: what is the upper bound on how much grinding can improve the adversary's leader schedule within some Genesis window?

Support​

  • We performed an analysis on number of file descriptors used by Consensus, this information can be used by the node operators to check if the number of file descriptors they want to support are enough, thus improving the user (eg node operator) experience.
  • We implemented a mempool fairness improvement, by which transactions are guaranteed to be processed irrespective of their size.

Technical debt​

  • We fixed a bug in followers logic, which was discovered by our QuickCheck property tests.
  • We created an immutable DB server. This tool allows to serve blocks from the immutable DB to a node that connects to it. This has a remarkable value for testing and benchmarking purposes. For instance, by using this component, we can benchmark the performance of different aspects of Consensus, such as syncing from scratch, without adding Network interference in the performance results.
  • We created a db-truncater tool, which can be used in disaster recovery and benchmarking scenarios.
  • We created a benchmarks comparison tool that we plan on using for comparing the performance of two Consensus releases. This will allow us to catch performance regressions early on in the process, before they make it to the node (and show in the system level benchmark tests), thus greatly saving development costs. As an example, the graph below shows the performance improvements introduced by the Ledger team in version 0.6.0.0 of Consensus wrt version 0.5.0.0.

Fostering collaboration​

  • We released fs-sim as open-source repository. This lowers the barrier to entry for external contributors, which will indirectly benefit the Cardano project.
  • We migrated the consensus code to a new repository, splitting it from the ouroboros-network repository. This will save development effort for both the Network and the Consensus teams, since there will be less interference (for instance when making releases).
  • We made several improvements to our release processes, which will translate in time savings. As an example, we went from 16 to 4 packages, which makes the release process simpler and smother. Our release process now makes it easier to align versions and make releases (both for us and for our downstream users).
  • We added an explanation of the hardfork-combinator forecast horizon, that will benefit not only our team, but future external contributors.

Next steps​

UTxO HD​

Genesis​

  • We will regularly liaise with the vendor(s) satisfying the Genesis Statement(s) of Work.

· 6 min read
Marcin Szamotulski

2023-04 - 2023-06​

Main achievements​

Eclipse Evasion​

We finalised the design of eclipse evasion and implemented its mechanism which relays on connectivity to big ledger peers. Big ledger peers are the largest ledger peers which accumulate 90% of stake (currently there are less than 1000 of them). The outbound governor has new targets for known, established and active big ledger peers which work in a similar way that such targets work for ledger peers. The ouroboros-network#4662 PR is currently in review.

As part of this work we also identified a bug which would prevent a node to connect to itself. Such connections are not easily detectable and are expected to be dropped by the churn mechanism, nonetheless they should not be buggy. The failure was discovered thanks to our e2e simulation of diffusion using io-sim & property based testing.

The PR also refactors the heart of the ouroboros-network interface reducing technical debt that would otherwise accumulate.

We also identified a possible improvement in the churn mechanism, which will be implemented in Q3. Churn needs to await for peers to terminate, we can improve the synchronisation. [ouroboros-network#4617]

Ecosystem P2P Deployment Progress​

We reached 50% of stake in hands of SPOs who run at least on P2P relay. Now also Emurgo and CF are running some P2P relays. Also 20% of IOG relays are running in P2P mode.

P2P Progress

Peer Sharing​

We implemented bootstrapping for peer sharing (also known as light peer sharing). New downstream (inbound) peers are now added to the known peers of the outbound governor. Together with peer sharing this allows for non registered relays to propagate through the network. ouroboros-network#3596

Please note that peer sharing is disabled by default and is not considered safe until Bootstrap Peers (see below) or Genesis is implemented.

Diffusion (P2P)​

  • We designed a feature which will reduce the load on IOG relays (in future also run by CF & Emurgo). The feature consists of two parts. A new source of peers called bootstrap peers (obtained from via an https request), the ability to switch from bootstrap peers to ledger peers if the node is synced (we are collaborating with the consensus team on the interface Bootstrap Peers IER). This feature will be completed in Q3. ouroboros-network#4530

  • We published a blog post about P2P design & implementation.

  • Karl Knutsson (CF) fixed an issue observed on a relay with a lot of outbound connections: ouroboros-network#4559.

  • We merged changes which allow the consensus layer to start / stop block forging thread. This will allow to deploy P2P block producing nodes which serve as a live backup node. ouroboros-consensus#140

  • We fixed a few bugs in local root peers DNS resolution service: ouroboros-network#4583, ouroboros-network#4571.

  • We limited concurrency of DNS name resolutions: ouroboros-network#4596.

  • Galois Inc implemented query option for Handshake: ouroboros-network#4256.

  • We fixed handshake query timeout: ouroboros-network#4608.

  • We implemented warm valency for local root peers. This can help when using DNS names in local root peers which resolve to many IP addresses. ouroboros-network#4575

  • We merged handshake changes which allow query protocol versions. Thanks to James Parker from Galois Inc.: ouroboros-network#4256, cardano-cli#30.

Other Improvements & Developments​

CDDL​

  • We added node-to-node and node-to-client CDDL specs / tests for encoding of NodeToNodeVersionData and NodeToClientVersionData.

  • We clarified an inconsistency between CDDL spec and implementation which is highly polymorphic. We designed and implemented a fix for tx-submission and local-tx-submission mini-protocols. Specs for other mini-protocols will be improved at a later stage. ouroboros-network#4580

Cardano Ping​

IOSim​

Typed Protocols​

Cardano Client​

  • We fixed a bug in cardano-client-0.1.0.2 release which results in clients (e.g. db-sync) negotiate an experimental protocol version.

Technical debt​

CI improvements​

GHC 9.4 & 9.6​

We made all repositories under our control compile with ghc-9.4 and ghc-9.6 which includes ouroboros-network, io-sim, typed-protocols and Win32-network.

Next steps​

We will continue towards our aspirational roadmap.

  • We will continue reviewing eclipse evasion.
  • As ouroboros-consensus#140 was merged, we are making progress towards releasing P2P on block production nodes. We hope to analyse performance regression on such nodes observed on the benchmarking cluster. roadmap-3887
  • We are also focused on roadmap-3969. Note that it was expanded in Q2.

· One min read
John Lotoski

2023-04 - 2023-06​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, main SRE achievements for this quarter include:

  • Expanding the darwin CI cluster and adding aarch64 builder support
  • Adding bare metal capability to bitte clusters
  • Creating a devx-ci cluster containing a Hydra build server and Linux build farm which is intended to replace Cicero functionality
  • Creation of pool performance analysis queries and scripting
  • Migration of testnet metadata server to cardano-world
  • Cardano shelley qa migration to cardano-world
  • Cardano sanchonet environment spin up to test Conway era functionality
  • Mainnet relay conversion to p2p topology usage is progressing with 50% of mainnet relays now using p2p topology and networking feature

Next steps​

  • Continue with the conversion of mainnet to using p2p topology

· 4 min read
Michael Karg

2023-07 - 2023-09​

Main achievements​

  • Release benchmarking
  • Developing and running UTxO-HD benchmarks - in-memory flavour
  • P2P benchmarks, facilitating rollout
  • Production-readiness of the new Nomad cluster has been reached
  • Optimization of and introspection capability for the new tracing system
  • GHC9 performance investigation (and possible remedy)
  • Conensus QTAs: first real-world application of prototype

Release benchmarking​

Ongoing release benchmarking is a crucial safeguard to cardano-node's release cycle from a performance perspective. We've performed and analyzed benchmarks for node versions 8.2.x to 8.5 throughout Q3.

UTxO-HD benchmarks​

Targeting a specific new feature in benchmarks requires development effort and fine-tuning the machinery. In Q3, we achieved that for the in-memory flavour of UTxO-HD, enabling benchmark delivery.

P2P benchmarks​

In Q3, we performed additional P2P benchmarks to facilitate the comprehensive rollout of that feature.

New nomad cluster​

The new hardware cluster for benchmarks, which is controlled through the new nomad backend, has received various rounds of validation and adjustments in Q3 - in addition to finalizing integration with the rest of our pipeline. The confidence in metrics gathered on the cluster is now sufficient for us to consider it ready for production use.

New tracing system​

Our new tracing system has received various rounds of optimization in Q3. We could verify in our benchmarks that it is roughly on par with the legacy system while offering a richer feature set and greater flexibility.

Additionally, in Q3 we equipped the system with an introspection capability. This is now used for generating end user documentation that stays in-sync with definitions in code, and for automated consistency checking of the entire system.

GHC9 performance​

In Q3, a joint investigation with DevX into GHC9's behaviour revealed where and how GHC9 misses opportunities for optimization of generated code. This led to an approach to annotate our codebase accordingly to re-enable those optimizations - which is still being validated.

Consensus QTAs​

In collaboration with Consensus and DevX, we advanced the Consensus QTAs prototype capturing ledger operations' performance characteristics. It's now applicable, and being applied, to a real-world task - namely gathering evidence of the effect of aforementioned changes allowing for performant GHC9 builds.

Next steps​

Benchmarking:​

In Q4, the focus will be on:

  • facilitating the next mainnet release
  • benchmarking runs in the Conway era
  • developing benchmarks / workloads for Conway-exclusive actions
  • implementing a specialized benchmark setup for the UTxO-HD on-disk variant
  • developing new Plutus benchmarks to safeguard Plutus V3
  • benchmarks regarding the rollout of P2P

Performance​

For certain blocking performance issues we've located the cause, or even found a solution in a cross-team effort. In Q4 we'll advance that work to ensure the ongoing release cycle for mainnet, as well as make GHC9 become a viable release platform.

New tracing system​

For the new tracing system, we'll finalize optimization - current results are already on par with the legacy system. Furthermore, we will finish up comprehensive documentation, as well as description of a recommended setup, for which we can provide initial support.

UTxO-HD monitoring​

We'll augment our analysis pipeline so it can process monitoring data from UTxO-HD nodes connected to mainnet in a meaningful way.

Nomad backend​

From Q4 on, this backend will be in production use. We plan on adding various UX and flexibility improvements, and further fine-tuning some profiles for nomad.

Workbench​

We will prepare for a future move of our performance workbench into a separate project. This entails restructuring, refactoring and reimplementation of certain few components that currently assume to always be in sync with cardano-node.

Consensus component QTAs (co-development)​

In Q4 there will be ongoing work with and support for the existing prototype. We plan to identify a fixed set of input data that yields results of high informative value, and to formalize the process to a point that enables future automation.

· One min read
John Lotoski

2023-07 - 2023-09​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, main SRE achievements for this quarter include:

  • Completion of mainnet relay networking conversion to p2p topology
  • Cardano sanchonet environment respins for testing new cardano-node pre-release Conway era functionality
  • Stabilization of cardano-explorer in cardano-world using high IOPS bare metal machines
  • Creation of a nix content addressed packages repository, capkgs:
    • To provide lightweight release binaries thereby avoiding sluggish nix flakes and devShells
  • Creation of a cardano performance benchmarking cluster, cardano-perf:
    • To replace legacy cluster benchmark tooling
  • Creation of a cardano cluster composition repository, cardano-parts:
    • For enabling multi-cluster, multi-role cardano network deployments
  • Creation of a cardano testnets repository, cardano-playground:
    • Utilizing cardano-parts for testnet deployments
  • Creation of a sanchonet demo repository, sanchonet-demo:
    • Utilizing cardano-parts for fast sanchonet test environment and demo purposes

Next steps​

  • Continue with migration of cardano-world testnets to cardano-playground
  • Proceed with spinup of mainnet p2p bootstrap cluster
  • Scale down mainnet non-p2p legacy cluster at the appropriate time

· 3 min read
Damian Nadales

Areas of focus​

IssueStatus
Implement legacy mode for UTxO-HD to keep baseline performance✅ Done
Assist mainnet node release with initial Conway capabilities✅ Done
Assist with test, benchmark, and improvements to CIP 1694✅ Done
Assist with P2P IOG relay network shut down✅ Done
Assist with repo transfer to Intersect✅ Done
Support vendors to deliver contracts✅ Done
Operation serenity Q4 2023✅ Done

Highlights​

Implement legacy mode for UTxO-HD to keep baseline performance​

  • ✅ We managed to run a UTxO-HD capable node in legacy mode, maintaining the baseline memory usage while keeping all the ledger state in memory.
    • While the legacy mode is not production-ready (it requires further integration and testing), it remains as a plan B should the need arise to release UTxO-HD if our stakeholders so demand it.
  • ✅ We pivoted to redesigning the Ledger DB API because:
    • This is needed for integrating the LSM-tree backend.
    • The redesign opened the possibility of implementing an in-memory backend that would keep the same performance and resource requirements as the baseline version (which needs to be confirmed by benchmarks).
  • ✅ We created a more general Ledger DB API.
  • 🛠️ We are integrating (into the feature branch) the existing Ledger DB implementations with the new API.
  • 🛠️ We are implementing the new in-memory backend.

Assist mainnet node release with initial Conway capabilities, test, benchmark, and improvements to CIP 1694​

  • ✅ We recognized that Conway introduces a new challenge in the versioning of NTC queries, and we resolved it (see 864 and 4770).

Assist with P2P IOG relay network shut down​

  • ✅ We created a prototype for the pre-Genesis State Machine for bootstrap peers, which is currently under test (see this PR).

Assist with repo transfer to Intersect​

  • ✅ We transferred the ouroboros-consensus repository to the Intersect GitHub organization.

Support vendors to deliver contracts​

  • Genesis
    • ✅ Interacted with the Consensus team and addressed resulting feedback on past deliverables.
    • ✅ Finished implementation of the testing infrastructure of Genesis
    • ✅ Started to refine the Proof of Concept demo into an actual implementation of the core components of the Genesis design.
  • 💾 LSM-tree implementation. Well Typed:
    • ✅ Finished the design of the public facing API.
    • ✅ Defined the LSM-tree database file-type formats.
    • ✅ Implemented property and model-based tests.

Operation serenity Q4 2023​

  • 🎉 We welcomed our newest team member @RenateEilers and assisted with her (ongoing) onboarding.
  • ✅ We implemented a simplification in the ChainSync mini-protocol that is also a step towards Ouroboros Chronos.
  • ✅ We added tests to check Consensus emits valid CBOR, which prevents the generation of invalid binary encoding.
  • ✅ We established and implemented an interface between Consensus tooling and P&T tooling, which constitutes a step towards incorporating component level benchmarks in our development process.

· 3 min read
Michael Karg

2023-10 - 2024-01​

Main achievements​

  • Release benchmarking, leading up to next mainnet release
  • Conway benchmarking of existing Babbage workloads
  • P2P benchmarks, validating viability as default topology
  • Added basic PlutusV3 capability of our tooling
  • Publication of benchmarking reports accompanying a mainnet release
  • GHC9 performance investigation
  • Finalized and validated all optimizations for the new tracing system
  • New Nomad benchmarking cluster: production use
  • Adjustment of our infrastructure to cover the migration to IntersectMBO
  • Conensus QTAs: prototype developed into alpha-stage benchmark
  • Successful on-boarding of a new team member

Release benchmarking​

We've performed and analyzed benchmarks for node versions 8.6.x to 8.7.3, which is projected to be the next mainnet release, throughout Q4. Along that way, we have identified, located and handled all performance blockers.

Additionally, we've started publishing benchmarking reports here on Cardano Updates. The format is meant to increase transparency and provide insight into those measurements that accompany mainnet releases - demonstrating the absence of performance regressions and development of specific metrics over time.

Conway benchmarks​

Furthermore, we've done first ever benchmarking of the Conway ledger. To that end, we've ported our Babbage workloads to Conway for immediate comparability. Fortunately, we've have not found any performance regression in the Conway ledger.

P2P benchmarks​

In Q4, we've validated P2P topology to be viable as default for both relay and block producer nodes. As a consequence, we've switched to P2P topology for benchmarking baselines ourselves.

GHC9 performance​

In Q4, evaluation of GHC9.2's and GHC9.6's optimizer in the context of the Cardano code base has been completed. Eventually, GHC9.6 has shown itself to be much more suitable from a performance perspective. We're convinced that with a few select annotations in the code, GHC9.6's optimizer can produce a result on par performance-wise with GHC8.10 - which just was a great release in that regard. With GHC9.2 unfortunately, the changes would have to be more invasive - and thus more time-consuming.

New nomad cluster​

We’ve moved the new Nomad cluster into production use and established new baselines for each workload on it. Additionally, we’ve shut down the legacy cardano-ops benchmarking cluster, and archived all raw data from it.

Consensus component QTAs​

We’ve developed the existing prototype into an automatable, self-contained benchmark called beacon, as well as systematized workloads and run structure for it. Moreover, we’ve demonstrated usefulness of the metrics, their reproducibility, and identified domains viable for QTAs with system-level benchmarks.

New team member​

We're happy to welcome a new joiner to our team! We've successfully onboarded him in Q4; he has taken over the cardano-tracer service - the node-external component of the new tracing system - and has already landed several valuable contributions.

· One min read
John Lotoski

2023-10 - 2023-12​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, main SRE achievements for this quarter include:

  • Cardano-parts support was added for cardano-db-sync, cardano-smash, cardano-faucet, cardano-metadata, grafana monitoring along with a number of other features

  • Completed migration of testnets from cardano-world to the cardano-playground cluster

  • Completed migration of the cardano book from cardano-world to the cardano-playground cluster

  • Completed migration of pools from cardano-ops to the cardano-mainnet cluster

  • Creation of a mainnet p2p bootstrap cluster

  • Cardano sanchonet environment respins during the quarter for testing new cardano-node pre-release Conway era functionality

  • All environments were upgraded to cardano-node 8.7.2 or 8.7.3 by the end of the quarter

  • Completion of a govtool backend deployment for Voltaire chain testing

  • Creation of a cardano-monitoring repository, cardano-monitoring:

    • A new repository enabling agile deployment of EC2 monitoring servers, compatible with OpenTofu grafana and mimir providers

Next steps​

  • Scale down the mainnet non-p2p legacy cluster

  • Add deployment support for new network services, such as Mithril

  • Continue cardano-parts and operations improvements

· 2 min read
John Lotoski

2024-01 - 2024-03​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, main SRE achievements for this quarter include:

  • All cardano release environments, including preview, preprod, mainnet legacy and mainnet new clusters were upgraded through cardano-node releases of 8.7.3, 8.9.0 and finally into 8.9.1 by the end of March

  • All cardano pre-release environments, including sanchonet, private chain, and shelley-qa clusters were upgraded through cardano-node releases of 8.7.3, 8.8.0-pre, 8.8.1-pre, 8.9.0 and finally into 8.9.1 by the end of March

  • Sanchonet and private chain environments were both re-spun once each during this quarter to support new pre-release versions of cardano-node in the Conway era

  • Cardano-parts added a cardano-db-sync process-compose stack for each environment

  • Cardano-parts added a cardano-node process-compose stack for each environment

  • Cardano-parts added enhancements for topology related nixos modules and functions to accomodate new bootstrapPeer functionality, new topology attributes and increased complexity network deployments.

  • Cardano-parts added support for mithril signers integrated with block producers and a mithril-signer-verifier service for monitoring

  • Sanchonet, preview, preprod and mainnet IOG block producers are now signing mithril certificates

  • Cardano-parts added support for mithril clients in nixos cardano-node systemd service, process-compose job stacks and nix cardano-node entrypoint, all of which also require any mithril snapshot to be signed by a trusted IOG pool prior to use

  • Cardano-parts added ip integration tooling, so that similar to other deployer tools like nixops, nixosConfigurations possess ip information which can be used in module configuration

  • Cardano-db-sync snapshots server migration from legacy mainnet cluster to new mainnet cluster with a rewrite of the snapshot service was completed

  • Cardano metadata server migration to Cardano Foundation was completed

  • BlockPerf, a cardano-node performance monitoring tool, was integrated into the new mainnet cluster relays

  • Cardano-node bootstrapPeer functionality was added with node 8.9.x, requiring effort to align nixos service module code between cardano-node nixos services, iohk-nix topology generation, cardano-ops legacy code, and cardano-parts module compatibility as well as feature test under various edge cases

  • Cardano-playground added govtool backend support for private chain voltaire testing team

Next steps​

  • Add support for the new cardano-node metrics system

  • Add IPv6 cardano-parts support

  • Extend govtool frontend and backend to a process-compose stack

  • Adapt network spin-up tooling for the new create-testnet-data cardano-cli command

  • Continue cardano-parts and operations improvements

· 2 min read
John Lotoski

2024-04 - 2024-06​

Main achievements​

In addition to ongoing general maintenance and support of cardano environments, main SRE achievements for this quarter include:

  • All cardano release environments, including preview, preprod, mainnet legacy and mainnet new clusters were upgraded through various cardano-node releases of 8.9.2, 8.9.3, 8.9.4, 8.12.1, and finally into 8.12.2 by the end of June

  • Cardano pre-release environments additionally iterated through pre-release upgrades of 8.11.0-pre, 8.12.0-pre, and finally into 8.12.2 by the end of June with the exception of sanchonet which remains pinned at 8.11.0-pre until the next respin to support node version 9.0.0 or greater

  • Sanchonet environment was re-spun two times for pre-release Conway testing of cardano-node versions 8.10.0-pre and 8.11.0-pre respectively

  • Private chain environment was re-spun three times to support fast epoch Conway testing

  • Cardano-playground and cardano-mainnet repos have added ten operations oriented documents for knowledge transfer

  • Block producers which participate in mithril signing will now produce metrics and can have them scraped with the default metrics agent

  • A cluster spin-up job to utilize the new cardano-cli create-testnet-data sub-command was created

  • A nixosModule, dashboards and alerts were added supporting the new cardano tracing system

  • Many new operations scripts and features were added, including a template diff and patch recipe to pull the latest cardano-parts improvements to consuming repositories more easily

Next steps​

  • Finalize support for the new cardano-node tracing system once the service is rewritten for general consumption

  • Add IPv6 cardano-parts support

  • Extend govtool frontend and backend to a process-compose stack once govtool is publicly buildable again

  • Continue cardano-parts and operations improvements