95 posts tagged with "consensus"

High level summary​

On the UTxO-HD front we reached another important milestone, where we handed over the implementation for benchmarking and testing. We are working on addressing the first issues that were found during the integration tests. On the Genesis front we are finalizing the argument that a checkpoint is not necessary for the initial release, including supporting to support said argument. We are also working on the new node release, and several improvements to our code-base.

UTxO-HD​

• We addressed the issues that came from the benchmarking and testing phase (see #314).
  • We fixed the operational-certificate update exception (#305).

Tech debt​

• We elaborated an inventory of the symbols used by downstream clients in an effort to clearly define the Consensus API, which will help managing changes and their impact to the downstream clients (see this branch).
• We extracted the SOP modules to a separate package as part of our effort to reduce the Consensus API surface area (see this branch).
• We took the first step towards improving how the Cardano hard-fork block is instantiated when running a node (see #275).
• We decreased the flakiness for our test suite (#284).
• We incorporated and merged the pull-request on query versioning (#273).
• We started working on new performance-based regression tests for mempool operations.

Support​

• Nick Frisby continues with his engineer responsibilities. Notably getting the Consensus release out for node pre-release 8.3. Nick also began integrating it into downstream branches (cardano{-api,-cli,-node}). The Node team has now taken over, since Nick had progressed up to the point where they needed to start making new design decisions. Related PRs:
  • #283
  • #301
  • #304
  • #308

High level summary​

This week the Consensus team worked on integrating the latest node release into the UTxO HD branch. Our ad-hoc benchmarks in which we sync a node from scratch showed that the LMDB backend uses around 8GB of memory. We also released new versions of fs-sim and fs-api, and released strict-checked-vars.

UTxO-HD​

• Rebased UTxO-HD on top of node 8.2 (issue)
• Address new Maybe translations for UTxOs in Conway 1.6. (PR).
• Performed ad-hoc benchmarking syncs with both implementations. The LMDB backend uses around 8GB of memory.
• Fixed an infinite loop on the mempool tracing code that was preventing the node from shutting down gracefully.
• Fixed an infinite loop on the ledger state query for traversing queries that prevented the node from running QueryUTxOByAddress (PR).
• Measured the speed of QueryUTxOByAddress under different queryBatchSize values. There is a significant performance degradation for these queries when using the UTxO-HD version. Using the mainnet UTxO set at slot 90 million, querying an address took 40 seconds using the in-memory backend and 90 seconds using the LMDB backend, whereas the baseline took 7 seconds. We need to investigate if we can improve this situation or whether an external service that runs alongside the node is a better solution.
• Refactored and implemented ledgertable-related classes for the general HardforkBlock.

Genesis​

• Frisby and Esgen continue to engage with the Researchers on grinding against the Genesis design.

Fostering collaboration​

• Drafted a document explaining versioning of local state queries #273.

Support​

• Frisby is the release engineer this cycle.
• Successfully created work-in-progress ouroboros-consensus and cardano-api commits that integrate the 2023 Aug 7 tip of cardano-ledger. This will require a release of ouroboros-consensus, which hasn't happened yet.

High level summary​

The db-sync team confirmed that the adaptor module the Consensus team provided for UTxO-HD integration, shows no performance degradation. Other clients who wish to integrate with UTxO-HD without using a on-disk storage could use that same approach for now. We also focused on completing tracing support for UTxO-HD in cardano-node. Currently we are working on UTxO-HD documentation, both for downstream consumers and for the general public; and generalizing the implementation further.

The team working on Genesis decided on a Genesis-motivated change of the epoch structure with the researchers, wrote a self-contained description of the statistical model for historical Genesis windows and eclipse durations, and merged two small improvements to the ChainSync client as part of the onboarding efforts for the team that will implement Genesis.

The Consensus team also investigated an issue on sanchonet (the Conway testnet) manifesting in long syncing pauses; identified a ledger bug which was promptly fixed by the Ledger team.

Genesis liaising​

Improvements to the ChainSync client:

• #222.
• #233.

High level summary​

This week the team working on UTxO-HD discovered a space leak in the peer metrics code. This was communicated to the Networking team who has a proposed fix. The ad-hoc benchmarks that the team ran using a local immutable DB server showed good memory and time performance. We still have to check the performance on a memory constrained machine.

The team working on the Genesis design started onboarding the team of engineers that will implement the new Genesis protocol. This team is also finalizing the statistical model for historical Genesis feasibility.

On the support front, the team drafted an information exchange requirement (IER) for the Networking team to safely and efficiently control peer load.

High level summary​

During the past two weeks the team working on the Genesis implementation continued to engage with the researchers, which resulted in various simplifications of the correctness argument for the historical Genesis window. They also decided on an approach for a syncing node to decide that it is (no longer) caught up. This functionality was requested by the networking team.

The team working on the UTxO-HD implementation ran ad-hoc benchmarks that showed performance issues, which are being investigated. They also merged several improvements required for the first UTxO-HD release, and added a package for easing integration with other downstream components.

Regarding our support activities, we integrated the latest Ledger changes into Consensus in preparation for release 8.2 of node.

Genesis​

• We continued to engage with the researchers on our probabilistic model for historical Genesis window, resulting in various simplifications that make the correctness argument more clear while not being excessively conservative.

• We decided on an approach of how to implement functionality requested by the Networking team; namely, how a syncing node can safely conclude that it is (no longer) caught up. Certain parameters are still subject to discussion with the researchers, and we have still have to agree on a concrete API for this functionality with the Networking team.

UTxO-HD​

• We merged the last of the PRs that were part of UTxO-HD improvements for version 0.1: expose UTxO-HD configuration options in the node, refactor ledger tables, and expose a method of computing the UTxO set size.
• We added a new "legacy" cardano block in a new ouroboros-consensus-cardano-legacy-block package that should ease the transition for some downstream packages to UTxO-HD, like db-sync. This is really only useful for downstream packages that use the parts of consensus that don't involve the storage components, in which case we can largely ignore ledger tables. Ignoring ledger tables could also make functionality like block (re-)application more performant for the legacy Cardano block as compared to the actual (UTxO-HD compatible) Cardano block.
• We performed ad-hoc benchmarks of the UTxO-HD implementation, observing a regression in sync speed in the LMDB implementation as well as a regression in memory usage on the in-memory implementation. We are investigating this.