Skip to main content

50 posts tagged with "consensus"

View All Tags

· One min read
Damian Nadales

High level summary​

This week the consensus team finished the UTxO-HD prototype refactoring. We are now working on improving the DB lock mechanism to improve performance. We also introduced several improvements to the file system abstraction and simulation layer (fs-sim), which culminated in the release of fs-sim-0.1.0.0 and fs-api-0.1.0.0 to CHaP.

On the Genesis front we distributed the updated Genesis design document, soliciting feedback from Networking Team and IOG Researchers. We also opened up a PR for the adversarial leader schedule QuickCheck generator, which is being reviewed.

On the support front, we got a new Consensus version that can use different fundamental VRF crypto primitives for Babbage and Conway eras.

On the tech debt front we fixed an bug in the followers logic, which was discovered by our QuickCheck property tests.

· One min read
Damian Nadales

High level summary​

This week the consensus team continued working on the refactoring of the UTxO HD prototype, and design and testing of Genesis. We also extracted the fs-sim package, which provides a file-system abstraction layer that can be used for testing and simulation. This makes the Consensus code base smaller, while providing a package that the community can reuse and contribute to. We also fixed a failing property test related to iterators. We are also working on mempool and VRF improvements.

Low-level details​

· 3 min read
Damian Nadales

High level summary​

The Consensus team continued working on refactoring and improving the UTxO-HD prototype, and introducing improvements to the lmdb related packages. In particular we identified an opportunity to gain performance by handling locks in a more optimal way.

On the Genesis front, we sketched a mitigation for an issue that PNSol and Researchers caught. We also came op with a road map for not only testing the Genesis prototypes, but also for enriching the tests we already have.

Regarding technical debt, next to some minor improvements, we created component-level micro-benchmarks for adding transactions to the mempool. The results of these benchmarks will be published in the ouroboros-consensus web page.

We also finished moving the Consensus documentation to the ouroboros-consensus repository, released ouroboros-consensus 0.3.0.0, and reduced the time GitHub actions take in ouroboros-network.

Workstreams​

UTxO HD Prototype​

We continued working on refactoring and improving the UTxO-HD prototype. As a result of the first round of sytem-level benchmarks, we identified an opportunity to optimise the way we handle locks to improve performance (#4393).

Also, we introduced several improvements to the lmdb related packages:

Genesis​

We sketched out a mitigation of the issue that PNSol and Researchers caught in the Genesis design.

We came up with a road map for testing the Genesis prototypes, including early milestones that are applicable to today's master branch, ie tests that are useful before Genesis, and that will be nicely enriched when we do add Genesis.

We developed the aforementioned tests, specifically a QuickCheck generator for the Honest leader schedule and one as-aggressive-as-possible Adversarial leader schedule that together satisfy the Praos properties that the Consensus design takes as invariants.

We investigated why the improved ChainDB queueing implementation behaves differently in the baseline compared to the prototype, and we are close to having a full picture of how the Consensus components interact during bulk sync.

Technical debt​

We created component-level micro-benchmarks for adding transactions to the mempool. We plan on extending this to more mempool actions and different types of blocks. We store the benchmark data to make it available to the GitHub action that publishes the benchmarks results.

Other minor improvements include:

  • Removal of Test.Util.Classify in favour of Test.StateMachine.Labelling.
  • Addition of -Wunused-packages to the default ghc-options for Consensus packages.

Fostering collaboration​

We finished moving the Consensus documentation from ouroboros-network to ouroboros-consensus, in preparation for migrating the code to the latter repository.

Support​

We released ouroboros-consensus 0.3.0.0.

We reduced the load in the ouroboros-network GitHub actions, thereby reducing the time CI jobs take.

· 3 min read
Damian Nadales

High level summary​

During the past two weeks we got the results from the system level benchmarks for UTxO HD. They showed a substantial performance regression, so we spent some time analyzing the results. We found out the frequency at which ledger snapshots were taken was too high, so we requested the benchmarking team a new run with a more realistic snapshotting policy. We continued refactoring and improving the prototype, and we released UTxO-HD related packages to CHaP.

We met with IOG researchers and networking specialists to discuss the Genesis design, which was well received. We continued working on testing and benchmarking different Genesis prototypes.

We are also working on solving a test failure related to iterators. This work derived in several improvements such as better documentation, a framework for writing unit (and regression) tests, and the possibility of debugging QuickCheck counter examples in the REPL.

Finally, we released ouroboros-consensus 0.2.0.0 and ouroboros-consensus-cardano 0.3.0.0 to CHaP

Workstreams​

UTxO HD Prototype​

We got the results of the first system level benchmarks for UTxO HD. They seemed to indicate a significant regression in performance. After looking into the benchmark logs we found that the benchmark runs took ledger state snapshots too often, due to the default snapshotting policy depending on k, and k being so small in the benchmark runs. Therefore, the next step is to re-run the benchmarks with a snapshotting policy that more closely resembles the one from mainnet.

At the same time, we continued refactoring and cleaning up the prototype.

Also, we prepared the anti-diff packages (fingertree-rm, diff-containers, simple-semigroupoids) and the lmdb related packages (cardano-lmdb and cardano-lmdb-simple) to CHaP.

Genesis​

The Genesis design was presented to the IOG researchers and Peter Thompson from NSol. It was well received. They pointed out one blindspot, but we think it'll be relatively simple to mitigate.

In parallel, we continued developing test and benchmarks for the Genesis prototypes. I particular we tested and implemented a potential fix for increased ChainDB dequeue timings, which partly behaved as we expected, but still needs further investigation. Also we obtained new benchmarking data for the prototype.

Technical debt​

Related to #4183, we developed a DSL for specifying ChainDB unit tests. This will allow us to better understand the counter-examples returned by QuickCheck tests, and to write regression tests for them. Also, we added a module to enable QuickCheck counter-examples to be run on the REPL, allowing for faster debugging feedback. Also, we improved the documentation related to followers (#4372).

We are also working on a design for optimizing the way we handle blocks from the future.

Support​

We released ouroboros-consensus 0.2.0.0 and ouroboros-consensus-cardano 0.3.0.0 to CHaP. Remember that we decided to split the packages related to Consensus into two bundles, one with the core functionality, Cardano-agnostic code, and another bundle with instantiations specific to Cardano.

· 2 min read
Damian Nadales

High level summary​

We continue refactoring the UTxO HD prototype while we wait for the system level benchmarks. We have created a new repository that contains the anti-diff packages used in this prototype.

On the Genesis front, we are preparing another meeting with the researchers to audit the implementation design, and we continued working on basic tests and simplifications.

During the past two weeks we also introduced two new tools. One for dumping CBOR encoded blocks to JSON, and another to serve a local immutable DB.

Workstreams​

UTxO HD Prototype​

We are in the process of refactoring the UTxO HD prototype, while we wait for the system level benchmarks to confirm if the performance of the prototype is satisfactory.

We also set up a repository for the anti-diff package, which required us to refactor the code, write documentation, and prepare a release to CHaP.

Genesis​

We worked on basic tests for the Limit on Eagerness property of Genesis. We also introduced further robustness and simplifications in the Genesis Density governor. Finally, we developed a presentation to engage again with the researchers on our Genesis implementation design.

Technical debt​

Fostering collaboration​

We are in the process of polishing the ouroboros-consensus documentation site, which we will use a the entry point for Consensus related documentation. The first version will not be complete, but we plan on systematically improving it.

Support​

We added a tool to ouroboros-consensus-cardano-tools which allows to dump the Chain DB blocks or any given CBOR encoded blocks as JSON.

We also added another tool that serves an existing immutable DB via BlockFetch and ChainSync. This tool can help in assisting our local benchmarking efforts (for instance Genesis' ChainSync jumping prototype).