High level summary
During the past two weeks, the Consensus team finalized the QSM tests for the
backing store and Mempool on the UTxO-HD branch with important discoveries
regarding parallel QSM testing. We also worked with the Ledger team to envisage
the modifications that are required in Ledger and Consensus to accommodate the
changes in the crypto VRF and KES. The db-analyser now supports bechmarking
the ledger operations, which will allow us to identify, debug, and profile
potential performance problems. We drafted a document that defines how to manage
the versions of Consensus-related packages. The top level documentation of
ouroboros-network now features a description of the consensus components and
provides a hyperlinked map to the modules documentation.
Workstreams
UTxO HD prototype
Whereas we had passing sequential state-machine tests for the mempool, the
parallel case proved to be more challenging than we thought. The operation of
adding a list of transactions to the mempool is not atomic and, as a result,
when adding a list of transactions, transactions from other processes can be
added in between. The mempool implementation handles this correctly, however
this required us to redesign the parallel model we had to take
the lack of atomicity into account.
Backing store property tests
We finished refactoring the backing store property tests. The second review
round is ongoing.
LSM tree implementation
We are working on benchmarking (in terms of time and number of IO operations)
fetching/looking up data from disk.
Genesis
We worked on the design of a mechanism to prevent a DoS attack on our Genesis
design related to rollbacks. This was arguably the biggest outstanding question.
During the discussions around Genesis, we noticed a design boundary that nicely
delineates a fundamental component. We almost have a full Haskell prototype of
it. It will be very nicely self-contained, perhaps even usable in the ultimate
implementation!
New VRF and KES crypto integration
We collaborated with the Ledger team on preparing the ledger state and crypto
types to avoid huge allocation on the epoch boundary when changing aspects of
the crypto that will only manifest in headers, not in the ledger states.
Technical debt
We merged the pull-request that adds a support to db-analyser for
benchmarking ledger operations. This will allow us to identify, debug, and
profile potential performance problems. The benchmark focus on the main 5 ledger
operations that are involved in chain syncing, block forging, and block
validation, namely:
- Forecast.
- Header tick.
- Header application.
- Block tick.
- Block application.
The following figure shows a plot of the benchmarking results for the first 65
million blocks (approximately) of the Cardano chain. The thin yellow lines under
the x-axis show the epoch boundaries, whereas the thick yellow lines correspond
to the era transitions.
As we can see in this figure, era and epoch boundaries require more computation
time. The ledger team are aware of this problem, and we are working to improve
this situation.
Fostering collaboration
We drafted a document motivating and defining how Consensus (and
possibly other core teams) will/should manage our package versions. This
pull-request garnered many great discussions from our team members and other
teams too: Sebastian Nagel, Arnaud Bailly, Michael Peyton-Jones, Ziyang Liu, et
al. We want to thank you all for your input, and we found this discussion very
enlightening!
We merged the pull request that adds an overview of consensus to
the top level
documentation of
ouroboros-network. This overview describes the consensus components and adds a
hyperlinked map to the modules documentation.
