High-level summary
During the past two weeks, the consensus team started documenting the
implementation of the UTxO HD feature and continued developing tests for it. As
part of our work on UTxO HD, we improved the Haskell support for LMDB. We also
spent time working on the LSM tree prototype, and designed a parameter tuning
algorithm for it. Regarding our work on Genesis, our investigation of the
"plateaus" pointed at the TICKF slowdown on era boundaries as culprit. This
led us to developing a caching strategy that will not only remove the
aforementioned "plateaus", but can help alleviating the growing block production
delay on epoch switch. We also helped reviewing the block forge credential
hotswap feature, which is intended for use in the adoption of P2P.
We also worked on paying technical debt and fostering collaboration. In
particular, we improved the io-sim framework, which is crucial for testing and
simulating Cardano components. We also removed thunks that appeared on era
translations, and improved our diffusion pipelining feature. We are working on a
presentation for explaining Praos and Genesis.
High-level status report
- Finish the UTxO HD prototype: in progress.
- We added documentation for this feature.
- We developed the second version of the mempool tests.
- We fixed benchmarks that were inflating the speedup we observed in the
anti-diff implementation of sequences of differences. Speedups are now in the
range of [3.33, 4.75], which remain significant.
- We continued improving Haskell LMDB support.
- We finished implementing a "parameter tuning algorithm" for the LSM tree
prototype. This enables us to run experiments to check the correctness of
the algorithm.
- Genesis: in progress.
- Work investigating the "plateaus" in the
ChainSync jumping prototype
pointed to the TICKF slowdown on era boundaries as culprit.
- Tech debt:
- We improved the capabilities of our
io-sim library, which is crucial for
testing and simulating Cardano components. - We removed thunks from epoch translations in the ledger.
- We added Linux CI support for
lmdb-simple. - We got pending diffusion pipelining improvements merged.
- Fostering collaboration:
- We are working on a explanation of Praos and Genesis protocols.
- Support:
- Investigation of CSJ "plateaus" led us to developing a caching strategy for
TICKF that will not only remove these "plateaus", but can help alleviating
the growing block production delay on epoch switch. - We reviewed the block forge credential hotswapping feature which is intended
for use in the adoption of P2P.
Workstreams
Finish the UTxO HD prototype
We merged PR #4060, which adds a report documenting the UTxO HD
feature, and puts emphasis in explaining how the mempool works in combination
with UTxO HD.
We opened a draft PR with the second iteration of the property tests for the
mempool (#4076).
We fixed the Arbitrary instances for keys and values in DiffSeq benchmarks
(#4143). The problem was that we were testing with mostly small
values, which artificially boosted the performance gains we saw on benhcmarks.
Speedups are now in the range of [3.33, 4.75] across the different
configurations.
Backing store property tests
We focused on incorporating feedback on the monadic cursor API PR (#1).
This required us to make small tweaks to quickcheck-lockstep to test the new
API. We also updated the backing store property tests to use the new version of
the monadic cursor API.
LSM tree implementation
We worked on the LSM tree prototype. In particular: finished implementing a
"parameter tuning algorithm" that adapts the LSM tree design based on factors
like:
- workload
- machine specs,
- and characteristics of the data being stored.
We are now running experiments to gather results and cross-reference them with
existing experimental results from the LSM tree paper to see if the algorithm is
working correctly.
Benchmarking the CSJ prototype
We focused on investigating the "plateaus" in the ChainSync tip, which turned
out to be due to the TICKF bug which we previously were only aware of in the
context of the long forging times near epoch boundaries. For the most drastic
patch by @nfrisby to speed up TICKF, full sync is speeding up by 7%.
The following plot shows that by caching the TICKF the ChainSync tip and the
VolatileDB tip progress at the same rate.
The plot below shows the speedup observed by caching the TICKF rule wrt the
baseline.
Technical debt
After addressing the PR comments, we merged PR #16, which implements
the MonadCatch instance for STM. This extends the capability of our io-sim
library, which is crucial for testing and simulating Cardano components PR #16
closed #1461. This new feature was published as version 0.4.0.0
of io-sim.
We continued with our work fixing the NoThunk errors required for enabling
nightly tests, with the help of TVarInvariant checks in strict-stm and
nothunks libraries. We proposed fixes in cardano-ledger that took care of
thunks that appeared in era translations (#3143). The fixes will be
integrated back into consensus when cardano-ledger approves and publish the
changes introduced in #3143.
We added CI support for lmdb-simple (#2). We currently test the build on
a Linux environment only.
We got pending diffusion pipelining PRs (#3857, #3860,
#3856) merged, after rebasing and addressing feedback.
Fostering collaboration
@nfrisby finished a visualisation tool and outlined scripts for the Praos and
Genesis explanation presentations. The idea is to produce a video that gives an
overview of these protocols.
Support
We started working on caching the computation of the TICKF rule
(#4054), since this was blocking our benchmarking work for
Genesis. In addition, this issue has the Cardano community quite
concerned, so we are hoping the work done in caching the
computation of the TICKF rule can help alleviating the growing block
production delay on epoch switch.
We reviewed the block forge credential hotswapping PR #3800 from the
networking team, which is intended for use in the adoption of P2P.
