Skip to main content

· 6 min read
Damian Nadales

High-level summary​

During the past two weeks, the consensus team started documenting the implementation of the UTxO HD feature and continued developing tests for it. As part of our work on UTxO HD, we improved the Haskell support for LMDB. We also spent time working on the LSM tree prototype, and designed a parameter tuning algorithm for it. Regarding our work on Genesis, our investigation of the "plateaus" pointed at the TICKF slowdown on era boundaries as culprit. This led us to developing a caching strategy that will not only remove the aforementioned "plateaus", but can help alleviating the growing block production delay on epoch switch. We also helped reviewing the block forge credential hotswap feature, which is intended for use in the adoption of P2P.

We also worked on paying technical debt and fostering collaboration. In particular, we improved the io-sim framework, which is crucial for testing and simulating Cardano components. We also removed thunks that appeared on era translations, and improved our diffusion pipelining feature. We are working on a presentation for explaining Praos and Genesis.

High-level status report​

  • Finish the UTxO HD prototype: in progress.
    • We added documentation for this feature.
    • We developed the second version of the mempool tests.
    • We fixed benchmarks that were inflating the speedup we observed in the anti-diff implementation of sequences of differences. Speedups are now in the range of [3.33, 4.75], which remain significant.
    • We continued improving Haskell LMDB support.
    • We finished implementing a "parameter tuning algorithm" for the LSM tree prototype. This enables us to run experiments to check the correctness of the algorithm.
  • Genesis: in progress.
    • Work investigating the "plateaus" in the ChainSync jumping prototype pointed to the TICKF slowdown on era boundaries as culprit.
  • Tech debt:
    • We improved the capabilities of our io-sim library, which is crucial for testing and simulating Cardano components.
    • We removed thunks from epoch translations in the ledger.
    • We added Linux CI support for lmdb-simple.
    • We got pending diffusion pipelining improvements merged.
  • Fostering collaboration:
    • We are working on a explanation of Praos and Genesis protocols.
  • Support:
    • Investigation of CSJ "plateaus" led us to developing a caching strategy for TICKF that will not only remove these "plateaus", but can help alleviating the growing block production delay on epoch switch.
    • We reviewed the block forge credential hotswapping feature which is intended for use in the adoption of P2P.

Workstreams​

Finish the UTxO HD prototype​

We merged PR #4060, which adds a report documenting the UTxO HD feature, and puts emphasis in explaining how the mempool works in combination with UTxO HD.

We opened a draft PR with the second iteration of the property tests for the mempool (#4076).

We fixed the Arbitrary instances for keys and values in DiffSeq benchmarks (#4143). The problem was that we were testing with mostly small values, which artificially boosted the performance gains we saw on benhcmarks. Speedups are now in the range of [3.33, 4.75] across the different configurations.

Backing store property tests​

We focused on incorporating feedback on the monadic cursor API PR (#1). This required us to make small tweaks to quickcheck-lockstep to test the new API. We also updated the backing store property tests to use the new version of the monadic cursor API.

LSM tree implementation​

We worked on the LSM tree prototype. In particular: finished implementing a "parameter tuning algorithm" that adapts the LSM tree design based on factors like:

  • workload
  • machine specs,
  • and characteristics of the data being stored.

We are now running experiments to gather results and cross-reference them with existing experimental results from the LSM tree paper to see if the algorithm is working correctly.

Benchmarking the CSJ prototype​

We focused on investigating the "plateaus" in the ChainSync tip, which turned out to be due to the TICKF bug which we previously were only aware of in the context of the long forging times near epoch boundaries. For the most drastic patch by @nfrisby to speed up TICKF, full sync is speeding up by 7%.

The following plot shows that by caching the TICKF the ChainSync tip and the VolatileDB tip progress at the same rate.

The plot below shows the speedup observed by caching the TICKF rule wrt the baseline.

Technical debt​

After addressing the PR comments, we merged PR #16, which implements the MonadCatch instance for STM. This extends the capability of our io-sim library, which is crucial for testing and simulating Cardano components PR #16 closed #1461. This new feature was published as version 0.4.0.0 of io-sim.

We continued with our work fixing the NoThunk errors required for enabling nightly tests, with the help of TVarInvariant checks in strict-stm and nothunks libraries. We proposed fixes in cardano-ledger that took care of thunks that appeared in era translations (#3143). The fixes will be integrated back into consensus when cardano-ledger approves and publish the changes introduced in #3143.

We added CI support for lmdb-simple (#2). We currently test the build on a Linux environment only.

We got pending diffusion pipelining PRs (#3857, #3860, #3856) merged, after rebasing and addressing feedback.

Fostering collaboration​

@nfrisby finished a visualisation tool and outlined scripts for the Praos and Genesis explanation presentations. The idea is to produce a video that gives an overview of these protocols.

Support​

We started working on caching the computation of the TICKF rule (#4054), since this was blocking our benchmarking work for Genesis. In addition, this issue has the Cardano community quite concerned, so we are hoping the work done in caching the computation of the TICKF rule can help alleviating the growing block production delay on epoch switch.

We reviewed the block forge credential hotswapping PR #3800 from the networking team, which is intended for use in the adoption of P2P.

· One min read
Jordan Millar

2022-11-16 - 2022-11-29

High level summary​

Completed​

docs​

cardano-cli​

cardano-api​

cardano-node​

cardano-testnet​

In Progress​

cardano-cli​

cardano-api​

cardano-node​

· 2 min read
Serge Kosyrev

High level summary​

  1. P2P performance investigation is ongoing, in support of the networking team.
  2. SECP benchmarking enablement is underway: we already have the script and are working on Plutus V2 generation support.
  3. Unexpected setback in the new tracing system: full scale benchmarks have shown a performance regression: local chain syncing benchmarks were an improvement over legacy tracing.
  4. On the open sourcing front we added an integrated data dictionary, which is necessary for explaining ourselves to the world. SRE collaboration on production deployment of performance data publishing has started.
  5. We have started bringing the Nomad-based workbench backend closer to a cloud deployment scenario.

Performance​

We are supporting the networking team on P2P performance investigation. Generation support for Plutus V2 was started. We have collaborated with the Plutus team to get a SECP benchmark script, which is now ready for use, pending Plutus V2 support. The transaction generator has also been updated to the cardano-api changes.

Tracing​

We ran an initial round of full-scale benchmarks for the new tracing system -- which uncovered a regression relative to legacy tracing, which is contrary to the local chain syncing benchmarks, that showed improvement instead. We added tracing to cardano-tracer, fixing some minor bugs on the way. Network and disk IO metrics are now collected once again and are integrated into analysis.

Infrastructure​

The first iteration of the Nomad-based local workbench backend was completed -- it has reached feature parity with the existing supervisor backend. The next iteration started, bringing it closer to the cloud scenario, by deploying to separate Nomad tasks connected by a virtual network. This will serve as basis for CI and full cloud backends.

We designed and implemented the authoring pipeline for the performance data dictionary, which will be henceforth embedded in our performance reports. We are collaborating with SRE on production deployment of data publishing.

A number of smaller workbench, data analysis & reporting improvements have been made.

· 2 min read
Iñigo Querejeta Azurmendi

High level overview​

The SECP primitives AC has been met, and the test-vectors PR has been merged. Another of the main short-term goals is to implement KES secure forgetting. In the past week the mempool PR has been merged, and we've adapted the KES secure PR to the new mempool design. On top of this, we are working in updating the VRF batch compat version to use the audited version of the libsodium fork (PR#). Finally, we've 'cleaned' our libsodium fork, and we directly fork upstream, rather than forking Algorand's fork.

Low level overview​

  • With the AC met, there was no more blockers from crypto to release the node tag with the SECP primitives. We are now working closely with dQuadrant to to implement some E2E tests with an actual bridge implementation.
  • We previously had a C implementation of a memory pool for usage in KES secure forgetting. After a discussion with Alexey, we decided to use his implementation of a mempool in Haskell. We are working in adapting KES Secure forgetting to use this Haskell mempool.
  • In parallel, we keep progressing on the KES agent
  • VRFBatchCompat was deactivated due to a lack of an external audit. This audit was finished a few weeks back, and we are updating cardano base to use this new librar. In particular, this library implements the latest changing version of the VRF draft (13, which seems to remain stable), and links to the libsodium fork which implements batch verification.
  • Our libsodium fork now links directly to upstream libsodium.

· One min read
Sebastian Nagel

High-level summary​

This week, the Hydra team published together with Obsidian Systems a light paper on our "Hydra for Payments" project (Link). They have created a draft scope RFP for the external audit and worked with the internal audit team to clear up the specification. From the development side, they have fixed a bug with chain-following when using persistence and improved logs for better observability of hydra-node processes.

What did the team achieve this week​

  • Published Hydra for Payments light paper (Link)
  • Have a draft RFP ready for a first review internally
  • Answered the internal auditor’s questions
  • Fixed a bug with following the chain when starting with persistence (#599)
  • Minor improvements to logging for better observability (#598, #600)
  • Non-achievement: Needed to work around flaky TUI ci, follow-up issue if anyone wants to have a look (#590)

What are the goals of next week​

  • Implement event-sourced persistence #580
  • Close more gaps #452
  • Attend the Cardano Summit in Lausanne. Hydra will be topic of one talk! Save the date+time: "Developing Hydra" on Day 2, Nov 21st, 13:50 CET