Skip to main content

· 3 min read
Jared Corduan

High level summary​

The ledger team is still primarily focused on addressing technical debt. We now have the infrastructure for versioning our serialization schemes, which we continue to put into action. We have made first steps towards getting proper support for the formal ledger repository (in particular, we've added nix builds and continuous integration support). We are wrapping up an investigation of the performance of a critical function used by the consensus layer for leader checks. Finally, we are improving the packaging and versioning of our code.

Lower level summary​

Completed Technical Debt​

  • Because the Shelley ledger era was a complete re-write of the Byron ledger era, a lot of our code lives in the cardano-ledger-shelley package, though with hindsight we can say that much of it should live in cardano-ledger-core. We continue to move things to cardano-ledger-core, and have much more to come. [pull-3059]
  • We now have the infrastructure to support versioned serialization schemes. The inability to do this has caused us a lot difficulties, such as [issue-3003], [issue-2965] and [issue-2444]. We are still in the process of switching to the versioned serialization scheme (such as [pull-3078]), but the infrastructure was completed in [pull-3063].
  • We now have proper nix and CI support for the formal ledger project. [pull-19]
  • A separate team is helping bring support for CHaP to all the cardano-node repositories. We have been helping out with this effort.
  • A separate team continues working on upgrading all the cardano-node repositories to work with ghc 9.2.4. We have been helping out with this effort.

In-progress Technical Debt​

We also have several fairly large pull-requests in review that we are working on.

  • In an on-going attempt to build out a more user-friendly API, we continue to remove HasField instances in place of using micro-lenses. The protocol parameters, in particular, are being worked on. [pull-3045]
  • We are also renaming record fields to be consistent across the repository. [pull-3062]
  • We are now cleaning up all the work we did to understand the performance of the TICKF transition. We have some improvements to the computatation as well. [pull-3068]
  • We are adding more documentation, in particular to our Twiddler functionality. [pull-3073]
  • The formal ledger is adding support for finite set theory. [pull-20]

· 2 min read
Damian Nadales

High level summary​

During the past two weeks, the consensus team worked on improving the performance of the ChainSync jumping logic, which is needed for Genesis. We also rewrote the implementation of the mempool in the UTxO HD prototype which solved the issues that prevented us from running system level benchmarks. Also on the UTxO HD front, we have an improved implementation of the sequence-of-differences (a crucial piece of UTxO HD), and we also elaborated a test sign-off list for the UTxO HD feature.

Executive summary​

  • With the latest implementation of ChainSync jumping we are closer to the baseline performance. In particular, the prototype seems to benefit from the extra concurrency provided by additional capabilities.
  • We rewrote the implementation of the mempool in the UTxO HD prototype. This rewrite was required due to performance problems we observed when running the workbench. These performance problems prevented us from running system level benchmarks. The rewrite solved these issues. After the UTxO-HD: mempool rewrite PR is merged, we will contact the Benchmarking team so that they run the system level benchmarks.
  • The implementation of sequences of differences based on anti-diffs was integrated into the UTxO HD prototype. It is pending review and we also need to run replay and syncing benchmarks to confirm that this will deliver a performance improvement, as observed in our micro-benchmarks.
  • The UTxO HD prototype inspection resulted in a list of tests needed for consensus to consider the UTxO HD prototype as fully tested.

Additional information​

Genesis​

Benchmarking setup: 50MBit/s, 50ms latency

  • Red: baseline
  • Green: Current CSJ prototype, 10 peers, jumps every 3000/f slots.

As ChainSync Jumping involves many concurrent network operations at every jump, we tried to run the node with 6 instead of the default 2 capabilties.

  • Orange: baseline with 6 capabilities
  • Blue: CSJ prototype with 6 capabilities

This diagram shows the respective syncing progress, starting at Genesis and continuing a good part into Shelley (with the dashed line indicating the Byron-to-Shelley transition).

Further work includes whether we can tune the prototype to better handle few capabilities, or to adapt the default number of capabilities (potentially just while syncing).

· 2 min read
Kostas Dermentzis

DBSync Update​

Fast restarts​

We fixed a long overdue issue in db-sync which caused long delays on restarts 1266. This has been one of db-sync main objectives for this period. Restarts are now very fast, because db-sync deletes almost nothing from the db, it just replays the ledger rules until it reaches the tip of the db. The fix also improves reconnection speed, in cases where the node restarts or the connection is temporarily lost. It also speeds up even more in cases where due to a deployment mess up a very old snapshot or no snapshot at all is used.

Property based testing​

We added stateful property based testing, using quickcheck-state-machine 1269. These tests use empty or almost empty blocks to test the new behaviour of restarts and rollbacks.

These tests generate arbitrarily a list of symbolic commands from these:

RollForward Int
RollBack BlockNo
StopDBSync
StartDBSync
RestartNode
AssertBlockNo BlockNo

The commands are translated into real commands. For example RollForward Int will forge a new block that fits on the current chain. These real commands are executed against db-sync using the mock chain-sync server. The symbolic commands are executed against a vesy simplistic Model of db-sync which looks like this:

  Model
    { serverTip :: BlockNo
    , dbSyncTip :: BlockNo
    , dbSynsIsOn :: Bool
    , dbSynsHasSynced :: Bool
    }

Finally a number of postconditions are checked, related to the eventual block number of db-sync.

Tech Debt​

We handled a number of tech debt in 1275 This improves the code format of db-sync, deletes many queries that were never used and groups the others. This tech debt resolution not only improves the experience of working in db-sync, but can facilitate some of our other objectives, as it makes it very explicit which queries are used during syncing and which indexes are necessary.

Smash​

We worked on fixing an issue related to fetching pool metadata 1276. The issue which is described in 1270

· One min read
Jordan Millar

Node-Api-Cli Update​

2022-10-04 - 2022-10-18​

Executive Summary​

The majority of the team's time was spent between getting 1.34.4 ready, addressing various feature requests/issues/bugs that have arisen and refactoring components in the api and cli. The current refactoring is aimed at the long term goal of empowering users to be able to easily build applications similar to cardano-cli.

Completed​

cardano-cli​

cardano-api​

cardano-node​

In Progress​

cardano-cli​

cardano-api​

cardano-node​

· 3 min read
Jared Corduan

Ledger Update​

We have continued focusing nearly entirely on addressing technical debt. A lot of design work has begun for the next ledger era, but we do not yet have anything concrete to share.

Technical debt issues completed​

  • [issue-1676][pull-2992] We have finally removed the ledger dependency on the cardano-prelude package. It was barely used in the ledger repository, and it added a dependency that we did not want to maintain. It was a bit difficult to remove, and we had to coordinate removing it from cardano-base. A lot ended up going into pull-2992, due to the coordination effort, and we ended up updating Plutus as well. This means that we've now also made a lot of progress on the problematic cost model serialization issues described in issue-2902. In particular, after we resolve issue-3014, we will not have to wait an epoch before releasing a cost model for a new version of Plutus, as we had to do for the Vasil HF.
  • [issue-3046][pull-3055] We moved a module that is now only used in Byron to a Byron package.
  • [issue-3047][pull-3054] We improved the interface to the Value (multi-asset) type.
  • [pull-3044] We debugged and fixed a tricky compilation issue. Certain kinds of field updates were adding approximately 20 minutes to our compile time!
  • [issue-2932][pull-3036] As a part of our ongoing re-organization of the codebase, we have added a Cardano.Ledger.[Era].Core module to each ledger era that has a TxBody class. Most classes defined in the era should go in this new module. We also re-export the Cardno.Ledger.Core module and the previous Cardano.Ledger.[Era].Core modules from each era.

Technical debt in progress​

  • [issue-3034][issue-3035][node-issue-4421] We are continuing to write benchmarks to understand exactly where all the time is being spent on executing the TICKF transition. The consolidation of the per-stake-credential stake distribution to the per-stake-pool distribution does seem to account for a large amount of time (near a second as written, which we have down to about half a second with some optimizations), but this does not account for everything. Applying the reward update may also be a big contributing factor.
  • [pull-3033][pull-3038][pull-3041] A separate team is working on upgrading all the cardano-node repositories to work with ghc 9.2.4. We have been helping out with this effort.
  • The nix scripts used to build our new formal ledger model do not work consistently for everyone, and we have been working on fixing these issues.
  • [issue-3014] We are still working on adding a versioning scheme to all of the ledger serializers.