High level summary​

1. P2P performance investigation is ongoing, in support of the networking team.
2. SECP benchmarking enablement is underway: we already have the script and are working on Plutus V2 generation support.
3. Unexpected setback in the new tracing system: full scale benchmarks have shown a performance regression: local chain syncing benchmarks were an improvement over legacy tracing.
4. On the open sourcing front we added an integrated data dictionary, which is necessary for explaining ourselves to the world. SRE collaboration on production deployment of performance data publishing has started.
5. We have started bringing the Nomad-based workbench backend closer to a cloud deployment scenario.

Performance​

We are supporting the networking team on P2P performance investigation. Generation support for Plutus V2 was started. We have collaborated with the Plutus team to get a SECP benchmark script, which is now ready for use, pending Plutus V2 support. The transaction generator has also been updated to the cardano-api changes.

Tracing​

We ran an initial round of full-scale benchmarks for the new tracing system -- which uncovered a regression relative to legacy tracing, which is contrary to the local chain syncing benchmarks, that showed improvement instead. We added tracing to cardano-tracer, fixing some minor bugs on the way. Network and disk IO metrics are now collected once again and are integrated into analysis.

Infrastructure​

The first iteration of the Nomad-based local workbench backend was completed -- it has reached feature parity with the existing supervisor backend. The next iteration started, bringing it closer to the cloud scenario, by deploying to separate Nomad tasks connected by a virtual network. This will serve as basis for CI and full cloud backends.

We designed and implemented the authoring pipeline for the performance data dictionary, which will be henceforth embedded in our performance reports. We are collaborating with SRE on production deployment of data publishing.

A number of smaller workbench, data analysis & reporting improvements have been made.

High level overview​

The SECP primitives AC has been met, and the test-vectors PR has been merged. Another of the main short-term goals is to implement KES secure forgetting. In the past week the mempool PR has been merged, and we've adapted the KES secure PR to the new mempool design. On top of this, we are working in updating the VRF batch compat version to use the audited version of the libsodium fork (PR#). Finally, we've 'cleaned' our libsodium fork, and we directly fork upstream, rather than forking Algorand's fork.

Low level overview​

• With the AC met, there was no more blockers from crypto to release the node tag with the SECP primitives. We are now working closely with dQuadrant to to implement some E2E tests with an actual bridge implementation.
• We previously had a C implementation of a memory pool for usage in KES secure forgetting. After a discussion with Alexey, we decided to use his implementation of a mempool in Haskell. We are working in adapting KES Secure forgetting to use this Haskell mempool.
• In parallel, we keep progressing on the KES agent
• VRFBatchCompat was deactivated due to a lack of an external audit. This audit was finished a few weeks back, and we are updating cardano base to use this new librar. In particular, this library implements the latest changing version of the VRF draft (13, which seems to remain stable), and links to the libsodium fork which implements batch verification.
• Our libsodium fork now links directly to upstream libsodium.

High-level summary​

This week, the Hydra team published together with Obsidian Systems a light paper on our "Hydra for Payments" project (Link). They have created a draft scope RFP for the external audit and worked with the internal audit team to clear up the specification. From the development side, they have fixed a bug with chain-following when using persistence and improved logs for better observability of hydra-node processes.

What did the team achieve this week​

• Published Hydra for Payments light paper (Link)
• Have a draft RFP ready for a first review internally
• Answered the internal auditor’s questions
• Fixed a bug with following the chain when starting with persistence (#599)
• Minor improvements to logging for better observability (#598, #600)
• Non-achievement: Needed to work around flaky TUI ci, follow-up issue if anyone wants to have a look (#590)

What are the goals of next week​

• Implement event-sourced persistence #580
• Close more gaps #452
• Attend the Cardano Summit in Lausanne. Hydra will be topic of one talk! Save the date+time: "Developing Hydra" on Day 2, Nov 21st, 13:50 CET

High level summary​

I am extremely excited to say that we now have a pull request up which introduces our new versioned CBOR serialization. This was an enormous effort, but it will solve a host of problems that we have had since the Shelley phase. It will take time to properly review it, and we will need to put in a lot of effort to integrate it with the downstream components, but this is a huge milestone. Additionally, we have a new CIP proposing a deprecation cycle for the transaction serialization schemes.

We also have a draft pull request that reworks how deposits are tracked. Users of the system will not notice any difference, but it is a necessary change needed to prepare the way for decentralizing the governance of Cardano.

Finally, we continued to address technical debt. In particular, we continued to make progress on bringing coherency and consistency to the code base with a common naming convention, and improving some error messages.

Lower level summary​

• We have a pull request up for our new versioned CBOR serialization. When we encounter a problem with our deserializers, it can be very difficult to implement a fix. It is difficult because we can only fix such issues during a hard fork, and leading up to the hard fork we must maintain two serializations for the same type in order to not cause unintended network splitting (the problematic version must be used before the hard fork, and the fixed version is used afterwards). This can be especially tricky with the FromCBOR typeclass, since it is not always easy to search for where all the problematic uses are located. The new versioned CBOR serialization allows us to gracefully handle this transition. See [pull-3138].
• We proposed a CIP for backwards compatibility of the transaction serialization schemes. See [pull-372].
• We have draft for the new deposit tracking. This draft is not as memory efficient as the final version will be, but it is a sufficient proof of concept that we can write property tests against, ensuring that we have not changed the semantics. We will optimize after we are sure of the correctness. See [pull-3127].
• We now provide better support for debugging failed Plutus scripts in an important helper function, named evaluateTransactionExecutionUnits. In particular, it now returns all the information needed to rerun the script with exactly the same arguments. This feature will end up appearing in the CLI and other tools from the Plutus tools team. See [pull-3135].
• We did a lot more renaming to bring coherency and consistency to the code base. See [pull-3126], [pull-3120], [pull-3118], and [pull-3116].
• We have added a few things to the ledger repository to make it conform to the Cardano Engineering Handbook See [pull-3139].

High-level summary​

In last sprint we got a performance report of P2P performance testing cluster (which consists of 50 nodes). There is a performance regression in the header notification metric. The P2P cluster is constructed with the same topology as the non-p2p reference one this indicates some regression which needs to be further investigated. This poses a risk for releasing P2P.

We also continued to work on peer sharing: pull #4019.

We continued working on dynamic block production which is required for P2P release for BP nodes: pull #3159.

We simplified the P2P topology format: issue #4559, pull #3888.

We added a new trace point for asynchronous demotions of local peers with Warning severity. This trace is important for SPOs.

Detail description​

Performance regression​

Below we include a graph which shows the performance regression of the P2P code base vs non P2P.

On the x axis is time in seconds which measures the delay from the start of the slot to when a header was received. The y axis is the percentile of nodes that received a header. We are currently investigating possible causes of the regression.

New P2P topology form​

The new topology file format is described in this issue #4559.

Tracing improvements​

• We improved a handshake error reporting, pull #4136
• We added TraceDemoteLocalAsynchronous rendered as DemoteLocalAsynchronous in json format, pull #4127. Such demotions should be investigated by the pool operator. They can indicate a problem in the deployed system, but also they could indicate a remote problem in arranged connections with other SPOs.

Open Source Improvements​

We improved documentation of io-sim and typed-protocols for open-source contributors and/or maintenance tasks: pull #22, pull #45, pull #48.