High-level summary​

This week, the Hydra team published together with Obsidian Systems a light paper on our "Hydra for Payments" project (Link). They have created a draft scope RFP for the external audit and worked with the internal audit team to clear up the specification. From the development side, they have fixed a bug with chain-following when using persistence and improved logs for better observability of hydra-node processes.

What did the team achieve this week​

• Published Hydra for Payments light paper (Link)
• Have a draft RFP ready for a first review internally
• Answered the internal auditor’s questions
• Fixed a bug with following the chain when starting with persistence (#599)
• Minor improvements to logging for better observability (#598, #600)
• Non-achievement: Needed to work around flaky TUI ci, follow-up issue if anyone wants to have a look (#590)

What are the goals of next week​

• Implement event-sourced persistence #580
• Close more gaps #452
• Attend the Cardano Summit in Lausanne. Hydra will be topic of one talk! Save the date+time: "Developing Hydra" on Day 2, Nov 21st, 13:50 CET

High level summary​

I am extremely excited to say that we now have a pull request up which introduces our new versioned CBOR serialization. This was an enormous effort, but it will solve a host of problems that we have had since the Shelley phase. It will take time to properly review it, and we will need to put in a lot of effort to integrate it with the downstream components, but this is a huge milestone. Additionally, we have a new CIP proposing a deprecation cycle for the transaction serialization schemes.

We also have a draft pull request that reworks how deposits are tracked. Users of the system will not notice any difference, but it is a necessary change needed to prepare the way for decentralizing the governance of Cardano.

Finally, we continued to address technical debt. In particular, we continued to make progress on bringing coherency and consistency to the code base with a common naming convention, and improving some error messages.

Lower level summary​

• We have a pull request up for our new versioned CBOR serialization. When we encounter a problem with our deserializers, it can be very difficult to implement a fix. It is difficult because we can only fix such issues during a hard fork, and leading up to the hard fork we must maintain two serializations for the same type in order to not cause unintended network splitting (the problematic version must be used before the hard fork, and the fixed version is used afterwards). This can be especially tricky with the FromCBOR typeclass, since it is not always easy to search for where all the problematic uses are located. The new versioned CBOR serialization allows us to gracefully handle this transition. See [pull-3138].
• We proposed a CIP for backwards compatibility of the transaction serialization schemes. See [pull-372].
• We have draft for the new deposit tracking. This draft is not as memory efficient as the final version will be, but it is a sufficient proof of concept that we can write property tests against, ensuring that we have not changed the semantics. We will optimize after we are sure of the correctness. See [pull-3127].
• We now provide better support for debugging failed Plutus scripts in an important helper function, named evaluateTransactionExecutionUnits. In particular, it now returns all the information needed to rerun the script with exactly the same arguments. This feature will end up appearing in the CLI and other tools from the Plutus tools team. See [pull-3135].
• We did a lot more renaming to bring coherency and consistency to the code base. See [pull-3126], [pull-3120], [pull-3118], and [pull-3116].
• We have added a few things to the ledger repository to make it conform to the Cardano Engineering Handbook See [pull-3139].

import NoticedHeaders from '@site/static/images/network/2022-11-11-noticed-headers.png';

High-level summary​

In last sprint we got a performance report of P2P performance testing cluster (which consists of 50 nodes). There is a performance regression in the header notification metric. The P2P cluster is constructed with the same topology as the non-p2p reference one this indicates some regression which needs to be further investigated. This poses a risk for releasing P2P.

We also continued to work on peer sharing: pull #4019.

We continued working on dynamic block production which is required for P2P release for BP nodes: pull #3159.

We simplified the P2P topology format: issue #4559, pull #3888.

We added a new trace point for asynchronous demotions of local peers with Warning severity. This trace is important for SPOs.

Detail description​

Performance regression​

Below we include a graph which shows the performance regression of the P2P code base vs non P2P.

On the x axis is time in seconds which measures the delay from the start of the slot to when a header was received. The y axis is the percentile of nodes that received a header. We are currently investigating possible causes of the regression.

New P2P topology form​

The new topology file format is described in this issue #4559.

Tracing improvements​

• We improved a handshake error reporting, pull #4136
• We added TraceDemoteLocalAsynchronous rendered as DemoteLocalAsynchronous in json format, pull #4127. Such demotions should be investigated by the pool operator. They can indicate a problem in the deployed system, but also they could indicate a remote problem in arranged connections with other SPOs.

Open Source Improvements​

We improved documentation of io-sim and typed-protocols for open-source contributors and/or maintenance tasks: pull #22, pull #45, pull #48.

High level summary​

This week, the hydra team first re-deployed the latest Hydra scripts to the re-spun preview network, see 0.8.0 release notes. They also completed implementation of ADR18 and worked on the validators, but development got impacted by some CI flakyness. The team also met to discuss hard forks & protocol parameter updates #195 and alignment of the specification document with auditors.

What did the team achieve this week​

• Complete and merge ADR18 #579
• Re-deploy hydra scripts to respun preview network, see 0.8.0 release notes #595
• Have first gap of #452 in review.
• Non-achievement: Flaky CI for TUI was impacting us, so we investigated this a lot.
• Engineering meeting to discuss hard forks and protocol parameter updates #195
• Met the internal audit team on the specification to set scope, expectations and collected requirements/open questions.
• Drafted project scope for an external audit RFP.

What are the goals of next week​

• Implement event-sourced persistence #580
• Answer the internal auditor’s questions
• Have a draft RFP ready for a first review internally
• Close some gaps #452

This sprint, the team has been working on the new continuous integration and delivery (CI/CD) pipelines and the automated deployment of environments as part of the new version of the release process. They also coordinated the migration of the pioneer SPO nodes to these new Mithril networks. They have been implementing the automatic data storage upgrade of the signer and the aggregator nodes. Finally, on the crypto side of things, we've implemented an efficiency improvement on the size of the mithril certificates.

Low level overview

• We have been moving forward on the implementation of the release process #500:
  • Setup of the new hosted environments for testing-preview, pre-release-preview and release-preprod with their terraform and GitHub environments #542
  • Adapted the CI workflows to work with the new release process #543
  • Publication of an ADR3
  • Publication of a dev blog post about Mithril networks evolution
  • Releasing our first Mithril distribution 2244.0
• Worked on the API versioning mechanism #565
• Worked on the implementation of the stores migration process for the signer and aggregator nodes #562
• Prepared a Mithril devnet video demo #526
• Implemented a batch Merkle Tree proof, which reduces the size of certificates considerably #484