High-level summary​

This week, the Hydra team has put their effort on finding solutions on many different problems, such as our broken head on mainnet #897, our fragile monthly report publications on the website and implementing “Option A” for external commits #215. Although most of these items are still open, huge progress has been made. To accelerate the investigation, they improved their logging to give more precise errors when a transition requirement fails, and to reduce duplication on effets logged content. Last but not least, the team is exploring formal methods and attended a workshop on formalizing cryptographic protocols in Agda.

What did the team achieve this week​

• Continued investigating broken head and opened an issue to keep track #897.

  As part of this issue, improvements were made to the node logs:

  • Give a precise error when a transition requirement fails #895.

  • Reduce duplication for effects logged content by using sequential eventId and effectId pair #896.

• Fixed references in the hydra specification #893.

• Attended a workshop on formal methods and crypto in Agda.

What are the goals of next week​

• Investigate and re-open our team-internal head on mainnet.
• Improve and provide regular benchmarks for Hydra #186.
• Complete journey for external commits implementing “Option A” and start implementing “Option B” #215.
• Authenticate network messages #727.
• Add hydra as tool to developr platform #872.
• Fix monthly report publication on docs website.

High level overview​

The Mithril team released a new 2321.1 distribution that fully implements the mechanism developed to sign generic data. They completed the upgrade of the Cardano node in the Mithril networks to v.8.0.0 and the implementation of the new computation of the stake distribution. They refactored the state machines of the signer and aggregator, and the signed entity service of the aggregator. Additionally, they worked on adapting the client and implementing a new sub-command for restoring the Cardano immutable file snapshots.

Finally, the team worked on adding a new certificate list route in the aggregator REST API, and started enhancing the infrastructure of the Mithril networks.

Low level overview​

• Released the new distribution 2321.1
• Worked on the epic that designs and implements generic signing/verification of entity services #780:
  • Completed the issue Enhance MessageAdapter for Artifact in aggregator REST API #925
  • Completed the issue Create the sub-command for 'Cardano Immutable Files Full' in client #895
  • Completed the issue Enhance state machines Aggregator/Signer #933
  • Completed the issue Adapt the aggregator REST API to list certificates #892
  • Worked on the issue Adapt end to end tests to handle new types of data #899
  • Worked on the issue Update client documentation #897
  • Worked on the issue Update architecture documentations for new types of data #898
• Worked on the epic that prepares the Mithril infrastructure for mainnet #767:
  • Worked on the issue Enhance terraform infrastructure #930
• Worked on the epic that implements the computation of the stake distribution for mainnet #880:
  • Completed the issue Upgrade Cardano node to '8.0.0' #920
• Completed the issue Add export path in Client CLI #512

High level summary​

During the Past two weeks we drafted an implementation path for concluding that a node is caught up, which will also be used to back Network's ledger-peer selection (see this issue). We also carried a thorough investigation into the exact feasibility of applying the Genesis rule to certain historical parts of the chain.

On the UTxO-HD front, we are working on improving the ledger tables design and wrapping up the improved DB locking mechanism. We also released packages that are required not only by UTxO-HD but are already used in cardano.

UTxO-HD​

• We have a plan for making the ledger tables in UTxO-HD more ergonomic by mimicking SOP classes like HPure and HAp . In short, we implement generalised versions of important classes like Applicative and Traversable.

Support​

• fs-sim-0.1.0.2 and fs-api-0.1.0.2 were released, which makes them now compatible with GHC up to 9.6.
• ouroboros-consensus-0.7.0.0 was released for cardano-node 8.1, including query serialization fixes for backwards compatibility.

High level summary​

• Benchmarking: We've performed and analysed first benchmarks with GHC9.2 builds. Additionally we have developed an early indicator for how build config changes might reflect on metrics from our model cluster.
• New tracing: Collaboration with Galois led to the new tracing system to be equipped with a re-forwarding mechanism.
• Nomad backend: Porting the 52 node model cluster to nomad cloud is ongoing, with the focus on deployment and health checks.

Low level overview​

Benchmarking​

The first set of runs with GHC9.2 as a build platform are in. We've discovered a significant difference in resource profile usage compared to GHC8.10. Further investigation uncovered the need for benchmarking another parameter change in the build configuration: As it stands, the ghc-bignum package is using the Haskell native-backend as a default. We strive to benchmark a build with the gmp-backend next.

A variant of our forge-stress local benchmark has been set up to serve as an early indicator for the resource usage profile we'd expect to observe on the model cluster. This provides us with a much tighter feedback loop, as local run duration is way shorter. This indicator is specific to changes in the configuration of build and the runtime systems, and will be of great support when evaluating different compiler versions or RTS flags incrementally.

Tracing​

The hub of the new tracing system cardano-tracer is designed with a fixed output behaviour, which is limited to various logging options. Thanks to the contribution from Galois, that design is now extended to be able to re-forward all, or a pre-filtered portion, of traces from the node in a configurable manner. This will enable downstream applications to directly receive the set of trace values relevant to their logic, without any additional cost for the node itself at all.

Nomad backend​

We're currently working out the details of efficiently deploying and monitoring a fleet of 50+ nodes, along with job definitions for tracing and transaction generation. Scaling up to those many instances, and monitoring an ongoing benchmarking run required us to fine-tune communications with the nomad server.

Related to that, the new cloud backend will provide a monitoring and health-checking mechanism which is far more flexible and offers more detailed insight than the previous iteration in cardano-ops. The backend will enable you to formulate very specific conditions for an ongoing run to be considered healthy, and offer automation of certain actions should these conditions not be met.

The team works on applied research and consulting in formal methods that is directly applicable to evidence based engineering in Core Tech and beyond.

High level summary​

This sprint the team has been working on final revisions for the pre-proceedings versions of two ICE 2023 papers.

Details​

• Porting Ouroboros Praos formalisation to new application layer.

• Preparing pre-proceedings version of two ICE 2023 papers.

• The team is hiring a performance engineering intern - https://apply.workable.com/io-global/j/BFEC5B0AFF/