Skip to main content

· One min read
Damian Nadales

High level summary

  • The team showed the prototype described in #1701 during the Leios monthly Demo.
  • For Peras, #1678 was merged after review with the Consensus team. The October demo was also completed.
  • Consensus patched versions for node 10.6 were released (#1729), which include making Dijkstra genesis optional (#1725) and streaming ledger tables in the snapshot converter (#1728).
  • Resource-registry 0.2.0.0 was released (io-classes-extra#11), which adds transferRegistry (io-classes-extra#9) and releases threads before closing (io-classes-extra#10).
  • CHaP's update ensures in-future packages cannot be released (cardano-haskell-packages#1159).
  • Work is ongoing to enable randomized snapshot delays.

· 3 min read
Alexey Kuleshevich

High level summary

This period marks a very nice milestone for the Ledger team. We have finalized CIP-118 - Nested Transactions with drastic simplifications through reliance on CIP-112 - Observe script type and CDDL specification with changes to the transaction. This step puts it in the ready state for the final reviews. Once merged it will conclude the last point on the first milestone listed in the Nested Transaction proposal. Furthermore we have also implemented the distinction between sub-transactions and the top level transaction in such a way that will allow us reuse most of the existing Ledger functionality for transaction validation, regardless of the level. This point takes us right to the finish line of having the second milestone completed as well for the Nested Transaction proposal that was promised by the Ledger team. We expect this milestone to be finalized in the next couple of days.

Beside significant progress on Nested Transactions we have also: implemented a proper solution for preventing invalid serialization for a few obscure edge cases in a transaction, tackled some outstanding tech dept and expanded our test suite.

Low level summary

Features

  • PR-5350 - Switch role of reqSignerHashes from Witness to Guard
  • PR-5341 - Shelley: Move withdrawals draining from DELEGS to LEDGER
  • PR-5351 - Various minor improvements
  • PR-5357 - Rename PoolParams to StakePoolParams
  • PR-5363 - Make Annotator capable of failing.
  • PR-5366 - Backport of a bugfix in queryPoolState
  • PR-5365 - Fix a bug in queryPoolState
  • PR-5368 - Add DecCBOR for ShelleyBbodyPredFailure
  • PR-5362 - CDDL: Switch to explicit exports and consolidate a few fields
  • PR-5334 - Multi level transaction definition

Testing

  • PR-5330 - Run Imp DELEG tests across eras
  • PR-5344 - Update formal-ledger and enable tests
  • PR-5267 - Remove Deleg.conwayEraSpecificSpec
  • PR-5358 - Update formal-ledger and enable conformance tests
  • PR-5352 - Remove tasty from all test suites except those in Byron

Infrastructure and releasing

  • PR-5311 - Check in CI if changelogs need a bump
  • PR-5335 - Improve error checking in CI changelog linting
  • PR-5354 - Remove LC_ALL from shellHook in flake.nix
  • PR-5353 - Fix broken link in RELEASING.md
  • PR-5359 - Add filename to diagnostics in undefined CI check
  • PR-5369 - Bump hls to 2.12 and cabal to 3.14.2

· 2 min read
Jean-Philippe Raynaud

High level overview

This week, the Mithril team prepared the pre-release for the 2543.0-pre distribution. This version introduces support for the default incremental backend (v2) for Cardano database restoration, enhanced integrity verification that reports any tampered or missing files in case of failure, and various bug fixes and improvements.

The team also completed the integration of the Haskell DMQ node into the end-to-end tests to enable decentralized signature diffusion. They implemented a simple aggregator discovery mechanism and continued work on the first phase of decentralizing configuration parameters. Additionally, they advanced the design of certificate snarkification.

Finally, they adapted the project to the latest NPM security changes for publishing packages and refactored the aggregator's HTTP client.

Low level overview

Features

  • Pre-released the new distribution 2543.0-pre
  • Completed the issue Integrate the Haskell DMQ node in the e2e test #2674
  • Worked on the issue Decentralization of configuration parameters - Phase 1 #2692
  • Worked on the issue Implement a simple aggregator discovery mechanism #2726
  • Worked on the issue Release 2543 distribution #2727
  • Worked on the design of the snarkification of the certificates

Protocol maintenance

  • Worked on the issue Implement a common aggregator client - Phase 1 #2640
  • Worked on the issue Enhance protocol security page on website #2703
  • Worked on the issue Support NPM security changes with trusted publisher tokens #2745

· 5 min read
Michael Karg

High level summary

  • Benchmarking: Various maintenance to prepare for upcoming Node 10.6 changes; metrics migration guide for SPOs.
  • Development: Prototyping a PromQL-based alert system for new tracing.
  • Infrastructure: Located a space leak that was interfering with on-disk benchmarks.
  • Tracing: Equipping dmq-node with the new tracing system; cardano-tracer library and API ongoing.
  • Leios: Impact analysis and tech design; preparation for simulations hand-off.
  • Hydra: Kick-off for development of system integration level benchmarks.
  • Node diversity: Trace semantics specification ongoing; example test case ready for merging.

Low level overview

Benchmarking

We've performed various maintenance tasks to accomodate our automation to several breaking changes that come with Node 10.6. There are new constraints on Plutus cost models in genesis files which make it more difficult to set up a customized testnet, like our benchmarking cluster. Furthermore, we've been assisting with integrating and debugging the upcoming release's components, such as tx validation in the cardano-api, or the implementation of new trace messages the Node emits.

As the new tracing system will be the default on Node 10.6, we've created a tool which, given a side-by-side listing of metrics names from both systems, will automatically generate an exhaustive migration guide for SPOs. As the metrics names have changed slightly, a migration for existing monitoring setups is required as a one-off effort. The migration guide will be published on the Cardano Developer Portal.

Whilst the legacy tracing system is still fully operational in Node 10.6, the release marks the begin of its deprecation period - giving SPOs sufficient time to adjust their setups.

Development

While cardano-tracer logs forwarded trace messages according to detailed configs, and exposes forwarded metrics, it does not provide any built-in functionality for monitoring and alerts. The experimental RTView component, which still can be built for cardano-tracer and remains fully functional, was an attempt to provide dashboards and alerts out of the box. However, due to its restricted design and low interoperability with existing monitoring standards, it has been discontinued.

Currently, we're taking another stab at this: We're building a prototype that creates timeseries directly from observed metrics, and is able to parse and evaluate PromQL queries referring to them. Based on that prototype, we'll assess resource usage and feasibility of fully building that feature into cardano-tracer. As most monitoring alerts can be (and are) defined as conditions on PromQL query results, and PromQL is an established industry standard, we see a low barrier for adaptation. Furthermore, if built sufficiently modular, it would eliminate the need to operate additional 3rd party services for scraping metrics and monitoring for alert conditions - at least in some usage scenarios.

Infrastructure

With help and support from the Consensus team (Gracias, Javier!), we were able to locate a space leak that affected on-disk benchmarks. While the current on-disk benchmarks are representative, the space leak prevented us from scaling memory limits for the Node process with finer granularity. This will get merged post 10.6 release, and will be of much use when we do comparative benchmarks of the LMDB and the new lsm-trees on-disk backing stores.

Tracing

We've been working on equipping the Network team's new dmq-node service with our new tracing system, trace-dispatcher - still a work in progress. Currently, the dmq-node uses plain contravariant tracing. Having a configurable system, with an API abstraction to define metadata and message serializations as well as process metrics, is a necessary step towards production-ready tracing. The added benefit of using trace-dispatcher is reusability of definitions already implemented in cardano-node, and a uniform way of how cardano-node and dmq-node are configured, as well as how the expose trace evidence and metrics.

The work on cardano-tracer as a library, with principled API and intra-process communications, is ongoing. Implementation is almost complete, and will enter testing phase soon.

Leios

We've contributed to Leios Impact Analysis. The Performance & Tracing section summarizes how implementing, and eventually benchmarking Leios will impact various aspects of our approach. This spans from adding new Leios-specific observables into component code, to deriving submission workloads suitable for Leios, to finding a scaling approach to be able to correlate performance observations to exact changes in Leios protocol parameters.

Additionally, we're currently working on the Leios technical design and implementation plan, which lays out our approach and derives some very high-level steps on how to realize it, based on the impact analysis.

Hydra

We've kicked off a collaboration with the Hydra team. The goal is to build system integration level benchmarks for Hydra, which can target system pressure points, and which are able to scale operations in various directions - much akin to what we're doing for the Cardano node. Eventually, those benchmarks should provide more than just immediate feedback for Hydra engineers; they should be able to yield realistic figures of what to expect from operating one (or more) Hydra heads, and what the resource requirements are to achieve that. Currently, we're familiarizing ourselves with the system and its built-in observability to assess if it meets the requirements for the benchmarks we have in mind.

Node diversity

The work on (multi-)node conformance testing is ongoing. We're in the process of creating a specification document for semantics of existing Node traces. While a few of them might be unique to the Haskell implementation, the majority documents how Cardano itself runs and behaves; those traces can implemented accordingly across diverse Node projects.

Our own demonstration test case for conformance testing is fully implemented and ready to be merged after the Node 10.6 release. It validates a metric exposed by a node wrt. trace evidence in its logs and internal events the metric is based on; see cardano-node PR#6322.

· 2 min read
Marcin Szamotulski

Overview of sprint 98 and 99

Cardano-Node 10.6 Release

We swiftly identified and resolved an issue in the Ouroboros.Network.Server.Simple.with function. This bug broke cardano-tracer component, [on#5224], [on#5223]. The function hasn't been used in cardano-node before. ouroboros-network-framework-0.19.2.0 was released ([chap#1157]).

We'd like to point out that cardano-node-10.6 will come supporting only P2P network stack.

For this coming release, we addressed some corner cases in topology parsing, cn#6304.

DMQ-Node

We continued working on dmq-node. Recently, an end-to-end test was successfully run using the mithril Rust client, submitting signatures to a dmq-node, which propagated them to another instance of dmq-node. on#5203

We found out and fixed a bug in typed-protocol's annotated driver, which is used by dmq-node: on#5207.

A static build of dmq-node is now also available for the x86_64-linux-musl target:

nix build .#dmq-node-static

The trace & monitoring team is helping us to integrate dmq-node with the cardano-tracer library. The aim is to give SPO the familiar user experience when monitoring dmq-node.

Peer Selection Improvements

A number of discrepancies were found and fixed in the peer selection logic. The peer-selection and net-sim tests were improved: on#5209, on#5232

WASM support in Ouroboros-Network

The node team contributed a partial WASM support to ouroboros-network, on#5229.

Technical Debt Reduction

Ouroboros-Network

We reorganised the ouroboros-network package structure to improve maintainability and simplify our release process, on#5200.

  • cardano-diffusion: everything related to diffusion for Cardano purposes, in particular cardano-node, but not only.
  • ouroboros-network-api, ouroboros-network-framework, ouroboros-network-protocols packages are now sublibraries of ouroboros-network. Some Cardano-specific APIs are only present in cardano-diffusion.
  • cardano-client package is now just a sublibrary: cardano-diffusion:subscription.
  • cardano-ping will become a sublibrary: cardano-diffusion:ping once on#5205 is merged.

Consensus

We addressed some outstanding TODOs in the ouroboros-consensus-diffusion package, see oc#1660.