High level summary​

In the current sprint the networking team focused on fixing bugs and pushing forward implementation of eclipse evasion. We also found a bug in our simulation testing setup (in integration of test node). We also overviewed the work on extending handshake protocol which is delivered by Galois Inc.

We published ouroboros-network-0.4.0.1 and ouroboros-network-protocols-0.3.0.0 to CHaP.

We also fixed a bug in cardano-node which results in not being able to configure inbound connection limits, see PR #4902.

Together with Karl Knutsson (CF) we realised an issue in cardano-cli: it's validation of DNS names, IP address & ports when registering a stake pool should be more strict to protect against common mistakes which we identified on the chain. See issue #4929.

Detailed work log​

In PR #4385 we fixed two bugs in peer state actions. First one results in a busy loop if demotion from hot to warm times outs. This busy loop is eventually exited when mux exits (we reported this in our previous report). This fix made it to 1.35.6 release as well.

In addition the PR #4385 also fixes another bug which results in hot -> warm -> hot demotion / promotion busy loop.

The PR #4385 also fixed a bug in a node only used in simulation which resulted in not using chain-sync or block-fetch mini-protocols. In the review process, we realised that the header-body split in the simulated node requires further work (see PR #4419, which is under review).

The PR #4385 also extend our generators, which together with the above fix, cover the hot -> warm -> hot demotion / promotion busy loop.

In PR #4419 we introduce a ChainDB for our simulation node, which plays similar role to ChainDB in the ouroboros-consensus: a persistent (across simulated restarts) store of blocks which does chain selection. This ensures that the simulated node is using block-fetch to download blocks announced by chain-sync mini-protocol.

We also made progress with reviewing PR #4019 - peer sharing.

We also fixed issue #4370 - a connection manager test failure, see PR #4384.

High-level summary​

This week, the Hydra team continue closing many gaps and aligning the implementation with the specification. More over, they have groomed a plan to make Hydra Mainnet compatible. Also, they continue moving forward with the internal auditing and have published some auditing guidelines to receive contributions from the community.

What did the team achieve this week​

• Completed #452 (what is remaining in there)
• Aligned the Head protocol logic implementation with the specification
• Completed full minting policy implementation and spec #720
• Clarified message authentication with researchers
• Groomed what is left to do for Mainnet compatibility #713 and drafted a 0.10.0 version
• Added a tutorial by @perturbing to the website LINK
• Published auditing guidelines LINK
• Made hydra-node work for macos #746 and added support for building on aarch64 #673
• Met with a potential customer of Hydra for Payments

What are the goals of next week​

• Have the monthly review meeting incl. the report
• Have smoke tests fixed and running regularly
• Release 0.9.0 with updated scripts
• Redraw transaction graphs for specification

High level overview​

The Mithril team released a new 2306.0 distribution that implements minor fixes and updates and completes the wiring of the era activation markers reader into the signer and aggregator nodes. They also designed an event store in the aggregator and completed its implementation as a monitoring solution to provide figures for deployment rates of versions on signer nodes before activating an era switch. Additionally, the team created a dedicated command on their nodes’ CLI that allows generating and signing an era marker’s payload to be stored on the Cardano chain, as well as creating dynamic cases for the end-to-end tests they run on the CI.

Low level overview​

• Released the new distribution 2306.0
• Completed the epic that implements signer versions deployment monitoring #718:
  • Completed the implementation an event producer/consumer via channel #738
  • Completed the creation of a database and its configuration to save the events on the consumer side #740
  • Completed the creation of events and sending them on the channel on the producer side #741
  • Completed the creation of the signer registration event #742
  • Completed the creation a query to extract the node versions stakes distribution #743
• Worked on the epic that implements eras behavior switch #707:
  • Completed the loading of era reader adapters from config in the signer and the aggregator #732
  • Completed the implementation of an era cli command in the aggregator #755
  • Completed the implementation of a dynamic matrix of cases in CI end to end tests #760
• Fixed some bugs:
  • Fixed the unsupported unixepoch() function #757
  • Fixed the problem that prevented some signers from signing on the testing-preview network #730
  • Update SQLite version to 3.40 in aggregator infrastructure #765

• SECP benchmarking: we concluded our benchmarking runs and analyses of the new SECP primitives for the Valentine hard-fork.
• Release benchmarking: we performed a round of benchmarks for the 1.35.6 release.
• UTxO-HD benchmarking: we performed first runs for UTxO-HD and are currently refining the benchmarking setup.
• New tracing: for better accessibility, the new tracing system is being outfitted with introspective capabilities.
• Infrastructure: with the Nomad cloud workbench backend we were able to perform our first test cluster runs successfully on SRE infrastructure.
• Infrastructure: the initial NixOps workbench backend has been completed; a PR containing this work, along with many quality-of-life improvements of our tooling, got merged.

Performance

SECP​

1. For SECP, we settled on a fixed tx count per block, while simultaneously spending as much as possible of the block budget. Thus we were able to minimize the impact of per-SC-call overhead.
2. The final runs were performed with various fractions, e.g. half, of the current block budget to ascertain how these workloads would fare compared to a value-only run.
3. The SECP machinery and profiles are currently being generalized into an approach to aim for very specific aspects of a smart contract for benchmarking.

UTxO-HD​

1. After analyzing initial UTxO-HD runs, it turned out that mempool snapshotting had to be throttled for benchmarking; it affects a lock that UTxO-HD had to introduce into the forging loop.
2. We're currently adapting the benchmark setup to that, and will then perform a new combination of baseline and UTxO-HD runs.

1.35.6 release​

Benchmarking the 1.35.6 release candidate could attest to a perfectly clean bill of health.

Tracing

Work on the new tracing system's introspective capabilites is ongoing: Immediate use cases of the new API include being able to statically validate generated tracer documentation, as well as providing information of a specific tracing setup in the node via traces themselves. These features will make the new system both more robust, and more accessible.

Infrastructure

Nomad backend​

1. Work on the cloud deployment capability of the Nomad workbench backend continued; for testing we can automate multiple Nomad clients.
2. Locality assumptions were removed and job monitoring was refactored.
3. To facilitate directly-executable derivations, Nomad Job specification files are now self contained with GitHub references and configs needed to run a cluster.
4. We're currently evaluating different options for genesis distribution in said cluster.

NixOps backend​

The NixOps workbench backend has reached an initial functional stage. Consequently, the relevant PR was merged. It also contained many improvements to our analysis tooling, as well as a structural overhaul of workbench itself. We consider this an important step of future-proofing our benchmarking machinery.

High level summary​

During the past two weeks we got the results from the system level benchmarks for UTxO HD. They showed a substantial performance regression, so we spent some time analyzing the results. We found out the frequency at which ledger snapshots were taken was too high, so we requested the benchmarking team a new run with a more realistic snapshotting policy. We continued refactoring and improving the prototype, and we released UTxO-HD related packages to CHaP.

We met with IOG researchers and networking specialists to discuss the Genesis design, which was well received. We continued working on testing and benchmarking different Genesis prototypes.

We are also working on solving a test failure related to iterators. This work derived in several improvements such as better documentation, a framework for writing unit (and regression) tests, and the possibility of debugging QuickCheck counter examples in the REPL.

Finally, we released ouroboros-consensus 0.2.0.0 and ouroboros-consensus-cardano 0.3.0.0 to CHaP

Workstreams​

UTxO HD Prototype​

We got the results of the first system level benchmarks for UTxO HD. They seemed to indicate a significant regression in performance. After looking into the benchmark logs we found that the benchmark runs took ledger state snapshots too often, due to the default snapshotting policy depending on k, and k being so small in the benchmark runs. Therefore, the next step is to re-run the benchmarks with a snapshotting policy that more closely resembles the one from mainnet.

At the same time, we continued refactoring and cleaning up the prototype.

Also, we prepared the anti-diff packages (fingertree-rm, diff-containers, simple-semigroupoids) and the lmdb related packages (cardano-lmdb and cardano-lmdb-simple) to CHaP.

Genesis​

The Genesis design was presented to the IOG researchers and Peter Thompson from NSol. It was well received. They pointed out one blindspot, but we think it'll be relatively simple to mitigate.

In parallel, we continued developing test and benchmarks for the Genesis prototypes. I particular we tested and implemented a potential fix for increased ChainDB dequeue timings, which partly behaved as we expected, but still needs further investigation. Also we obtained new benchmarking data for the prototype.

Technical debt​

Related to #4183, we developed a DSL for specifying ChainDB unit tests. This will allow us to better understand the counter-examples returned by QuickCheck tests, and to write regression tests for them. Also, we added a module to enable QuickCheck counter-examples to be run on the REPL, allowing for faster debugging feedback. Also, we improved the documentation related to followers (#4372).

We are also working on a design for optimizing the way we handle blocks from the future.

Support​

We released ouroboros-consensus 0.2.0.0 and ouroboros-consensus-cardano 0.3.0.0 to CHaP. Remember that we decided to split the packages related to Consensus into two bundles, one with the core functionality, Cardano-agnostic code, and another bundle with instantiations specific to Cardano.