High-level summary​

This week, the Hydra team focused their efforts on continuing to investigate and experiment operating a head on mainnet. They collected several bugs and issues and worked on fixing them. Now the team is soon to release a new version, 0.11.0, which comes with a lot of improvements and bug fixes.

What did the team achieve this week​

• Restored and fixed a bug which stalled our head on mainnet #927
• Solved one user issue #914
• Reduced significantly local state size and logs by removing the full scripts from it #928
• (pending review) Reduced snapshot size in the API, by only including tx ids #922

What are the goals of next week​

• New release 0.11.0
• Monthly report & review meeting.
• Fix some minor bugs discovered when operating our head on mainnet
• Complete journey for external commits using multiple script UTxOs #903
• Publish benchmarks and provide regular benchmarks for Hydra #186

High level overview​

The Mithril team completed the implementation of the new sub-command for restoring a Mithril stake distribution in the client. They also updated the client’s developer documentation and architecture documentation, and did some refactoring on the client and its dependency injection mechanism. Additionally, they completed and deployed infrastructure enhancements on the test Mithril networks. They also completed the performance tests of the new stake distribution computation on the Cardano mainnet.

Finally, the team worked on fixing a bug on the client multi-platform test, a bug on the aggregator state machine, and some flakiness on the CI.

Low level overview​

• Completed on the epic that designs and implements generic signing/verification of entity services #780:
  • Completed the issue Create the sub-command for 'Mithril Stake Distribution' in client #896
  • Completed the issue Adapt end to end tests to handle new types of data #899
  • Completed the issue Update client documentation #897
  • Completed the issue Update architecture documentations for new types of data #898
  • Completed the issue Refactoring client #960
• Worked on the epic that prepares the Mithril infrastructure for mainnet #767:
  • Completed the issue Enhance terraform infrastructure #930
• Completed the epic that implements the computation of the stake distribution for mainnet #880:
  • Completed the issue Check performance impact of new stake distribution command on the 'mainnet' #962
• Worked on the epic Prepare Mithril Signer deployment model for SPO #862:
  • Worked on the issue Design recommended deployment model for SPOs on 'mainnet' and 'preview'/'preprod' #961
• Worked on bugs and optimizations:
  • Completed the issue Aggregator does not always detect new immutable file #953
  • Completed the issue CI tests fail with Rust '1.70.0' #958
  • Worked on the issue End to end tests are flaky #954
  • Worked on the issue Certificate dates in metadata are not on the same timezone #946
  • Worked on the issue Refactor 'MithrilStakeDistribution' entity #967
  • Completed the issue Fix 'Mithril Client multi-platform test' with new client interface #956
  • Completed the issue Enhance 'ImmutableDigesterError::NotEnoughImmutable' error #969
  • Completed the issue Client 'snapshot download' command fails with option '--download-dir' #979

High level summary​

The Consensus team had a very productive meeting with IOG Researchers. We now seem to be in alignment in regards to a strong argument that the Byron and TPraos eras do not need to be checkpointed for an MVP. There is one remaining question (which applies also to the Praos era): how to assess the threat that short forks pose against historical windows that underperformed? We are currently collaborating on that. We also drafted an argument that the updated "Limit on Patience" timeout sufficiently bounds how long the adversary can inflate a victim's overall sync time.

On the UTxO-HD front, the prototype branch was rebased on top of the latest ouroboros-consensus main branch and integrated on top of cardano-node 8.1.1-pre. As a result, the mempool fairness fix that was released recently is now integrated into UTxO-HD. We managed to run a node again with UTxO-HD enabled. We also identified a race condition in the UTxO-HD prototype and fixed it. In addition, we started performing UTxO-HD ad-hoc benchmarks for cardano-node, which uncovered a performance regression on the Network component when using GHC-9.2/9.4. This is being addressed.

Regarding our support activities, we Released fs-sim-0.2.0.0 and are in the process of preparing the 8.2 release of cardano-node. We also identified and started fixing incorrectly-unevaluated thunks in preparation for enabling CI NoThunks tests.

High level summary​

• Benchmarking: We've continued release benchmarking and established a new baseline for 8.0.0.
• New tracing: Our benchmarking profile for measuring new vs. legacy tracing performance has been refined.
• Nomad backend: The healthcheck system for the the nomad cloud has been completed. We've performed the first full runs on the new backend.

Low level overview​

Benchmarking​

In our release benchmarking cycle, we established a new performance baseline for 8.0.0. Additionally, we've measured performance under various workloads for 8.1.1-pre; the results look promising and validate the optimization efforts done on several system components.

In the meantime, we've finalized a build plan with GHC9.2 that matches the current one with GHC8.10; a requirement for benchmarking as a large amount of differences in the dependency graph can confound the results for the application code proper.

Tracing​

The legacy and the new tracing system differ fundamentally in design, implementation and handling. So for metrics to be meaningful in a comparison, benchmarking profiles have to be tuned such that not only log line frequency but frequency of specific trace messages are closely aligned. We've found that higher granularity in this regard was necessary, and done additional work on our dedicated profiles.

Additionally, we've had a first glance of what additional traces could be valuable in the context of benchmarking UTxO-HD.

Nomad backend​

As the new backend's healthcheck system in its first iteration can now serve as a guardrail to ensure sanity of a full-length run, we've performed our first 52-node cluster runs on nomad cloud. We're currently smoothing the edges around cluster deployment, and analysing the metrics gathered from those runs.

This means the backend is entering validation phase, where we systematically compare all metrics taken from the new infrastructure to the existing ones, including determining reproducibility and variance.

Network Update​

Key contributions and advances​

We merged light peer sharing feature, which allows to include inbound peers into outbound governor known peers. This is the primary way for new unregistered nodes to enter the network, which then can be shared using peer sharing. Note that peer sharing is an experimental feature which is disabled until genesis & eclipse evasion as fully implemented. See #3596.

We are making progress reviewing eclipse evasion, #3886.

We fixed another bug in local root peers. We found out that if the local roots where ignored until the first domain name was resolved, see #4583. The bug fix was backported and released in ouroboros-network-0.8.1.1.

We re-started working on dynamically enabling block forging to address issue #3159, which will enable us to release P2P on block producing nodes. See #140.

New cardano-ping / cardano-cli ping release​

We prepared a new release of cardano-ping library which supports the new query feature (query supported versions). See #4589, #4593 and #5313. The new version of cardano-cli ping will use ISO8601 formatted timestamps; also the formatting of ping results is slightly improved, and it will introduce the new --query-versions (-Q) switch. If the remote site supports the query parameter, the command will print:

redacted-ip:port network rtt: 0.064
redacted-ip:port handshake rtt: 0.064010896s
redacted-ip:port Queried versions [NodeToNodeVersionV11 764824073 InitiatorAndResponder,NodeToNodeVersionV10 764824073 InitiatorAndResponder,NodeToNodeVersionV9 764824073 Initiat
orAndResponder,NodeToNodeVersionV8 764824073 InitiatorAndResponder,NodeToNodeVersionV7 764824073 InitiatorAndResponder]

otherwise it will print the negotiation results

redacted-ip:port network rtt: 0.045
redacted-ip:port handshake rtt: 0.101867615s
redacted-ip:port Negotiated version NodeToNodeVersionV10 764824073 InitiatorAndResponder

Note that in that case cardano-cli ping offers InitiatorAndResponder, which allows us to detect whether the remote side is an InitiatorOnly or InitiatorAndResponder. Also cardano-cli ping will no longer announce itself as InitiatorAndResponder, except for the case mentioned above.

Other smaller contributions​

On a request from the Marlow Team, we published haddocks of typed-protocols, which are now available here (#40, #41).

We made a new release of strict-stm-1.1.0.1 on Hackage, which fixed a bug in package description file, #101 .

We also helped to debug a deadlock when using named pipes on Windows in the new RawBearer API. The API is being used to store secret keys only in memory. The PR #4395 is under review.

We also have two more PRs which are under review:

• #4530: enabling ledger peers on a fixed number of slots before the tip of the chain;
• #4580: a PR which fixes inconsistencies in one of our cddl specs.