High level summary​

1. SECP benchmarking: we ran several rounds of SECP benchmarks, refining the benchmark setup as we discovered the properties of the system. After formulating an initial suggested change to the protocol parameters, we're currently running what we consider the final benchmark, to validate the underlying assumptions.
2. Release benchmarking: we've performed a round of benchmarks for the hotfix 1.35 release update and initiated the 1.35.6 benchmarks.
3. New tracing: the improvement in the tracing API, with the underlying restructuring, was completed and merged into the node.
4. New tracing: before going live, we're performing the documentation update, as well as reworking the end user migration guide.
5. Open sourcing: the benchmarking data publishing has been completed and deployed. After populating it with relevant benchmark data and providing basic user documentation we can go live.
6. Infrastructure: the cloud workbench backend is progressing well, the networking aspects of multi-region deployment are currently being worked on.
7. Infrastructure: the NixOps workbench backend is still being worked on, as part of migration from cardano-ops and benchmarking infrastructure unification.

Performance​

We are approaching the end of a chain of SECP benchmarks, as we gradually eliminated deficiencies in the setup as we were discovering them and answering newly appearing questions:

• we improved the tx/block filling strategy in the generator, to maximise the per-block utilisation of resources and so better approximate the worst-case,
• after a discovery of what looked like significant per-SC-call overhead, we again tweaked the the tx/block filling strategy,
• finally, we're redoing all benchmarks together with a value-only run against the backdrop of Mainnet-sized datasets, to balance the suggested adjustment. That also ran into difficulties wrt. limitations of our benchmarking hardware.

In addition, we started benchmarks of the 1.35.6 release.

Tracing​

A rework of the new tracing system's internals and API was merged. It extended the system with introspection, which enabled a range of improvements, some of which were implemented along the way.

Specifically, we were able to completely short-cut processing of messages generated by the tracers that were made provably ineffective by current tracing configuration. Further, now ongoing work enabled by the introspection facilities, includes static validation of documentation and enhanced node state reporting.

Infrastructure​

On the opensourcing/transparency front, the benchmark data publishing machinery was finally fully assembled and put online. As resources permit, we'll work on populating it with benchmarking data, preparing basic documentation and engaging the stakeholders.

The work on the cloud deployment capability of the Nomad workbench backend continued with focus on setting up inter-node networking and removal of locality assumptions. A major step besides those, was completion of a switch-over to the directly-executable derivations, which eliminate the need for creation and distribution of images -- thereby increasing the speed of deployment.

The Nixops workbench backend progressed steadily, reaching minimal deployment capability. The remaining parts are proper shared configuration generation, and porting of the run control functionality from cardano-ops.

High level summary​

The open fronts that the crypto team is working on are:

• Mithril: We are creating helper functions to single out the usage of unsafe to facilitate auditing. We are also preparing a RFP for an audit of mithril's core library. Exploring future paths of mithril.
• cardano-base: Decision of whether to continue with BLS12-381 or switch curves. Conversion Praos to PraosBatchCompat ready, as well as KES secure forgetting.
• KES agent: using snockets and making things testable in IOSim
• MuSig2: GH actions updated for checking the files whether they end with an empty line. Also, we reorganized the library.

Low level summary​

Mithril​

• Given that removing the usage of transmute really affects the benchmarks, we decided to group all unsafe functions to facilitate auditing PR#722
• We have progressed with the RFP document for the mithril-stm library. We are documenting the differences with respect with the original paper.
• We are exploring possible paths of how mithril could be used 'as-a-service'. Other projects such as sidechains or Catalyst would benefit of such a service. We are at a very early stage of brainstorming how it could work.

cardano-base​

• There has been a very thorough discussion with potential users of the BLS12-381 bindings if that is the best curve. We have considered alternatives such as Pasta curves, Pluto-Eris or BLS12-377, and considering it's trade-offs. Seems that the most interesting curve to have on main-net is still 381.
• The team is gaining expertise in SNARKs to be able to experiment with them, and conclude whether the bindings will allow for SNARK verification on main-net in a timely manner.
• The update VRF PR#341 is finally merged, and we are ready to merge PR#344, which implements conversion functions from the compatible types between Praos and PraosBatchCompat.

KES agent​

• Use of snockets to send the data directly from the socket to secure memory.
• We realized that in order for the DirectSerialise / DirectDeserialise classes to work against IOSim, we have to generalize a bunch of additional primitives
• Our plans are to: (1) Split up MonadSodium into separate typeclasses, each capturing a more sensible concern; (2) Rename those typeclasses to something that reflects their nature better.

MuSig2​

The GitHub Action linelint is used to check the files.

• A new job for linting is added to the file /.github/workflows/ci.yml. The rules are configured in the file /.github/workflows/.linelint.yml. Some files from the configuration of libsecp were failing, so in the rules in .linelint.yml the failing files are denoted to be ignored by the linter.
• Folders are reorganized. We created a folder to handle the example. This folder includes the examplemusig2.c, a distinct config.h, and helper.c. The example is enhanced by implementing the functions in the helper for the configuration given in config.h. The number of messages is different than the tests. The example is made more generic to run with a loop.

The team works on applied research and consulting in formal methods that is directly applicable to evidence based engineering in Core Tech and beyond.

High level summary​

This sprint the team has been preparing several papers for peer review and knowledge sharing and consulting within IO and the community.

Details​

• R&D Seminar on experience participating in a member based organisation
• Full day tutorial on performance engineering presented at HiPEAC 2023 conference
• Finished preparing and submitted a paper on performance engineering to an ACM workshop
• Preparing a draft paper for future submission about verifying design refinements for distributed system design
• Consulting on performance design of other IO projects

High-level summary​

This week, the Hydra team completed the Hydra specification with a section about rollbacks and updated graphs (#448). In a next step, the specification will be made more approachable and an open standard. They improved their hydra-cluster tool to launch a local --devnet sandbox environment and continued aligning the hydra-plutus scripts with the specification by hardening the checks on close and contest transactions.

What did the team achieve this week​

• Completed the specification with a section about rollbacks and updated graphs #448, with a follow-up on making it more approachable and an open standard.
• Continued spec review with internal auditor and incorporated changes.
• Talked to TxPipe about how Demeter and Hydra could work together
• The hydra-cluster executable can be used to launch a local --devnet sandbox environment.
• Reproduced the “head being stuck on network outage” bug
  • relates to what liveness guarantees we can / want to give on the protocol level and needs to be discussed.
• Progressed with higher velocity by addressing more and more gaps #677

What are the goals of next week​

• Upstream our JSON instances to the ledger
• Close all transaction security related on-chain gaps
• Ideally release 0.9.0 with updated scripts
• Groom & plan (or not) protocol changes yielded from specification discussions -> updated roadmap

High level summary​

Much of the work the past two weeks involved integration efforts, cleaning up and debugging some serialization issues, adding tests, and work on large projects that are still ongoing. We also released a CIP this week that aims to make the ledger a registered CIP category.

Lower level summary​

Ledger evolution CIP​

We published a CIP that will make the Cardano ledger a registered category of the CIP process.

Serialization issues​

We had previously thought that we had found a serialization problem with the redeemers, but it turned out to just be particularly confusing code. We have now clarified the issue for the future.

See pull-3263, pull-3269, and pull-3268.

Starting in protocol version 9, we will no longer accept duplicate keys in CBOR maps.

See pull-3277.

New tests​

We added a new property test suite for some of our custom containers.

See pull-3270.

Progress on a better cost model serialization situation.​

We are still working our way through issue-2902. Towards this end, we are now properly gating the new flexible encoders until version 9.

See pull-3274.

Nightly tests​

We are still experimenting with moving more CI actions to GitHub actions.

See pull-3276.

Constraint based generators​

We continue to add to our proof of concept for constraint based generators. See the previous ledger update for more information about this project.