High level summary​

We were able to successfully run the system-level benchmarks for the UTxO-HD implementation, for the first time. There was an important regression in block forging performance that will have to be addressed before UTxO-HD is released. We also revisited the implementation of our query processing logic, which was needed to address the performance regression found in the query-by-address command. The preliminary performance results show that now the performance of this query is on-par with the Cardano baseline version, but we need further confirmation. On the Genesis front, we presented the grinding-aware safety argument for the proposed historical Cardano Genesis windows to the IOG Researchers. The Consensus release engineer finished his rotation: version 8.3.0-pre of cardano-node is releasing 2023 September 5.

UTxO-HD​

• We ran the first successful system-level benchmarks for UTxO-HD (see #203) using the in-memory backend.
  • We observed a factor 12 regression in the forging performance, which we will have to address. There are strong indications that the regression is due to the backing store accesses that take place when taking a mempool snapshot.
  • After the mempool regression is fixed the benchmarks need to be ran again.
  • System-level UTxO-HD benchmarks with the LMDB are still pending.
• UTxO-HD will eventually be necessary due to the growth of the UTxO set and other ledger state structures that live in memory at the moment. However, we are trying a strategy by which we could preserve the baseline performance of the node, in case SPOs and other node users are not ready to migrate yet (see #344).
• We implemented a new way of processing queries at the hard-fork block level, which resolves the performance regression observed in GetUTxOByAddress (see this comment). Preliminary results are promising.
• Regarding the roll out plan, UTxO-HD requires a significant change in the Consensus codebase. Even though we might be able to hide any potential performance impact in the node by keeping all data in memory (#344), the Consensus component was significantly changed, so we might have to postpone releasing this feature to mitigate any risks of conflicting with the implementation of CIP-1694 and release of Conway.

Tech debt​

• We added tests that Consensus emits valid CBOR (#3099). This helped us detect a couple of serialization bugs. The tests still need to be merged into the main branch (#323).

Support​

• Nick Frisby finished his release engineer rotation; cardano-node 8.3.0-pre is releasing 2023 September 5.
• We helped to investigate a protocol version bug in Sanchonet (see #3491).
• We started to implement the Network interface for bootstrap peer functionality, from which Genesis will benefit as well (see #91.

The team works on applied research and consulting in formal methods that is directly applicable to evidence based engineering in Core Tech and beyond.

High level summary​

The team is currently formalising mini protocols and also further developing the performance modelling prototype.

Details​

• finalising a presenting performance analysis internship work to the formal methods team

• developed a new Isabelle mini-protocol framework and examples

• planning an extended version of the ICE DeltaQ paper

• working on algebraic rules for properisation of any-to-finish

High-level summary​

This week, the Hydra team focused primarily on changes needed in the network layer and have the first draft document related to needed design. They also improved the user experience by allowing a commit using inline datums. Discussed the off-chain governance with researchers and improved internal model tests.

What did the team achieve this week​

• Monthly report published
• Small changes to hydraw and tutorial in light of the Masterclass
• Investigated a bug and saw it was solved by recent developments
• Improved the model tests by fully validating L1 transactions
• Enhanced the /commit API to also allow commit from scripts with inline datums (user request)
• Discussed off-chain governance with IOG and CF researchers
• Drafted a first network specification document in the context of Network resilience

What are the goals of next week​

• Have a clear understanding of the changes we need for the "Improve network resiliency" feature
• Groomed and agreed plan on incremental commits/decommits
• Updated tutorials including CI workflows to check consistency
• Update to GHC 9.6 and latest cardano dependencies (ledger/plutus)

High level summary​

Broadly speaking the Ledger team focused on a few main areas of Conway era:

• Creation of voting state snapshots in order to correctly delay ratification for one epoch
• Validation of the Governance Actions sequencing and ordering
• Proper expiry of DReps and Proposal Procedures
• Expanding Conway Genesis functionality
• Utilization of some of the new Protocol Parameters in ledger validation rules

Low level summary​

Conway era​

• pull-3659 - Validate Network for ProposalProcedure and TreasuryWithdrawal
• pull-3637 - Avoid using sequence of tuples, by adding GovActionId to GovActionState
• pull-3651 - Inactive DReps
• pull-3664 - Track proposal expiry
• pull-3668 - Add min committee size predicate to NewCommittee
• pull-3669 - Add Proposal deposit check against PParam
• pull-3676 - Fix inactive PoolStake not counting as Drep Stake
• pull-3635 - Make snapshots of GovActionsState
• pull-3670 - Validate previously enacted govAction
• pull-3694 - Improve error reporting on the positive coin decoder
• pull-3674 - Added RATIFY thresholds
• pull-3684 - Add proposal delaying, remove predicate failure from ENACT
• pull-3688 - DRep Refunds and update evalTransactionBalance

Improvements and releasing​

• pull-3677 - Minor patch that fixes the DRep distribution computation
• pull-3686 - Post patch release fixup
• pull-3695 - Changelog for cardano-node-8.3 release
• pull-3683 - Add two new bench mark programs

Testing​

• pull-3662 - Equality on raw types

High-level overview of sprint 43​

In this sprint, we received contributions from CF & Galois. Karl Knutsson (CF) has addressed various issues regarding peer churning in P2P, timeouts and our WireShark dissector. While the Galois developers focused on addressing issues from their review last year. See below for more details.

We continued working on bootstrap peers ouroboros-network-#4661.

We refactored our test suites: they are split into io-tests which require to be run natively on all platforms (these tests mostly contain tests that require IO system calls) and sim-tests which are platform independent. We run io-tests on all supported platforms (e.g. x86_64-linux, x86-64-darwin, aarch64-darwin and x86_64-w64-mingw32 (Windows)) natively. The sim-tests are not executed on Windows due to memory limitations on GitHub Actions runners. ouroboros-network-#4653

We also started rebasing typed-protocols refactoring branches.

Marcin was appointed as the cardano-node release engineer for the 8.4.0-pre version. So far he integrated cardano-ledger-conway-1.8 and ouroboros-network-0.9.1.0 to ouroboros-consensus, cardano-cli and cardano-api. Once we will have an integration branch for cardano-node, cardano-ledger-conway-1.8 and ouroboros-consensus packages can be released to CHaP and PRs can be merged once they go through review & CI.

We also fixed some smaller issues regarding peer sharing (both were discovered by Karl from CF). More details are included below.

Progress on P2P addoption​

SPO relays​

There are currently ~2000 relays running P2P enabled nodes that belong to 557 pools with a combined stake of 7900Mil Ada. On 16th of August it was ~1700 relays, 531 pools with a combined stake of 7700Mil Ada.

The following graphs show several different versions of relays running on the mainnet. The green line NodeToNodeVersionV10.True denotes P2P relays, which slowly increase over time. The V9 and earlier versions of the node-to-node the protocol indicates nodes version 1.35.x or earlier.

Data has been kindly provided by CF and their mainnet monitoring infrastructure.

IOG relays​

As of this week, 90% of IOG relays are running a P2P setup. In the next sprint all IOG relays will be running P2P.

Detailed description​

In this sprint, we got a few contributions from CF:

• Karl made peer churning mechanism less aggressive ouroboros-network-#4656; and
• he added timeouts for idle states in ChainSync & KeepAlive miniprotocols. These timeouts help a node remove idle connections from the responder (server) side ouroboros-network-#4648.
• he improved the WireShark dissector by adding support for the peer-sharing mini-protocol ouroboros-network-#4656.

Galois has been making progress in addressing some of the issues they raised in their review (last year):

• Mark Tullsen (Galois) has been working on refactoring the main entry function of P2P ouroboros-network-#4661 (in review);
• Sam Cowger (Galois) has been reviewing all use cases of error and unsafe calls ouroboros-network-#4661 (in review).

Peer Sharing​

• Light peer sharing is only enabled when peer sharing is turned on ouroboros-network-#4652;
• Handshake incorrectly reports peer sharing value. It's supposed to relay the remote value, but instead, it returns the local value. ouroboros-network-#4642 (in review).

Async Demotion Test Fix​

• We fixed an async demotion test failure which turned out to be a weakness of the test itself rather than a bug in the connection manager. ouroboros-network-#4655