High level summary​

This update spans four weeks, so it covers a larger than usual body of work. The headline item is that Dijkstra's batch validation now executes Plutus scripts from sub-transactions and accounts for their execution budgets. Previously it skipped Plutus execution for sub-transactions, so spending validators and minting policies inside a sub-transaction were not enforced and their ExUnits were left out of transaction and block limits and of script fees; that gap is now closed, and sub-transaction budgets count toward limits and fees. Alongside this, withdrawals handling in the ENTITIES rule was reworked to implement the full set of responsibilities from the specification, covering the network and existence checks, the legacy-mode full-drain and per-account sub-aggregate cap, and the non-legacy pre-batch existence check and batch-wide aggregate cap over the top transaction together with all of its sub-transactions. This moved the batch withdrawal checks out of UTXO and into ENTITIES, and was accompanied by conservation of value in legacy mode and by SUBUTXO/SUBUTXOW predicate failures.

A second major thread was the Plutus V4 context: we implemented the TxInfo translation for protocol version 4, redid the PlutusPurpose translation, made further TxInfo improvements, fixed the Dijkstra cost model injection, and bumped plutus-ledger-api to at least 1.68.

Progress continued on the Dijkstra era rules and on Leios. A POOL rule was added for Dijkstra, an EMax /= 0 well-formedness check was introduced, the stake calculation was made consistent by moving SNAP to the end of the EPOCH, and account balance intervals are now enforced. On the Leios side, the Leios protocol parameters were added to the Dijkstra era and the leios_certificate signer set is now size-constrained. We also removed the hardforkConwayMoveWithdrawalsAndDRepChecksToLedgerRule transition flag and the WithdrawalsNotInRewardsCERTS failure, finishing the migration of those withdrawal and DRep checks into the ledger rule.

The rest of the work spanned serialisation, more abstract interfaces, conformance, and releasing. On serialisation we adjusted transaction serialisation in the block body, made the OutputVRF decoder more resilient, removed a guard from the IndividualPoolStake codecs, and improved the custom PParamUpdate encoding. The move toward era-parametric interfaces continued with an era parameter on StakePoolParams, forcing of the Shelley initial-funds injection result, and ToJSON/FromJSON instances for EraTxOut and for TxBody/Tx across all eras. On testing we landed two formal-ledger-specification updates (one enabling the corresponding tests), conformance for the Dijkstra EPOCH rule and for starting account balance intervals, the Shelley Pool Imp tests in Dijkstra conformance, an era-generic Conway CERTS test, and a fix for a produced vs. consumed conformance failure. Finally, we updated to crypton == 1.1 (replacing memory with ram), hardened the branch-history CI job, and landed two rounds of backports of fixes for node 11.1.1.

Low level summary​

Features​

• PR-5983 - Implement TxInfo translation for PV4
• PR-6017 - shelley: force the initial-funds injection result
• PR-6031 - Add EMax /= 0 as well formed check to Dijkstra era
• PR-6012 - Conservation of value in legacy mode
• PR-5888 - Constrain the size of leios_certificate signers
• PR-6033 - Make OutputVRF decoder more resilient
• PR-5986 - Make stake calculation consistent in dijkstra by moving SNAP to the end of the EPOCH
• PR-5985 - Adjust transaction serialization in BlockBody
• PR-6001 - Enforce (starting_)account_balance_intervals
• PR-6002 - Add Leios protocol parameters to DijkstraEra
• PR-6041 - Remove hardforkConwayMoveWithdrawalsAndDRepChecksToLedgerRule and WithdrawalsNotInRewardsCERTS
• PR-6040 - Add era parameter to StakePoolParams
• PR-6046 - Bump plutus-ledger-api to >=1.68
• PR-5855 - Add ToJSON/FromJSON instances for EraTxOut
• PR-6038 - Redo PlutusPurpose translation
• PR-6030 - Fix CostModels injection for Dijkstra
• PR-6051 - TxInfo improvements
• PR-6029 - Add POOL rule for Dijkstra
• PR-5997 - Withdrawals rework in ENTITIES
• PR-6071 - Execute Plutus scripts from sub-transactions and count their ExUnits
• PR-6070 - Improve versatility of custom PParamUpdate encoding
• PR-6042 - Remove guard from IndividualPoolStake codecs
• PR-6076 - Add ToJSON/FromJSON instances for TxBody l and Tx l in all eras

Testing​

• PR-6019 - Improve tests in UtxoSpec
• PR-6022 - Update formal-ledger-specification
• PR-5995 - Update formal-ledger-specification and enable tests
• PR-6024 - Enable Shelley Imp tests for Pool in conformance (Dijkstra)
• PR-5842 - Add a submitFailingSubsetTx helper
• PR-6009 - Make Conway CERTS test era-generic
• PR-6056 - Conformance for EPOCH rule in Dijkstra
• PR-6044 - Conformance for starting account balance intervals
• PR-6034 - SUBUTXOW predicate failures
• PR-6055 - SUBUTXO predicate failures
• PR-6074 - Fix conformance failure in produced vs. consumed

Infrastructure and releasing​

• PR-6014 - Make the branch-history CI job more robust
• PR-5999 - Update to crypton == 1.1 and replace memory package with ram
• PR-6025 - Backport of fixes for node 11.1.1
• PR-6028 - Backport of fixes for node 11.1.1 - take 2
• PR-6032 - Add changelog for cardano-ledger-shelley-1.19.0.1 backport

High level overview​

This week, the Mithril team completed the missing IVC off-circuit checks, the property-based tests for the IVC pure helpers, the caching of the circuit keys with the reuse of the SNARK prover setups, and the enhancements of the end-to-end tests for the IVC SNARK. They continued implementing the circuit verification key whitelist/revoke mechanism and supporting an IVC follower aggregator joining the network at any time. They also started implementing the Relation trait for the IVC circuit and adding a Proof of Bound Possession for the SNARK verification keys.

The team completed the IPFS support in the end-to-end tests, the test infrastructure for IPFS nodes, and the summary of the IPFS PoC. They also started adding a certificate chain cache committed after chain verification in the client library.

Finally, they kept upgrading to Cardano node 11.1, reduced the aggregator compile time, and kept investigating the blocks and transactions verification failure on the testing-preview network.

Low level overview​

Features​

• Completed the issue Add missing IVC off-circuit checks #3381
• Completed the issue Add property-based tests for the IVC pure helpers #3469
• Completed the issue Cache the circuit keys and reuse the SNARK prover setups #3424
• Completed the issue Enhance end-to-end tests for IVC SNARK #3390
• Completed the issue Implement IPFS in the end to end tests #3460
• Completed the issue Prepare summary for IPFS PoC #3519
• Worked on the issue Implement circuit verification key whitelist/revoke mechanism #3148
• Worked on the issue Support an IVC follower aggregator joining the network at any time #3446
• Worked on the issue Implement Relation trait for the IVC circuit #3428
• Worked on the issue Add a SNARK verification key Proof of Bound Possession #3537
• Worked on the issue Add a certificate chain cache committed after chain verification in the client library #3520

Protocol maintenance​

• Completed the issue Implement test infrastructure for IPFS nodes #3461
• Completed the issue Reduce the aggregator compile time #3538
• Completed the issue Publication to doc.rs failed for distribution 2630 #3507
• Worked on the issue Upgrade to Cardano 11.1 #3346
• Worked on the issue Blocks and transaction verification fails on testing-preview #3526

High level overview​

This week, the Mithril team completed the behavior tests for the IVC wiring, the harmonization of the recursive and non-recursive SNARK implementations in STM, and the STM README with recursive and non-recursive examples. They continued adding the missing IVC off-circuit checks, implementing the circuit verification key whitelist/revoke mechanism, and supporting an IVC follower aggregator joining the network at any time. They also started adding property-based tests for the IVC pure helpers.

The team completed the adaptation of the documentation for the on-chain configuration parameters. They also completed the IPFS file downloader of the client and started implementing the IPFS support in the end-to-end tests.

Finally, they kept upgrading to Cardano node 11.1, started fixing the failed publication of the distribution 2630 to docs.rs, and started investigating the blocks and transactions verification failure on the testing-preview network.

Low level overview​

Features​

• Completed the issue Add behavior tests for the IVC wiring #3465
• Completed the issue Harmonize the recursive and non-recursive SNARK implementations in STM #3421
• Completed the issue Adapt documentation for on-chain configuration parameters #3398
• Completed the issue Implement an IpfsFileDownloader in the client #3459
• Completed the issue Update STM README with recursive/non-recursive examples #3385
• Worked on the issue Add missing IVC off-circuit checks #3381
• Worked on the issue Implement circuit verification key whitelist/revoke mechanism #3148
• Worked on the issue Support an IVC follower aggregator joining the network at any time #3446
• Worked on the issue Add property-based tests for the IVC pure helpers #3469
• Worked on the issue Implement IPFS in the end to end tests #3460

Protocol maintenance​

• Worked on the issue Upgrade to Cardano 11.1 #3346
• Worked on the issue Publication to doc.rs failed for distribution 2630 #3507
• Worked on the issue Blocks and transaction verification fails on testing-preview #3526

High level summary​

Casing on Data has been merged: case on a Data.Constr value now dispatches directly on the constructor tag, which, combined with the recently added dropList, significantly reduces the cost of consuming Data-encoded values. This is a Plutus V4 feature, and we are also working on specifying casing on built-in types in the Plutus Core specification.

Work on the Value builtins (CIP-0168) continues: the keepPolicies and dropPolicies builtins have been added, the cost model for policies has been merged, and the cost models for keepPolicies and dropPolicies are in progress. Like the other new builtins, these are gated behind a future protocol version and cannot be used on-chain until Dijkstra.

On the Plutus V4 ScriptContext front, we are implementing a different encoding for product types using List instead of Constr, which makes them cheaper to decode. We are also improving the usability of the certifier in the uplc and plc tools.

Key Pull Requests Merged​

• Add Data.Constr casing to Case
• Add keepPolicies and dropPolicies builtins (CIP-0168)
• Cost model for policies (CIP-0168)
• Remove deferred builtin applications from the specification

Pull Requests In Progress​

• Builtin Pattern Matching (CIP-0194)
• Specify casing on built-in types
• Cost model for keepPolicies (CIP-0168)
• Cost model for dropPolicies (CIP-0168)
• Encode single constructor ledger types using Data.List
• Improve uplc/plc certifier usability
• Compile BuiltinData transparently in the plugin

High level summary​

• Leios prototype development (Treasury Funding Initiative 4: Ouroboros Leios Implementation):
  • Rewrote the logic that decides which Leios data to download from which peer. A node now bounds both the time it spends on each decision and the memory it holds for pending downloads. It also prioritises the newest Endorser Blocks, and randomises the order in which it asks for missing transactions, so that several requests do not saturate the same limits at once (#2237, for ouroboros-leios#1021).
  • Added a cache that holds the transactions of recent Endorser Blocks. A node that already saw a transaction does not need to download it again. The download logic reads from this cache and from the mempool (#2188, for ouroboros-leios#1021).
  • Fixed a second cause of unwanted disconnects between honest nodes. A syncing node has a chain tip that is too old to check young Leios messages. Such a node now stops asking for those messages until it catches up, instead of rejecting the replies (#2255, cardano-node#6673).
  • Capped the time the mempool spends to read a ledger state from disk. A block producer that waits on disk no longer delays the forging of its own block past its slot (#2217, for ouroboros-leios#911).
  • Extended the call tracing of the block forging and chain selection threads, and improved the dashboard that shows the results. This tells us where a prototype node spends its time on the Leios devnet (#2183, ouroboros-leios#1088, for ouroboros-leios#911).
  • Moved the classification and rendering of Leios trace messages from cardano-node into the consensus layer. We can now add or change a Leios trace message without a matching change in cardano-node (#2263).
  • Integrated the two changes above into the prototype node used on the devnet (cardano-node#6681).
• Storage layer (Treasury Funding Initiative 10: LSM including UTXO-HD):
  • Ledger state garbage collection now runs in its own thread. Before this change, the delay before a snapshot also blocked garbage collection, so a node kept every ledger state in memory until the snapshot completed (#2227).
  • Turned off IO wait accounting on the LSM storage backend. The accounting itself was a source of measurement error, see well-typed/blockio-uring#55 (#2247).
• Maintenance and support (Treasury Funding Initiative 17: Maintenance and Support):
  • Released ouroboros-consensus 4.2.0.0 (#2242).
  • Fixed a silent failure in the GetGenesisConfig query. Version 1.19.0.0 of cardano-ledger-shelley added a field to the Shelley genesis record without a change to the node-to-client protocol version. A client that linked the new ledger then waited forever for bytes that a node with the old ledger never sends. cardano-wallet hit this against node 11.0.1 and stopped following the chain. The query now sends the original 15 fields again, and the decoder accepts a reply from an already deployed node, so a client needs no node upgrade. The fix shipped in ouroboros-consensus 4.2.1.0 (#2251, #2257, #2258).