High level overview​

This week, the Mithril team completed adding a Proof of Bound Possession for the SNARK verification keys, renaming the future_snark feature to snark, adding the missing conversion and error-path tests in IVC SNARK, and testing the SNARK infrastructure to full security parameters with the Midnight fix. They kept working on hardening the SNARK primitives.

The team completed adding an IndexedDB backend for the certificate chain cache in the WASM client and documenting the certificate chain cache, fixed the IPFS end to end tests failing when dist.ipfs.tech is unreachable, and fixed the crates publication dry run failing in the nightly. They also worked on the new distribution release, the upgrades to Cardano node 11.1.3 and 11.2, the update of the DMQ node to 0.7.2.0, a keypair generation tool in the aggregator CLI, and kept investigating the blocks and transactions verification failure on the testing-preview network.

Finally, they completed the upgrade to Rust 1.99, and fixed the missing SPO aliases in the explorer, the flakiness of the end to end tests caused by the devnet scripts, and the cloud uploader cache which only held the first 1000 files.

Low level overview​

Features​

• Completed the issue Add a SNARK verification key Proof of Bound Possession #3537
• Completed the issue Rename future_snark feature to snark #2814
• Completed the issue Add the missing conversion and error-path tests in ivc_halo2_snark #3470
• Completed the issue Test SNARK infrastructure to full security parameters with the Midnight fix #3593
• Completed the issue Add an IndexedDB backend for the certificate chain cache in the WASM client #3524
• Completed the issue Document the certificate chain cache #3525
• Worked on the issue SNARK primitives hardening #3557

Protocol maintenance​

• Completed the issue Upgrade Rust to version 1.99 #3578
• Completed the issue IPFS e2e tests fail when dist.ipfs.tech is unreachable #3582
• Completed the issue Crates publication dry run fails in the nightly since mithril-common 0.8.0 #3581
• Completed the issue Some SPO aliases are missing in the explorer #3518
• Completed the issue Fix flakiness of the end to end tests in the CI caused by the devnet scripts #3596
• Completed the issue Fix the cloud uploader cache which only holds the first 1000 files #3594
• Worked on the issue Release 26XX distribution #3498
• Worked on the issue Upgrade to Cardano 11.1.3 #3601
• Worked on the issue Upgrade to Cardano 11.2 #3585
• Worked on the issue Update DMQ node to 0.7.2.0 #3602
• Worked on the issue Add generate keypair tool in aggregator CLI #3584
• Worked on the issue Fix make -C mithril-client-wasm test which terminates itself #3590
• Worked on the issue Blocks and transaction verification fails on testing-preview #3526

High level summary​

The Plutus team has published release 1.71.0.0, which includes the full Plutus V4 script context and will be integrated into node release 11.2. The release also adds Plutus Core language version 1.2.0 and the cost models for keepPolicies and dropPolicies (CIP-0168).

We've opened CIP-0205, which proposes to remove scope checking for Plutus scripts. As usual, we encourage everyone to read it and leave comments - we would like as much feedback as possible.

Other than these, casing on built-in types is now specified in the Plutus Core specification, and the guardrail script has been made smaller and cheaper by using BuiltinList and BuiltinPair. Work in progress includes making the uplc and plc executables easier to obtain (publishing them to CHaP, and shipping prebuilt macOS binaries built by Hydra), and bringing the specification further up-to-date.

Key Pull Requests Merged​

• Add Plutus Core version 1.2.0
• Cost models for keepPolicies and dropPolicies (CIP-0168)
• Add semvar F and G for Dijkstra
• Specify casing on built-in types
• Change ttisScriptPurposes to ttisRedeemerHashes
• Add property tests for V4 script context helpers
• Fix scope check
• Guardrail: improve cost/size using BuiltinList/BuiltinPair
• Fix plugin unable to resolve variable reference when parsing list

Pull Requests In Progress​

• Publish plutus-executables to CHaP
• Ship prebuilt executables for macOS, built by Hydra
• Apply script deserialisation bounds unconditionally
• Add plugin option to dump compilation timing

High level summary​

• Leios prototype development (Treasury Funding Initiative 4: Ouroboros Leios Implementation):
  • Moved the Leios database (LeiosDB) from the leios-prototype branch to main. The SQLite backend, which was one large module, is now split into modules by job: schema, queries, inserts, reads, the write queue, and maintenance (#2311).
  • The mempool now reserves room for an Endorser Block, in addition to the room for a ranking block. Each transaction gets a second measure, its cost in an Endorser Block, which includes its entry in the Endorser Block (a 32-byte hash and the transaction size). In the Dijkstra era, the Endorser Block capacity comes from the protocol parameters. In every earlier era it is zero, so the mempool capacity of those eras does not change (#2312, #2325).
  • A mempool snapshot can now split its transactions into two parts: the longest prefix that fits a ranking block, and the next run that fits an Endorser Block. The forge still passes a zero Endorser Block capacity, so it selects the same transactions as before (#2320).
  • Added the trace TraceMempoolCapacityChanged. The mempool capacity comes from the protocol parameters, so a parameter update or an era transition changes it, and no trace reported this before. With the Endorser Block term, one such change can be several megabytes (#2304).
  • Two performance fixes in the prototype. A mempool read no longer waits while a writer holds the mempool state (#2308). Block application now reads the LeiosDB through a read-only connection, so it does not compete with the writer thread for the database (#2315).
  • Fixed a crash in the database tools. A LeiosDB handle links its background copier thread to the thread that opened it. No tool closed its handle, so the runtime raised an exception in the copier, and the link passed it on to the tool. Every tool now closes its handle. The same change fixes three shutdown defects that closing exposed, for example a close that blocked forever (#2314).
  • The Leios CBOR decoders now reject malformed input from a peer. A hash must be exactly 32 bytes, and the decoder checks each count that a peer declares against maxTxsPerEb before it allocates memory. This closes findings LEI-004, LEI-005 and LEI-009 from the Anastasia Labs audit (#2358, for ouroboros-leios#1125).
  • To stress test the Leios testnet, LeiosNotify now sends up to 1000 requests before it waits for a reply (#2366).
  • Released the prototype versions prototype-2026w39 and prototype-2026w40a (ouroboros-leios#1117, ouroboros-leios#1131, with the node integration in cardano-node#6708 and cardano-node#6722). The first release has the mempool, database and shutdown fixes above. It also shrinks the SQLite write-ahead log back to 64 MiB after a burst of writes, for example a testnet sync (ouroboros-consensus#2332). The second release keeps votes for the 128 most recent points only, so the vote state no longer grows without limit. This bound lets us take load readings, but it does not resist an adversary (ouroboros-consensus#2367).
• Maintenance and support (Treasury Funding Initiative 17: Maintenance and Support):
  • Released ouroboros-consensus 5.0 (#2356, with the dependency updates in #2341 and #2352). The release has the reworked Peras API and drops node-to-client versions below v23. The default ledger snapshot policy now takes a snapshot every 40 * k slots, where k is the security parameter, which is one snapshot a day on mainnet. A background thread now writes the snapshots. The mempool now moves a transaction from an older era to the current era in one step: it encodes the transaction in its own era and decodes it in the current era. Before, it translated the transaction through every era in between.
  • Released ouroboros-consensus 5.1.0.0 (#2368, ported to main in #2369). At startup, the node now traces its ledger snapshot policy. The snapshot interval is in slots, but the write delay and the rate limit are in seconds. So settings that suit mainnet, where a slot is one second, can be wrong on a testnet with shorter slots. If the delay or the rate limit is as long as the interval or longer, the node emits the warning ImplausibleSnapshotPolicy.
  • Moved the tracing instances for consensus types from cardano-node into the new ouroboros-consensus:tracing sublibrary. A change to a traced type and the change to its log output can now go in one commit. Some log output changes: a mempool trace at the detailed level logs a transaction as hex CBOR instead of a Haskell Show rendering, and a rejected transaction always logs the reason (#2244).
  • The database tools now read the node configuration with cardano-config, and db-synthesizer reads its forging keys with cardano-keys. The tools now apply the same rules as cardano-node. Before, db-synthesizer could forge blocks in an era that a node with the same configuration file would not enter. One visible change: every genesis file in the configuration must now come with its hash (#2259, #2335).

High level summary​

Most of this cycle went into Leios and the block header abstraction that supports it. The EraBlockHeader interface was reshaped into a separate type class per protocol that can be specified on a per-era basis, which required a dedicated BbodySignal type and a redefined BBODY rule for Babbage, where the TPraos to Praos switch happened. Related to this, the block header now uses a more lax VersionInfo in place of ProtVer for Dijkstra and no longer relies on a self-reported version when constructing headers. On Leios itself we now select the Leios committee at the epoch boundary, expire BLS keys after KES expiry, check the BLS proof of possession in Dijkstra's POOL rule, added a lens for the announcement block header reference on LeiosEraBlockHeader, and added Dijkstra forecasting for Leios on the prototype branch. We also began laying groundwork for Peras by adding its protocol parameters, while keeping Peras certificates out of the Dijkstra block body for now.

Work on the Plutus V4 context and sub-transactions continued, with the PlutusV4 context implemented for the GuardingPurpose and the sub-transaction index now available in the Plutus context. On the Dijkstra era itself we added a DijkstraEraTx type class with a hashTx function, fixed a superclass constraint in DijkstraEraPParams, added reference-input cost protocol parameters (ppRefInputsCostPerMultiAssetPolicyL and ppRefInputsCostPerDatumByteL) starting at Dijkstra, and fixed conservation of value for phase-2-invalid transactions. Elsewhere we introduced a distinct type for the pool distribution used for governance, fixed VRF reuse when re-registering a pool, and removed some unnecessary memory overhead in native scripts.

On the testing side we continued filling in predicate-failure coverage for the sub-rules and related rules (MEMPOOL, SUBGOV, SUBLEDGER), enabled tests in CertSpec, added a BBODY ImpSpec, added regression tests for conformance, LeiosCert serialization tests, and more withdrawal tests. For infrastructure and releasing we updated the Allegra changelog, bumped the CHaP index-state, and dropped cardano-crypto-leios as a source-repository-package from the cabal project.

Low level summary​

Features​

• PR-6054 - Use a distinct type for the pool distribution used for governance
• PR-6062 - Fix VRF reuse when re-registering pool
• PR-6052 - Select the Leios committee at the epoch boundary
• PR-6047 - Expire BLS keys after KES expiry
• PR-6089 - Fix superclass constraint in DijkstraEraPParams
• PR-6077 - Fix conservation of value for phase2- invalid transactions
• PR-6099 - Add DijkstraEraTx typeclass with hashTx function
• PR-6088 - BlockHeader cross era interface
• PR-6084 - Use more lax VersionInfo in place of ProtVer in BlockHeader for Dijkstra
• PR-6101 - Add ebReferencesAnnouncementBlockHeaderL to LeiosEraBlockHeader
• PR-6067 - Add Peras protocol parameters
• PR-6068 - PlutusV4 context for GuardingPurpose
• PR-6117 - Do not use self-reported version when constructing block headers
• PR-6064 - [Leios Prototype] Add Dijkstra forecasting for Leios
• PR-6110 - Add ppRefInputsCostPerMultiAssetPolicyL and ppRefInputsCostPerDatumByteL protocol parameters starting at DijkstraEra
• PR-6085 - Check BLS proof of possession in Dijkstra's POOL rule
• PR-6121 - Sub Tx index in plutus context
• PR-6122 - Remove unnecessary memory overhead in NativeScripts

Testing​

• PR-6082 - Add regression tests for conformance
• PR-6091 - Add LeiosCert serialization tests
• PR-6095 - MEMPOOL predicate failures
• PR-6083 - SUBGOV predicate failures
• PR-6115 - Enable tests in CertSpec
• PR-6116 - BBODY Impspec
• PR-6111 - Additional withdrawal tests
• PR-6112 - SUBLEDGER predicate-failure tests

Infrastructure and releasing​

• PR-6087 - Update Allegra changelog
• PR-6107 - Update CHaP index-state and remove cardano-crypto-leios as SRP from cabal project

High level overview​

This week, the Mithril team completed the support for an IVC follower aggregator joining the network at any time, the circuit verification key whitelist/revoke mechanism, the scaling tests of the SNARK infrastructure to full security parameters, the finalization of the Mithril SNARK Book, and the activation of the dual genesis keys. They also completed renaming the circuit test assets and production keys, and identifying the circuit verification keys by their transcript representation. They worked on adding a Proof of Bound Possession for the SNARK verification keys, renaming the future_snark feature to snark, hardening the SNARK primitives and adding the missing conversion and error-path tests in ivc_halo2_snark.

The team completed leveraging the certificate chain cache with the EarlyStopVerification mode, adding a file-system backend for the cache and enabling it in the client CLI. They also started adding an IndexedDB backend for the cache in the WASM client and documenting the certificate chain cache.

Finally, they completed the upgrade to Cardano node 11.1 and kept investigating the blocks and transactions verification failure on the testing-preview network.

Low level overview​

Features​

• Completed the issue Support an IVC follower aggregator joining the network at any time #3446
• Completed the issue Implement circuit verification key whitelist/revoke mechanism #3148
• Completed the issue Test scaling SNARK infrastructure to full security parameters #3153
• Completed the issue Finalize the Mithril SNARK Book #3383
• Completed the issue Activate dual genesis keys #3388
• Completed the issue Rename the .bin circuit test assets and production keys #3552
• Completed the issue Identify circuit verification keys by transcript representation instead of bytes #3556
• Completed the issue Leverage the certificate chain cache with the EarlyStopVerification mode #3521
• Completed the issue Add a file-system backend for the certificate chain cache #3522
• Completed the issue Enable the certificate chain cache in the client CLI #3523
• Worked on the issue Add a SNARK verification key Proof of Bound Possession #3537
• Worked on the issue Rename future_snark feature to snark #2814
• Worked on the issue SNARK primitives hardening #3557
• Worked on the issue Add the missing conversion and error-path tests in ivc_halo2_snark #3470
• Worked on the issue Add an IndexedDB backend for the certificate chain cache in the WASM client #3524
• Worked on the issue Document the certificate chain cache #3525

Protocol maintenance​

• Completed the issue Upgrade to Cardano 11.1 #3346
• Worked on the issue Blocks and transaction verification fails on testing-preview #3526