Consensus Team Update

High level summary

• Leios prototype development (Treasury Funding Initiative 4: Ouroboros Leios Implementation):
  • Moved the Leios database (LeiosDB) from the leios-prototype branch to main. The SQLite backend, which was one large module, is now split into modules by job: schema, queries, inserts, reads, the write queue, and maintenance (#2311).
  • The mempool now reserves room for an Endorser Block, in addition to the room for a ranking block. Each transaction gets a second measure, its cost in an Endorser Block, which includes its entry in the Endorser Block (a 32-byte hash and the transaction size). In the Dijkstra era, the Endorser Block capacity comes from the protocol parameters. In every earlier era it is zero, so the mempool capacity of those eras does not change (#2312, #2325).
  • A mempool snapshot can now split its transactions into two parts: the longest prefix that fits a ranking block, and the next run that fits an Endorser Block. The forge still passes a zero Endorser Block capacity, so it selects the same transactions as before (#2320).
  • Added the trace TraceMempoolCapacityChanged. The mempool capacity comes from the protocol parameters, so a parameter update or an era transition changes it, and no trace reported this before. With the Endorser Block term, one such change can be several megabytes (#2304).
  • Two performance fixes in the prototype. A mempool read no longer waits while a writer holds the mempool state (#2308). Block application now reads the LeiosDB through a read-only connection, so it does not compete with the writer thread for the database (#2315).
  • Fixed a crash in the database tools. A LeiosDB handle links its background copier thread to the thread that opened it. No tool closed its handle, so the runtime raised an exception in the copier, and the link passed it on to the tool. Every tool now closes its handle. The same change fixes three shutdown defects that closing exposed, for example a close that blocked forever (#2314).
  • The Leios CBOR decoders now reject malformed input from a peer. A hash must be exactly 32 bytes, and the decoder checks each count that a peer declares against maxTxsPerEb before it allocates memory. This closes findings LEI-004, LEI-005 and LEI-009 from the Anastasia Labs audit (#2358, for ouroboros-leios#1125).
  • To stress test the Leios testnet, LeiosNotify now sends up to 1000 requests before it waits for a reply (#2366).
  • Released the prototype versions prototype-2026w39 and prototype-2026w40a (ouroboros-leios#1117, ouroboros-leios#1131, with the node integration in cardano-node#6708 and cardano-node#6722). The first release has the mempool, database and shutdown fixes above. It also shrinks the SQLite write-ahead log back to 64 MiB after a burst of writes, for example a testnet sync (ouroboros-consensus#2332). The second release keeps votes for the 128 most recent points only, so the vote state no longer grows without limit. This bound lets us take load readings, but it does not resist an adversary (ouroboros-consensus#2367).
• Maintenance and support (Treasury Funding Initiative 17: Maintenance and Support):
  • Released ouroboros-consensus 5.0 (#2356, with the dependency updates in #2341 and #2352). The release has the reworked Peras API and drops node-to-client versions below v23. The default ledger snapshot policy now takes a snapshot every 40 * k slots, where k is the security parameter, which is one snapshot a day on mainnet. A background thread now writes the snapshots. The mempool now moves a transaction from an older era to the current era in one step: it encodes the transaction in its own era and decodes it in the current era. Before, it translated the transaction through every era in between.
  • Released ouroboros-consensus 5.1.0.0 (#2368, ported to main in #2369). At startup, the node now traces its ledger snapshot policy. The snapshot interval is in slots, but the write delay and the rate limit are in seconds. So settings that suit mainnet, where a slot is one second, can be wrong on a testnet with shorter slots. If the delay or the rate limit is as long as the interval or longer, the node emits the warning ImplausibleSnapshotPolicy.
  • Moved the tracing instances for consensus types from cardano-node into the new ouroboros-consensus:tracing sublibrary. A change to a traced type and the change to its log output can now go in one commit. Some log output changes: a mempool trace at the detailed level logs a transaction as hex CBOR instead of a Haskell Show rendering, and a rejected transaction always logs the reason (#2244).
  • The database tools now read the node configuration with cardano-config, and db-synthesizer reads its forging keys with cardano-keys. The tools now apply the same rules as cardano-node. Before, db-synthesizer could forge blocks in an era that a node with the same configuration file would not enter. One visible change: every genesis file in the configuration must now come with its hash (#2259, #2335).