88 posts tagged with "ledger"

High level summary​

This update covers a longer stretch than usual because there was no report since 2026-05-27: the whole Ledger team was first at the ZuriHac conference and then had an in-person team meetup, so the last several weeks were lighter on routine reporting but no less productive. A good chunk of the work below is the continuation of storylines from earlier updates, alongside a few new fronts.

On the performance side the state-annotated transaction (StAnnTx) work reached back to the older eras. Integration of the memoized values held in StAnnTx had previously been done for Babbage onward, and this cycle it was extended to Alonzo, which needed special handling, along with a small follow-up removing an unnecessary EncCBOR instance for AlonzoStAnnTx. We also finished the long-running effort to remove the legacy Coders machinery: the final piece for protocol version 12 and up (dropping the Summands usage) landed, Coders was removed from the testlibs and tests, and StakePoolParams was switched from CBORGroup-derived encoding to manual EncCBOR/DecCBOR instances in preparation for dynamic CBOR list lengths.

A notable new front is Leios support beginning to land in the ledger. The LeiosCert type, defined in the new cardano-crypto-leios package on the cardano-base repository, was added as an optional field to DijkstraBlockBody, and the Dijkstra CDDL for header_body was updated to reflect what the Leios prototype should be producing. We also introduced a new cardano-protocol package to house protocol-level code. On the ledger rules themselves there was a round of structural cleanup: a new ENTITIES rule (with an equivalent subrule) was introduced between LEDGER and CERTS to handle withdrawals and direct deposits, the era prefix was dropped from rule names, and the Rewarding script purpose was renamed to the more accurate Withdrawing.

The Dijkstra era work continued on several smaller fronts, including requiring top-level guards in the top-level transaction body, adjusting the Conway transaction_witness_set CDDL rule to use non-empty lists for the relevant fields. Conformance testing for Dijkstra advanced to its third installment, this time adding SpecTranslate instances for Dijkstra types and ExecSpecRule instances for the LEDGER and NEWEPOCH rules. Two user-visible bugs were fixed: injected extraConfig cost models were being ignored for PlutusV1/V3 because of an argument-ordering issue in overrideCostModels, and TxInfo translation could fail spuriously in LegacyMode. We also added ToJSON/FromJSON instances for EraScript and native scripts.

On the tooling and infrastructure side the generate-cbor CLI tool added last cycle was moved into cardano-ledger-api, refactor of the singleton-key namespaces (committee and constitution) for Canonical Ledger State, new Conway UTxO and Script generators were added, and CI was updated to allow automated agents to invoke the /update-changelogs skill. The dependency stack was moved forward with updates to cardano-crypto-class-2.5, the haskellNix flake input, and the usual GitHub Actions bumps.

Low level summary​

Features​

• PR-5837 - Remove unnecessary re-exports from Shelley Rules
• PR-5859 - Remove era prefix from rule names
• PR-5748 - Refactor namespaces with singleton keys
• PR-5867 - Prevent TxInfo translation failure for LegacyMode
• PR-5866 - Rename Rewarding to Withdrawing
• PR-5845 - Add ToJSON/FromJSON instances for EraScript
• PR-5860 - Stop using Summands from Coders for PV12+
• PR-5857 - Further StAnnTx integration
• PR-5872 - Add LeiosCert to DijkstraBlockBody
• PR-5897 - Remove unnecessary EncCBOR instance for AlonzoStAnnTx
• PR-5899 - Fix injected extraConfig cost models being ignored for PlutusV1/V3
• PR-5878 - Entities and Subentities Rules
• PR-5889 - Fix Dijkstra CDDL for Leios changes to header_body
• PR-5887 - Create cardano-protocol package
• PR-5903 - Require top-level guards in top-level transaction body
• PR-5908 - Use non-empty list for fields without duplicate enforcement in Conway transaction_witness_set cddl rule
• PR-5890 - Replace CBORGroup-derived EncCBOR/DecCBOR for StakePoolParams with manual instances

Testing​

• PR-5844 - Prepare conformance for Dijkstra 3
• PR-5868 - Add Test.Cardano.Ledger.Conway.Gen.{UTxO,Script} modules.
• PR-5883 - Add isPotentialFutureMember ImpTest

Infrastructure and releasing​

• PR-5864 - Move generate-cbor to cardano-ledger-api
• PR-5870 - Update to cardano-crypto-class-2.5
• PR-5876 - Bump actions/checkout from 6 to 7 in the actions group
• PR-5877 - Allow agents to invoke /update-changelogs
• PR-5865 - Remove Coders from testlibs and tests
• PR-5884 - Bump actions/cache from 5 to 6 in the actions group
• PR-5894 - Update haskellNix flake input

High level summary​

The most visible result of the past two weeks is that the Plutus context performance arc has reached its first real payoff. Two cycles ago we memoized the Plutus related parts of transaction validation, last cycle we threaded the state-annotated transaction StAnnTx through the LEDGER, UTXOW, UTXO and MEMPOOL rules across every era, and this cycle the rules actually start consuming the precomputed information stored in StAnnTx instead of recomputing it. In parallel we kicked off a sizeable effort to remove the legacy Coders machinery from the codebase: several call sites have been switched off Coders, a new SparseKeyed alternative has been introduced, and decodeSparseKeyed was extended to cover additional types. This is part of a broader cleanup that is expected to continue for several more cycles.

On the Dijkstra era CDDL front the work begun with the new block_body definition continued: we introduced a dedicated transaction_mempool rule so that the transaction decoder used in the mempool can be distinguished from the one used inside the block body, tightened the chain_code specification to 32 bytes, and landed a thorough pass of fixes to the Dijkstra CddlSpec so that all of the CDDL tests pass. We also pushed forward the second installment of conformance testing preparation for Dijkstra, this time adding an era type parameter to SpecTranslate and SpecTransM, instantiating it for the Conway modules, and updating the formal-ledger-specifications dependency accordingly.

The namespace work for the canonical ledger state initiative also continued, with a new entities/accounts/v0 namespace and a refactor of gov/proposals/v0 that preserves the proposals order, which is required to reconstruct ledger state faithfully.

On the benchmarking and tooling side we added two new criterion microbenchmarks for the TICK and NEWEPOCH rules, which we expect to be a tremendously useful tool for analysing epoch transition performance. Vector-map benchmarks were also fixed and we switched some hot paths to unsafe vector indexing where bounds checking is redundant. A new generate-cbor CLI tool was added, which makes it possible to generate CBOR for any rule in any era from the command line and should be useful both for our own debugging and for downstream consumers, such as serialization libraries and alternative node implementotions. The test reorganization started last cycle continued with the CddlSpec hierarchy being inverted to give cleaner test output and the predicate failure CBOR roundtrip tests being folded into the per-era ImpSpec set, which also brought in missing tests for the BBODY, MEMPOOL and Dijkstra-specific rules. Finally on the infrastructure side, the doctesting setup that was temporarily disabled in the last cycle has been properly refactored to use cabal repl --with-repl=doctest instead of cabal-doctest, and the index-state was bumped to pull in cardano-base 0.1.4.

Low level summary​

Features​

• PR-5807 - Stop using Coders in some places
• PR-5815 - Switch to unsafe vector indexing
• PR-5804 - Use precomputed data from state-annotated transactions
• PR-5834 - Coder's SparseKeyed alternative
• PR-5843 - Add transaction_mempool
• PR-5846 - Update Dijkstra chain_code
• PR-5738 - Refactor gov/proposals/v0 namespace to include the proposals order
• PR-5699 - Add entities/accounts/v0 namespace
• PR-5847 - decodeSparseKeyed for more types

Testing​

• PR-5810 - Add examplePParams[Update] to EraTest for goldens.
• PR-5814 - Add TICK and NEWEPOCH benchmarks
• PR-5841 - Invert CddlSpec hierarchy
• PR-5830 - Prepare conformance for Dijkstra 2
• PR-5851 - Organize predicate failure CBOR roundtrip tests
• PR-5779 - Fix various issues in Dijkstra CddlSpec to make all the tests pass

Infrastructure and releasing​

• PR-5798 - Refactor doctesting
• PR-5849 - Bump idna from 3.10 to 3.15 in /doc
• PR-5762 - Add generate-cbor CLI tool
• PR-5861 - Update index-state
• PR-5852 - Minor updates

High level summary​

• CIP-118 - Nested Transactions:

  Over the past two weeks the Ledger team has continued to push the Dijkstra era forward on multiple fronts. The CDDL specification of block was updated to match CIP-0176 - Non-segregated Block Body Serialization, a dedicated block_body was added and an EncCBORGroup instance for DijkstraBlockBody was added to unblock downstream integration in Consensus. Serialization for DijkstraTx was fixed up as well. We also followed up on the Plutus context performance work introduced in the previous update by integrating StAnnTx (the state annotated transaction representation that memoizes the Plutus parts of validation) into the LEDGER, UTXOW, UTXO and MEMPOOL rules across every era. With this integration the performance benefits of the earlier memoization work are now realized end to end. Further work in this area will let us rip the benefits of this optimization upon block construction as well, not just during initial validation. On top of that, several guards were added to the Plutus V1-V3 context translation so that any of the new Dijkstra features that those older language versions cannot represent (subtransactions, direct deposits, balance intervals, etc.) cause translation to fail explicitly rather than silently producing a wrong context. We also did preparation work for running conformance tests in Dijkstra and started reorganizing the test hierarchy around a new class-based ImpSpec approach that enforces a common set of tests across all eras.

• LedgerHD

  The streaming infrastructure for injecting large amounts of data from genesis files, which was started in the previous cycle, has been completed. It now covers stake pools and the remaining state needed for the testing and benchmarking work that targets reading data in constant memory from disk and paves the way for further direct injection into on-disk ledger state.

• CIP-165 - Canonical Ledger State:

  The expansion of the Canonical Ledger State continued with two more namespaces being added under the new namespace-based query interface, entities/dreps/v0 and entities/stake_pools/vrf_key_hashes/v0.

• Maintenance

  We've expanded examples and added golden tests for all of the migrated queries from Consensus to Ledger done last cycle. Furthermore we did the same for transaction examples. All of this effort not only will let us remove golden tests unrelated to the Consensus component from the ouroboros-consensus repository, but also improve their quality.

  In addition to features we published the changelog for the recent cardano-node-11.0 release, made ZeroTreasuryWithdrawals a permanent governance check for every era starting with Conway, instead of not having it in the bootstrap phase. We switched ChainData and Attributes to use ByteArray instead of ByteString to avoid pinned-memory fragmentation. A few NoThunks instances were tightened up where consensus needs them for storing on disk and relaxed where the values are already forced to normal form. Finally, we fixed a cuddle failure on Plutus Constr data and a bug in its definition in the CDDL spec discovered by our new generator-based tooling.

Low level summary​

Features​

• PR-5787 - Remove redundant import
• PR-5785 - Relax header protocol version for testnets
• PR-5786 - Changelog for cardano-node-11.0
• PR-5733 - Dijkstra era CDDL for block, add block_body
• PR-5789 - Integrate StAnnTx in rules across all eras
• PR-5777 - Full support of data injection from genesis files using streaming
• PR-5793 - Change DijkstraTx decoder to support variable length encoding
• PR-5669 - Fail PlutusV1-V3 translation for non-empty subTransactionsTxBody
• PR-5797 - Make ZeroTreasuryWithdrawals a permanent check in gov state transation rule
• PR-5700 - Add entities/dreps/v0 namespace
• PR-5808 - Relax NoThunks instance for BlockTransitionError
• PR-5781 - Fail Plutus V1-V3 translation with guards
• PR-5813 - Implement EncCBORGroup instance for DijkstraBlockBody
• PR-5812 - Switch ChainData and Attributes to use ByteArray
• PR-5811 - Fix TranslateErainstance for DijkstraEra CertState
• PR-5743 - Add entities/stake_pools/vrf_key_hashes/v0 namespace
• PR-5823 - Add NoThunks instances for ChainTransitionError and ChainPredicateFailure
• PR-5816 - Import Rules as qualified
• PR-5799 - Add example transaction golden test for each era

Testing​

• PR-5778 - Add golden tests for queryConstitution
• PR-5761 - Expand upon transaction examples by adding missing fields.
• PR-5794 - Add golden tests for the rest of the query result types
• PR-5801 - Fix failing cuddle test for Plutus data Constr.
• PR-5775 - Preparation for conformance testing in Dijkstra
• PR-5803 - Start to re-organize test hierarchy: ImpSpec
• PR-5826 - Fix trace construction in TestChain

Infrastructure and releasing​

• PR-5790 - Bump slackapi/slack-github-action from 3.0.2 to 3.0.3 in the actions group
• PR-5796 - Disable doctest to unblock master
• PR-5806 - Bump cuddle to 1.7.0.0
• PR-5817 - Bump urllib3 from 2.6.3 to 2.7.0 in /doc
• PR-5822 - Remove weird hyperlinked space between badges
• PR-5758 - Refactor testing in GitHub CI

High level summary​

Ledger team has made significant progress on Ledger rules for Nested Transactions as well as translation of newer features to Plutus context. We've also introduced a major change to how Plutus context is prepared for each transaction entering a mempool, which when fully integrated will result in significant performance improvements. On the efforts of expanding Ledger we have transferred all of the ledger state queries from Consensus component to the Ledger component, as well as introduced an initial stage of streaming injection functionality that we'll be needed for testing and benchmarking, especially for on disk storage where huge amount of data needs to be injected into ledger state without any impact on the operating memory.

Aside from features we have fixed a few more cddl bugs that we discovered with our new AntiGen and cuddle tools. Improved examples for golden tests and golden tests facilities.

Low level summary​

Features​

• PR-5692 - Adjust Dijkstra UTXOW rule to subtransactions
• PR-5684 - Remove unused FromCBOR/ToCBOR instances
• PR-5572 - Add streaming interface to era transition for initial funds injection
• PR-5737 - Fix treasury donations handling for SubTransactions
• PR-5755 - Clean up CddlSpec, fix Datum and AccountBalanceIntervals generators
• PR-5736 - Fix DecCBOR instances to reject ProtVer values exceeding the era maximum
• PR-5636 - Memoize Plutus related parts of transaction validation
• PR-5764 - Bring all remaining queries from consensus over to ledger
• PR-5767 - [Leios Prototype] Backwards compat fix
• PR-5757 - Remove allowLeftOver flag from binaryGetDecoder and simplify IP address decoders
• PR-5754 - Fail PlutusV1-V3 translation for direct deposits and balance intervals
• PR-5770 - Simplify Dijkstra constraints
• PR-5751 - UTXO with subtransactions
• PR-5747 - Add gov/proposals/roots/v0 namespace
• PR-5742 - Add entities/stake_pools/v0 namespace

Testing​

• PR-5749 - Fix and re-enable BbodysSec
• PR-5687 - Convert pool related CHAINExamples to ImpTests
• PR-5732 - Expand transaction content in examples in Test.Cardano.Ledger.[era].Examples
• PR-5753 - Use hspec-golden for Alonzo golden tests
• PR-5760 - Golden testing facilities for JSON and CBOR
• PR-5731 - Add negative tests for Metadatum int decoding range
• PR-5765 - Add a custom generator to cost_models, create Huddle.Gen module

Infrastructure and releasing​

• PR-5750 - Run tests for ghc 9.6.7 only
• PR-5756 - Bump actions/upload-pages-artifact from 4 to 5 in the actions group
• PR-5766 - Drop x86_64-darwin from Hydra builds
• PR-5768 - Bump slackapi/slack-github-action from 3.0.1 to 3.0.2 in the actions group
• PR-5594 - Test failure summaries in CI
• PR-5774 - Handle single failing suite case in CI test failure summaries

High level summary​

These were our main areas of focus last period on the Ledger project:

Maintenance​

We worked really hard in order to prepare Ledger for the cardano-node-10.7.0 release that had all of the bug fixes and features needed for the upcoming van Rossem Hard Fork into protocol version 11. Furthermore, we prepared Ledger for a subsequent cardano-node-10.7.1 release that had some of the performance issues resolved, which were identified in the Ledger component of 10.7.0 release.

We've also identified and fixed a few bugs, most of which were in CDDL specification. This was largely thanks to the recently integrated tool AntiGen, that we developed specifically for creating negative property tests for Ledger serialization.

Besides bug fixes, we've added quite a few nice optimizations, which do result in noticeable speed up in chain validation and replay.

Some very nice improvements were implemented for the Ledger API used by Consensus, which will result in clearer and safer boundaries between those two components.

CIP-118 - Nested Transactions:​

Two more major Ledger rules UTXO and UTXOW were defined for processing transactions in the Dijkstra era that now take sub-transactions into consideration.

CIP-165 - Canonical Ledger State:​

Fixes were applied to canonical ledger state representation for the Constitutional Committee

Low level summary​

Features​

• PR-5654 - Normalize StakeRefPtr addresses better and improve UTxO decoding
• PR-5647 - Change B constructor of Metadatum from ByteString to ByteArray
• PR-5655 - Add ToJSON/FromJSON instances for NonEmptyMap and NonEmptySet
• PR-5630 - Remove distinct_bytes hack, make CDDL reflect reality in previous eras
• PR-5560 - *EraBlockHeader typeclasses deprecating BHeaderView
• PR-5658 - Optimize Text byte length check in metadata and other places
• PR-5632 - Forecast API
• PR-5671 - Remove NoThunks instances for predicate failures and ContextError types
• PR-5635 - ScriptPurpose aware memoization of TxInfo
• PR-5676 - Remove state update from UTXOS
• PR-5641 - Avoid unnecessary computation of PlutusWithContext
• PR-5672 - Make ContextError constructors from Alonzo to Dijkstra era lazy
• PR-5614 - SUBUTXO implementation
• PR-5668 - SUBUTXOW fixes
• PR-5704 - Backport: revert ConwayAccountState type unrolling
• PR-5697 - Remove 5 redundant NativeScript era ~ Timelock era constraints (#5319)
• PR-5665 - Fix metadatum decoder and chain_code CDDL
• PR-5681 - Add/refactor committee-related canonical namespaces
• PR-5725 - Revert decoders to a more efficient implementation
• PR-5716 - Move applyTick into its own ApplyTick typeclass
• PR-5719 - Add protocol version validation to createInitialState
• PR-5729 - Fix ConwayAccountState overhead
• PR-5727 - Add storable instances

Testing​

• PR-5656 - Translate all transaction building examples in our test files to use Lens interface.
• PR-5609 - Adequate conformance to Version restriction
• PR-5667 - Fix Proposal generation to generate HardForkInitiation with bounded major version
• PR-5695 - Update formal-ledger-specifications and re-enable tests
• PR-5529 - Add simpleRewards reproducer from cardano-db-sync and migrate TwoPools example to an ImpTest
• PR-5715 - Bump cuddle to 1.5.0.0

Infrastructure and releasing​

• PR-5653 - Add cabal-gild and shellcheck as part of the pre-commit hook in our Nix devShell.
• PR-5648 - Changelog for cardano-node-10.7
• PR-5657 - Migrate scripts to cardano-ledger-release-tool
• PR-5599 - Enable ghc 9.14 to be used for tests in CI and locally
• PR-5677 - Bump requests from 2.32.4 to 2.33.0 in /doc
• PR-5675 - Add nixfmt in our Github CI checks
• PR-5670 - Add CHANGELOG and cabal file updater skill
• PR-5691 - Bump pygments from 2.19.1 to 2.20.0 in /doc
• PR-5702 - Update contribution policy to note how contributions should be vetted
• PR-5703 - Add a CI workflow for checking nix hashes
• PR-5708 - Bump the actions group with 3 updates
• PR-5709 - Update slack-github-action configuration for v2 onward
• PR-5712 - Fix missed version bump for allegra package
• PR-5713 - Stop running bump-changelogs on release branches
• PR-5666 - Refactor transaction and transaction body of examples in Test.CardanoLedger.<era>.Examples.hs.
• PR-5717 - claude: improve changelog update skill
• PR-5710 - Update microlens dependency
• PR-5726 - Update ghc-9.12 on CI and Nix to ghc-9.12.4, and cabal-3.16 to cabal-3.16.1.0
• PR-5728 - Backport of fixes for cardano-node-10.7.1
• PR-5741 - Fix nix build failure in plutus-preprocessor
• PR-5721 - Leios: Make the (leios enhanced) block decoding backwards compatible