93 posts tagged with "ledger"

High level summary​

We have focused the last two weeks on CIP-1694, integration of the last several months of ledger work into consensus and node, and testing infrastrutcture that we will use in the conway ledger era.

Low level summary​

Conway rules​

We made a major update to the conway era so that the implementation is now in sync with the spec with respect to the ratification and enactment logic.

See the notes in pull-3291 for more details.

Constraint based generators​

This week we hit a major milestone in our efforts to build out better property based testing support for the main ledger properties. The new constraint based generators can now generate full ledger states with what is probably very close to the real constraints (if anything, it is under constrained). Next we will work on generating a transaction in the context of a ledger state, which would allow us to actually start using these generators for real tests.

See pull-3219.

Preparing a release, now with proper versioning​

After quite some time, we are ready to release a version of ledger that will work with a new version of consensus, using CHaPs.

See pull-3308.

Technical debt​

• We cleaned up the redeemer serialization code (to prevent future mistakes). See pull-3269.
• We added a note to the Alonzo spec, specifying that the transaction inputs are lexicographically ordered in the Plutus script context. See pull-3306.
• We fixed a problem with the address deserialiazation (we mistakenly fixed a bug in the Babbage era which cannot be fixed until Conway). See pull-3307.
• We fixed a problem with our nix build. See pull-3311.
• We fixed a problem with our NoThunks tests. See pull-3310.
• We improved our nightly tests. See pull-3316.

High level summary​

This past two weeks saw many months worth of ledger changes integrated with the cardano-base and ouroboros-network repositories. The vast majority of the effort involved all the changes to the ledger serialization libraries (and the interplay with cardano-base) which now support proper versioning. Supporting the conway ledger era, and in particular allowing the ledger state to transition to a whole new system of governance, also played a noticeable part in the integration.

Besides the integration work, the team continues to address technical debt, improve the documentation, make our testing infrastructure better (such as experimenting with better nightly tests), and formally specifying more parts of CIP-1694.

Lower level summary​

Integration work​

Below is all the integration work completed, which will enable a release to node from the current ledger master. Many thanks to Alexey for this colossal undertaking!

• pull-3303
• pull-3302
• pull-3300
• pull-3299
• pull-3298
• pull-3297
• pull-3292
• pull-3289
• pull-3288
• pull-3279
• pull-3282
• pull-4349

Technical debt​

Starting in major protocol version 9, zero-valued multi-assets will no longer be permitted in the CBOR specification. See pull-3241.

We now have our .cabal files being checked for a consistent formatting given by cabal format in our CI. See pull-3286.

We are still experimenting with better nightly tests for our long running property based tests. See pull-3276 and pull-3296.

Small documentation improvements​

The hand proofs of the preservation ADA property have been added back to the Shelley ledger specification. See pull-3295.

We have clarified how the script integrity hash is computed in the CDDL description. See pull-3290.

Specifying CIP-1694​

Our new formal specifications backed by Agda have seen a lot of progress! Upgrading to Agda 2.6.3 fixed our main build infrastructure problems, and we no longer have to rely on our custom fork. See pull-50.

Our progress on formally specifying CIP-1694 can be followed here:

• in Agda: pull-40
• the corresponding Haskell: pull-3291

Constraint based testing​

We are still actively working on our new constraint based property testing framework. We have nearly all of the constraints for an entire ledger state hooked into generators, and tests that the generators do indeed obey the constraints. The variable count in the constraints is over 100! There is still much work to do on shrinking, making the generators faster, and writing actual property tests for the ledger, but the approach still seems viable and we are hopeful that it could replace our trace generators. The WIP can be followed here: pull-3219.

High level summary​

Much of the work the past two weeks involved integration efforts, cleaning up and debugging some serialization issues, adding tests, and work on large projects that are still ongoing. We also released a CIP this week that aims to make the ledger a registered CIP category.

Lower level summary​

Ledger evolution CIP​

We published a CIP that will make the Cardano ledger a registered category of the CIP process.

Serialization issues​

We had previously thought that we had found a serialization problem with the redeemers, but it turned out to just be particularly confusing code. We have now clarified the issue for the future.

See pull-3263, pull-3269, and pull-3268.

Starting in protocol version 9, we will no longer accept duplicate keys in CBOR maps.

See pull-3277.

New tests​

We added a new property test suite for some of our custom containers.

See pull-3270.

Progress on a better cost model serialization situation.​

We are still working our way through issue-2902. Towards this end, we are now properly gating the new flexible encoders until version 9.

See pull-3274.

Nightly tests​

We are still experimenting with moving more CI actions to GitHub actions.

See pull-3276.

Constraint based generators​

We continue to add to our proof of concept for constraint based generators. See the previous ledger update for more information about this project.

High level summary​

The ledger team completed some preliminary ground work in preparation for CIP-1694 (restructuring the ledger state), fixed the PDF hosting problem (mostly the formal specs), built out more of the new user-friendly ledger API, finished a proof of concept for constraint-based generators for property tests (with the hopes of being able to replace our trace generators one day), and addressed technical debt.

Lower level summary​

Restructuring the ledger state​

The existing governance structures will be replace in the conway ledger era, as described in CIP-1694. In particular, the ledger rules will be restructured as follows:

   BBODY
    |
    |-------------------------------|
    v                               v
   TICK                            LEDGERS
    |                               |
    |---------|                     |
    v         v                     v
   RUPD     ~NEWEPOCH~            ~LEDGER~
              |                     |
              |----------|          |--------|-------------------|-----------|
              v          v                   v                   v           v
            ~EPOCH~     +ENACTMENT+         DELEGS              UTXOW       +TALLY+
              |                              |                   |
              |---------|------------|       v                   |
              v         v            v      DELPL                v
             SNAP      POOLREAP    -UPEC-    |                  UTXO
                                             |--------|          |
                                             v        v          v
                                            POOL     DELEG     ~UTXOS~

-..- Removed
+..+ Added
~..~ Modified

Moreover, the ledger state will also be restructured in accordance with the new rules. In the conway code, we have now removed UPEC, added TALLY, and stubbed ENACTMENT. We have also adapted all the data structures in the ledger state.

See:

• pull-3216

PDF hosting​

We now build all of our PDFs using a GitHub action which is triggered by pushing a tag with a specific form, cardano-ledger-spec-YYYY-MM-DD. The action creates a GitHub release containing the PDFs. The links in the main ledger README now point to the PDFs in the latest release.

See:

• pull-3245
• pull-3260

Powering the new ledger API​

We have now removed all the HasField instances from the protocol parameter data types, and replaced them with lenses. This is probably the last major restructuring that the ledger team will do on the code base for the API for a while (the Plutus tools team will be working on it next, see here). We also added a new helpful function ensureMinCoinTxOut.

See:

• pull-3242
• pull-3248

Constraint-based generators​

Our largest scale property tests generate an initial ledger state and a long sequence of valid blocks which span several epochs, mimicking a real network. These tests are, in theory, excellent for checking properties. They are, however, very difficult to maintain and are not as random as we would like (a lot of bias has to be introduced to keep the ledger state in enough order to keep generating blocks).

We would like to switch to tests which instead generate a random ledger state representative of not just an initial state, generate a single random valid block, and then test our properties. The hope is that these will be much more random and easier to maintain.

We have finished a proof of concept are encouraged that this approach could work!

See:

• pull-3219

Technical debt​

• pull-3244 massive CI speedup
• pull-3249 better types for fees in the protocol parameters
• pull-3264 move our annotator code to the cardano-ledger-binary package where it belongs
• pull-3239 move the Wdrls type to the Core module.

High level summary​

The ledger team finished up the remaining work for tracking individual depots, built out the new Conway era transaction body (in line with CIP-1694), greatly reduce some problematically large calculations on the epoch boundary, and addressed technical debt.

Lower level summary​

Finishing the deposit tracking​

The initial work on the individual deposit tracking project focused only on correctness. As this is a large data structure (since its size is linear with respect to the number of registered stake credentials), it is very important that we also reduce the memory overhead as much as possible. Fortunately, we were able to add very little overhead for the deposits by using existing efficient data structures. The extra tracking now only incurs one word (8 bytes) per registered stake credential.

See:

• pull-3195
• pull-3202
• pull-3217

New Conway era transaction​

We implemented the Conway era transaction body, which is in line with CIP-1694. Note that the Conway era implements, losing speaking, the parts of CIP-1694 that are not related to the liquid democracy (the "DReps"). The new transaction body adds the new governance actions and votes, while also deprecating the old governance structures (i.e. the old protocol parameter updates and MIR certificates).

We also now have the wire specification (CDDL file) and serialization code in place. The wire specification is still subject to change while we work on the Conway era, but it is now usable and has proper testing support (so that, for example, the serialization round-trips, etc).

See:

• pull-3176
• pull-3226

Optimizing the TICKF transition​

Every since the release of the Shelley era, we have been working to reduce the computational load placed on the node by the ledger at the epoch boundary. While still not perfect, we believe that we have removed one of the final problematically long epoch boundary computations that exacerbate situations like this. In particular, the problem involved the way in which the consensus layer obtains a view of the ledger for the purposes of checking the leadership schedule in a new epoch. We implemented a stopgap measure which now only incurs a single multi-second cost once per epoch instead of potentially several multi-second costs while the networks waits for the first block of a new epoch to be minted.

See:

• An ADR about the decision.
• pull-3209

Technical debt​

We closed the year out with a lot of reduction to the technical debt!

Improved ledger event​

• pull-3212 - The ledger events are not guaranteed to appear in any given order within a block. For this reason, motivated by the use case in db-sync, the TotalDeposits event now includes a transaction ID and emits the change in deposits instead of the value.

Improved type saftey​

• pull-3208 - We replaced NominalDiffTime with a newtype wrapper. The problem was that our CBOR encoders and decoders were using the wrong level of precision, having to due with with the Shelley genesis file. We removed the potential problem with a newtype wrapper.
• pull-3167 - We now use a GADT to ensure consistency of the Plutus language in the types for TransactionScriptFailure and PlutusDebug.

Code/Module organization​

• pull-3175 - The Allegra and Mary eras had an unusual relationship in our codebase, due to the uncertainly of release dates while we were implementing them. In particular, they were coupled in way that is different from the rest of the code base. With hindsight on our side, we split the combined shelley-ma Haskell package into two separate ledger era packages, which is now consistent with the rest of the repository and module structure.
• pull-3184 - We created a core test sub-library, cleaning up a lot of our property test generator code.
• pull-3210 - We moved the KeyPair type to the test library. Outside of testing, the ledger does not need to deal with signing keys, and since this is a topic that deserves the utmost care, it is best to make it clear that our use of signing keys is only for testing.
• pull-3229 - We split the Cardano.Ledger.Alonzo.Data module, which is more consistent with the rest of the codebase.

Revert pointer address deprecation​

Thanks to one of our excellent internal auditors, @jmhrpr, we now have a better plan for deprecating pointer addresses. This meant that we had to revert the previous work to deprecate them.

See:

• issue-3161
• pull-3215

Miscellaneous​

• pull-3205 - We removed deprecated type synonyms.
• pull-3218 - We cleaned up the address deserialization.
• pull-3223 - We fixed faulty address deserialization tests.
• pull-3222 - We switched to a general type family TxOut from concrete ones, reducing many constraints.
• pull-3224 - ShelleyGenesis is now parameterized by crypto instead of by era.
• pull-3170 - We set the cabal-version to 3.0 in our projects.
• pull-3172 - We removed the now useless EncodeMint/DecodeMint classes.
• pull-3225 - We switch from ormolu to fourmolu. The reason was to be able to finally have more diff friendly code!