Skip to main content

· 3 min read
Damian Nadales

High level summary​

The consensus team is resuming its activities after the Christmas break. During these weeks we focused on cleaning and benchmarking the UTxO-HD prototype, and discussing with the Ledger team the changes that might be required for the next iterations. The pull request that adds the Conway era is waiting for a second review round and we hope to merge it soon. On the technical debt side we are looking into a property-test failure found in the iterators. We are investigating if this is an error in the model or in the implementation. We also improved the documentation of our testing code.

Workstreams​

UTxO HD Prototype​

We worked with the Ledger team to start preparing the next versions of UTxO-HD. The Ledger team is concerned that for the remaining maps we might need the full ledger state on epoch boundaries. Since the main consumer of the ledger rules is Consensus, the code that requires access to a full state could be moved from the ledger to some Ledger-Consensus bridge. Eg. the traversal of rewards could take place in such bridge, instead of querying the ledger for the values that are required in the epoch-transition computations.

We relocated some UTxO-HD definitions, in preparation for merging the prototype into master.

We also completed updated local benchmarks comparing the replay time and memory consumption of:

  • the baseline node (f2fc76ef45647275c98634da1718290b976ff364)
  • the UTxO-HD node with the in-memory backend
  • the UTxO-HD node with the LMDB backend

The following plot shows the results: we can see that the LMDB node barely reaches 8GB of memory, but it takes 1.78 times longer to replay the chain. The in-memory backend is about 30 minutes faster, but still slower than the baseline version. We are aware of this phenomenon and it is inherent to the problem of maintaining sequences of differences of the last k ledger states that allows us to perform rollback and roll-forward. We are in the process of measuring syncing from scratch times.

We also added StaticEither accessors that helped us to simplify the UTxO-HD prototype.

New Conway era​

We incorporated the feedback of the pull request, and rebased this branch on top of master. The PR is pending a second review round and we hope to merge this soon.

Technical debt​

We are investigating a property-testing failure involving iterators. Solving this requires understanding the expected behavior of iterators in the counterexample found by QuickCheck to determine if the error is in the model or in the implementation.

Fostering collaboration​

We moved the contents of docs/Testing.md closer to the code, so that the explanations about the tests are easier to find in the relevant modules, and the documentation is easier to keep up to date.

· 2 min read
Serge Kosyrev

High level summary​

Since our last update, we focused on infrastructure work: benchmark enablement, tracing system, benchmark environment merge and open source support:

  1. SECP benchmarking enablement is underway: enabling SECP runs in our cardano-ops benchmarking environment is still in progress.
  2. The new tracing system: the improved API of the new tracing system was implemented, and we're now porting the tracing integration layer over.
  3. Infrastructure: the mainnet protocol parameter history is now encoded in the workbench profile machinery at epoch-level granularity, which gives us a systematic approach towards description of past and future benchmarks.
  4. New benchmark deployment infrastructure: we've made some progress on Nomad deployment backend, shared by both of the data publishing and benchmarking needs.
  5. Legacy benchmarking: we've started merging the legacy benchmark deployment infrastructure into the workbench.
  6. Open sourcing: the benchmarking data publishing tool was adapted to the Nomad execution environment provided by SRE, pending final deployment.

Performance​

The AWS cluster infrastructure necessary for SECP benchmarking is still being worked on.

Tracing​

The improved tracing internals were implemented, and we're now into the phase of updating the tracing integration, which is also mostly done.

Infrastructure​

Thanks to collaboration with the DevX team, we have identified and pursued a design that would enable our Nomad workbench backend to execute deployments of both the benchmarking cluster and our data publishing components.

On the benchmark parametrisation front, we have eliminated a long-standing weakness in the way we were specifying the protocol parameters. We now have a very clear and granular method to keep track of protocol parameter evolution -- e.g. the mainnet history changes are now tracked at epoch granularity, while also allowing for systematically described change overlays. This makes the benchmark profile definition much more clear and robust against mistakes.

We also started a merge of the legacy benchmarking environment (based on cardano-ops) into the workbench. The separation between environments was too costly, causing us to reimplement any benchmarking change twice -- first, during development, in the workbench, then in cardano-ops. In addition, maintenance of compatibility code was incurring additional costs, slowing benchmark data analysis development. Once this merge is complete, this will allow us to sharply cut the benchmark development cycle and overheads.

· 2 min read
Iñigo Querejeta Azurmendi

High level summary​

The four open fronts that the crypto team is working on are:

  • MuSig2: The API features (error handling) has been merged.
  • Mithril: We update the KES library and published it in crates.io. Similarly, we merged batch verification of STM signatures.
  • cardano-base: VRF and BLS are still waiting to be merged, as we have some problems with the CI.
  • KES agent: We keep progressing with KES secure forgetting implementation.

We also started working in the cryptography engineering handbook, which will be a cross team effort.

Low level summary​

MuSig2​

  • Merged the more granular error handling mechanism PR#33

Mithril​

  • Adapted KES library so that sk contains its corresponding period, making key handling easier. Avoid making unnecessary copies of the KES key. Expose function to get associated period. All included in Version 0.1.1, which was published in crates.io.
  • Updated new KES format to mithril library PR 674
  • We implemeneted batch verification of stm signatures, which improves the benchmarks for mithril chain-sync PR 531
  • We keep progressing on reducing the use of transmute in mithril-stm.

cardano-base​

  • Nothing new to report. Still working in merging these PRs. We are only missing nix merge of PR#520 and the updates on VRF will be merged.

KES agent​

  • We keep progressing in the secure forgetting PR. Added NoThunks tests, error hasndling to detect when mlocking fails, resolved bugs due to incorrect usage of mlocked memory and wrote more efficient tests PR#255.

Cryptography handbook​

We have made available the (temporary) cryptography handbook https://input-output-hk.github.io/cryptography_spec/, and included the description of some primitives (Ed25519, KES and VRF). This is an ongoing effort, and we plan on further expanding the list of available primitives.

· 5 min read
Jared Corduan

High level summary​

The ledger team finished up the remaining work for tracking individual depots, built out the new Conway era transaction body (in line with CIP-1694), greatly reduce some problematically large calculations on the epoch boundary, and addressed technical debt.

Lower level summary​

Finishing the deposit tracking​

The initial work on the individual deposit tracking project focused only on correctness. As this is a large data structure (since its size is linear with respect to the number of registered stake credentials), it is very important that we also reduce the memory overhead as much as possible. Fortunately, we were able to add very little overhead for the deposits by using existing efficient data structures. The extra tracking now only incurs one word (8 bytes) per registered stake credential.

See:

New Conway era transaction​

We implemented the Conway era transaction body, which is in line with CIP-1694. Note that the Conway era implements, losing speaking, the parts of CIP-1694 that are not related to the liquid democracy (the "DReps"). The new transaction body adds the new governance actions and votes, while also deprecating the old governance structures (i.e. the old protocol parameter updates and MIR certificates).

We also now have the wire specification (CDDL file) and serialization code in place. The wire specification is still subject to change while we work on the Conway era, but it is now usable and has proper testing support (so that, for example, the serialization round-trips, etc).

See:

Optimizing the TICKF transition​

Every since the release of the Shelley era, we have been working to reduce the computational load placed on the node by the ledger at the epoch boundary. While still not perfect, we believe that we have removed one of the final problematically long epoch boundary computations that exacerbate situations like this. In particular, the problem involved the way in which the consensus layer obtains a view of the ledger for the purposes of checking the leadership schedule in a new epoch. We implemented a stopgap measure which now only incurs a single multi-second cost once per epoch instead of potentially several multi-second costs while the networks waits for the first block of a new epoch to be minted.

See:

Technical debt​

We closed the year out with a lot of reduction to the technical debt!

Improved ledger event​

  • pull-3212 - The ledger events are not guaranteed to appear in any given order within a block. For this reason, motivated by the use case in db-sync, the TotalDeposits event now includes a transaction ID and emits the change in deposits instead of the value.

Improved type saftey​

  • pull-3208 - We replaced NominalDiffTime with a newtype wrapper. The problem was that our CBOR encoders and decoders were using the wrong level of precision, having to due with with the Shelley genesis file. We removed the potential problem with a newtype wrapper.
  • pull-3167 - We now use a GADT to ensure consistency of the Plutus language in the types for TransactionScriptFailure and PlutusDebug.

Code/Module organization​

  • pull-3175 - The Allegra and Mary eras had an unusual relationship in our codebase, due to the uncertainly of release dates while we were implementing them. In particular, they were coupled in way that is different from the rest of the code base. With hindsight on our side, we split the combined shelley-ma Haskell package into two separate ledger era packages, which is now consistent with the rest of the repository and module structure.
  • pull-3184 - We created a core test sub-library, cleaning up a lot of our property test generator code.
  • pull-3210 - We moved the KeyPair type to the test library. Outside of testing, the ledger does not need to deal with signing keys, and since this is a topic that deserves the utmost care, it is best to make it clear that our use of signing keys is only for testing.
  • pull-3229 - We split the Cardano.Ledger.Alonzo.Data module, which is more consistent with the rest of the codebase.

Revert pointer address deprecation​

Thanks to one of our excellent internal auditors, @jmhrpr, we now have a better plan for deprecating pointer addresses. This meant that we had to revert the previous work to deprecate them.

See:

Miscellaneous​

  • pull-3205 - We removed deprecated type synonyms.
  • pull-3218 - We cleaned up the address deserialization.
  • pull-3223 - We fixed faulty address deserialization tests.
  • pull-3222 - We switched to a general type family TxOut from concrete ones, reducing many constraints.
  • pull-3224 - ShelleyGenesis is now parameterized by crypto instead of by era.
  • pull-3170 - We set the cabal-version to 3.0 in our projects.
  • pull-3172 - We removed the now useless EncodeMint/DecodeMint classes.
  • pull-3225 - We switch from ormolu to fourmolu. The reason was to be able to finally have more diff friendly code!

· 2 min read
Jordan Millar

2022-12-28 - 2023-01-10

High level summary​

Completed​

docs​

CI & project maintenance​

Developer experience​

cardano-cli​

cardano-api​

cardano-node​

cardano-testnet​

In Progress​

CI & project mainteance​

cardano-cli​

cardano-api​

cardano-node​

cardano-testnet​