Skip to main content

· 2 min read
Jean-Philippe Raynaud

High level overview

This week, the Mithril team completed the activation of the ‘Pythagoras’ Mithril era on the mainnet, which has occurred as expected at the transition to Cardano epoch 539. They also created the pre-release for the new distribution 2506.0-pre.

The team continued implementing incremental certification of the Cardano database, working on the client library, client CLI, and WASM client. They also kept implementing monitoring and dashboarding. Additionally, they completed the dependency builder refactoring in the aggregator.

Finally, the team fixed some flakiness in end-to-end tests in the CI and prepared for sunsetting the SanchoNet network.

Low level overview

  • Pre-released the new distribution 2506.0-pre
  • Completed the issue Activate Pythagoras Mithril era #2034
  • Completed the issue Support aggregator response compression in HTTP clients #2286
  • Completed the issue Simplify dependency builder in aggregator #2254
  • Completed the issue Aggregator stress test is failing #2299
  • Worked on the issue Release 2506 distribution #2207
  • Worked on the issue Implement Incremental Cardano DB in client library #2214
  • Worked on the issue Implement Incremental Cardano DB in client CLI #2246
  • Worked on the issue Implement monitoring and dashboarding for Incremental Cardano DB #2249
  • Worked on the issue Flakiness in e2e tests in CI #2222
  • Worked on the issue testing-sanchonet network decommission #2296
  • Worked on the issue Upgrade the deprecated ubuntu-20 builders in CI #2216
  • Worked on the issue End of support for MacOS x64 builds in the CI #2250

· One min read
Damian Nadales

High level summary

  • Opened a Draft PR for a CIP to remove EBBs from the historical chain.
  • Opened a Draft PR for a CPS regarding low participation scenarios.
  • Identified the threads in Cardano node which were unlabeled (#6086). We're working to add the missing labels, which will help with debugging and error tracing efforts.
  • Got the results for the UTxO-HD V2 InMemory cluster run, which show that there is a regression in mempool snapshotting. Investigation is ongoing.
  • Fixed the block replay tracer in the new tracing system (#6091).
  • Adapted UTXO-HD to take advantage of the fact that TxOuts are forward deserializable (#1360).
  • Merge PRs upstream to enable the new QueryStakePoolDefaultVote (#5055, #4860, and #1373).
  • Merged LocalTxMonitor changes into network-protocols, which constitutes a first step towards enabling the LocalTxMonitor to report other dimensions of the transaction size.
  • Completed first pass at a script cataloguing all of the consensus symbols used in downstream packages, as part of our contributions to the Cardano Blueprint efforts.

· 2 min read
Jean-Philippe Raynaud

High level overview

This week, the Mithril team created the transaction to activate the switch to the next ‘Pythagoras’ Mithril era, which will occur at the transition to Cardano epoch 539. They also started communicating the upcoming bump in the minimum version of glibc required to run Mithril pre-built binaries on Linux.

The team continued implementing incremental certification of the Cardano database, working on the client library, client CLI, and WASM client. They also began implementing monitoring and dashboarding. Additionally, they started refactoring the ‘mithril-common’ library and simplifying the dependency builder in the aggregator.

Finally, the team worked on dropping support for pre-built binaries for macOS x64 architecture and attempted to fix some flakiness in end-to-end tests in the CI.

Low level overview

  • Published a dev blog post about the Minimum required glibc version bump
  • Completed the issue Update explorer for Incremental Cardano DB #2212
  • Completed the issue Enhance Snapshotter to avoid file deletion on error if already exists #2215
  • Completed the issue Split mithril-common crate - Preliminary work #2253
  • Completed the issue Add command to create Era keypair in aggregator #2271
  • Completed the issue release-mainnet was unavailable #2264
  • Completed the issue Create a manual workflow to test emails alert delivery #2251
  • Worked on the issue Implement Incremental Cardano DB in client library #2214
  • Worked on the issue Implement Incremental Cardano DB in client CLI #2246
  • Worked on the issue Implement monitoring and dashboarding for Incremental Cardano DB #2249
  • Worked on the issue Simplify dependency builder in aggregator #2254
  • Worked on the issue Flakiness in e2e tests in CI #2222
  • Worked on the issue Upgrade the deprecated ubuntu-20 builders in CI #2216
  • Worked on the issue End of support for MacOS x64 builds in the CI #2250
  • Worked on the issue Activate Pythagoras Mithril era #2034

· One min read
Noon van der Silk

High-level summary

We are very excited to finally release 0.20.0 of Hydra with incremental commits. We continue to work on memory enhancements and other work around stability and resiliance.

What did the team achieve?

What's next?

  • Continue to work on memory usage enhancements #1618
  • Finish Hydra explorer supporting multiple versions #1282
  • Start work on API command to clear pending transactions #1284
  • Start working on new networking stack #1720

· 4 min read
Michael Karg

High level summary

  • Benchmarking: Release benchmarks and performance baselines on 10.2 for UTxO-HD, new GHC, Genesis; 'Perdiodic tracer' benchmarks.
  • Development: Pervasive thread labeling in the Node; fix a race condition in monitoring dependency ekg-wai.
  • Infrastructure: Haskell profile definition work passed testing, ready for merge; continued 'Byron' support in our tooling.
  • Tracing: C library for trace forwarding reached prototype stage; last batch of documentation updates ready for publication.
  • Community: Support and valuable feedback on Discord for new tracing system rollout.

Low level overview

Benchmarking

We've performed a full set of release benchmarks and analyses for Node version 10.2. We could not detect any performance risks, and expect network performance to be equivalent or slightly better than 10.1.x releases, albeit using slightly more CPU resources under rare conditions.

Furthermore, we're building several performance baselines with 10.2 to compare future changes, features or node flavours to. For comparative benchmarks, it's vital every change be measured individually, as to be able to discern their individual performance impact. For Node 10.3, there are several of those we want to capture, such as crypto class simplifications in Ledger, UTxO-HD with a new in-memory backend, Ouroboros Genesis, and last not least a new GHC9.6 release addressing a remaining performance blocker when building Cardano.

Additionally, we've validated the 'Periodic tracer' feature on cluster benchmarks and now have evidence of its positive impact on performance. This feature decorrelates gathering metrics from the ledger from the start of a block producer's forging loop, without sacrificing predictability of performance. By removing this competition on certain synchronization primitives, the hot code path in the forging loop now executes faster. The feature will be integrated in a future version of the Node.

Development

We've tracked down a race condition in a community package that both tracing systems depend on for exposing metrics. In ekg-wai, a ThreadKilled exception could be re-thrown to the thread where it originated from. It is a low-risk condition, as it occurs only when then Node process terminates; however, when terminating due to an error condition, it caused the process to end prematurely, before the error could be logged. We've opened a PR (ekg-wai#12) against the package containing the fix and pre-released on CHaP.

Tracking down this condition could have been improved by providing pervasive, human-readable labels for all the threads that the Node process spawns. So in coordination with the Consensus team, we made sure this is the case for future builds of the Node - including locations in the code where dependency packages internally use forkIO to create green threads. This will enhance usability of debug output when looking into concurrency issues.

Infrastructure

The Haskell definition of benchmarking workloads - and the removal of its bash/jq counterpart - is complete, and has passed testing phase. This includes a final alignment between all profile content defined using either option. Once merged, this will open up the path for simplification of how nix interacts with the performance workbench - and hopefully reduce complexity for our CI runners.

As cardano-api is deprecating some protocol parameter related data types which do not have relevance for Cardano anymore, we've had a discussion with stakeholders about the implications for our tooling: This would effectively disable our ability to benchmark clusters of BFT nodes which do not use a staking / reward-based consensus algorithm - as it used to be in Cardano's Byron era. The decision was made to not drop that ability from our tooling, as there are potential applications for the benchmarks outside of Cardano. As a consequence, we've startied porting those types to live on in our toolchain, representing an additonal maintenance item within our team.

Tracing

The self-contained C library implementing trace forwarding is now in prototype state. It contains a pure C implementation of our forwarding protocol over socket, as well as pure C CBOR codecs for data payload to match the TraceObject schema used within the context Cardano. That ensures existing tooling can process traces emitted by non-Cardano applications, written in languages other than Haskell.

The latest updates to Developer Portal: cardano-tracer are ready to be published and awaiting a PR review on the Cardano Developer Portal.

Community

We've been quite busy on our new Discord channel #tracing-monitoring on the IOG's Technical Community server. There's been an initial spike of interest and we've been able to provide support and explain various design decisions of the new tracing system. Additionally, we've gotten valuable feedback about potential features that would greatly help adoption of the new system. These are typically highly localized in their implementation, and non-breaking wrt. to API and design, such that addressing this feedback promptly adds much value at low risk - Thank You for your input!