Performance & Tracing Update

High level summary

• Benchmarking: Further Governance action / voting benchmarks on Node 10.0.
• Development: New protoype for database-backed persistence layer in our analysis tool locli.
• Workbench: More fine-grained genesis caching; export cluster topology for Leios simulation.
• Tracing: Final round of metrics alignment complete; prepared for typed-protocols-0.3 bump; new tracing system rollout starting with Node 10.2.

Low level overview

Benchmarking

We've been working on improving the voting workload for benchmarks along two axes: Firstly, reduce the (slight) overhead that decentralized vote submission induces. Secondly, introduce a scaling parameter - namely the number of votes submitted per transaction, and hence the number of proposals to be considered simultaneously for tallying and ratification. On the way, we improved upon timing of submissions, as this has caused benchmarks to abort mid-run every now and then: in those cases, a newly created UTxO entry just hadn't settled across the cluster when it was supposed to be reused for consumption.

Scaling of the voting workload is currently under analysis.

Development

Our analysis and reporting tool, locli ("LogObject CLI") has a few drawbacks as far as system resource usage goes; it requires a huge amount of RAM, and initialization (i.e. loading and parsing trace output) is quite slow. Moreover, there is no intermediate, potentially exposable or queryable, representation of data besides the trace messages themselves.

We're working on a prototype that introduces a database persistence layer as that intermediate representation. Not only does that open up raw benchmarking data to other means of querying or processing outside locli. Initializing the tool from the database has also shown to require much less RAM, and to improve duration of the initialization phase. Furthermore, on-disk representation is much more efficient that way - which is no small benefit when raw benchmarking data for a single run can occupy north of 64GiB.

The prototype has yet to be fully integrated into the analysis pipeline for validation, however, initial observations are promising.

Workbench

For our benchmarks, we rely on staked geneses, as the cluster needs control all stake, and such, block production, to yield meaningful performance metrics. As creating a staked genesis of that extent is an expensive operation, we use a caching mechanism for those. Small changes in the benchmarking profile, such as protocol version or parameters, Plutus cost models or execution budgets would usually trigger the creation of a new cache entry. We've now factored out from cache entry resolution all those variables that do not impact staking itself. We then created a mechnanism to patch those changes into genesis files after cache retrieval, when preparing them for a benchmarking run. This adds flexibility for creating profiles, and reduces the time to deploy a run to the cluster.

We also delivered a comprehensive description of our cluster to the Leios innovation team. This includes the definition of our artificially constrained topology, as well as a latency matrix for node connections in that topology, assigning a weight to all edges in the graph. The Leios team intends to use that material to implement a large-scale simulation of the algorithm, and thus gain representative timings for diffusion and propagation.

Tracing

The alignment of metrics names between legacy and new tracing system is now complete - which should minimize the migration effort of existing dashboards for the community. The only differences that remain are motivated by increasing compliance with existing standards like e.g. OpenMetrics. Furthermore, a few metrics still missing in the new system have now been ported over, such as node.start.time or served.block.latest.

We're all set for the expected bump to typed-protocols-0.3: both forwarder packages trace-forward and ekg-forward for the new tracing system have been adapted to the new API and are passing all tests.

Last not least, we've settled on a rollout plan for the new tracing system. The new system will set to be the default with the upcoming Node release 10.2. This is achieved by a change of configuration only - there is no need for different Node builds. The cardano-node binary will contain both tracing systems for a considerable grace period: 3 - 6 months after release. This should give the community ample time to adjust for necessary changes in downstream services or dashboards that consume trace or metrics output.

We'll provide a comprehensive hands-on migration guide summarizing those changes for the user.