6 posts tagged with "system-test"

High-level summary​

• Release testing: Node 10.5.2, 10.5.3, 10.6.0, and 10.6.1.
• Development: Ongoing testing framework improvements.
• Maintenance: Analysis and fixes for nightly job failures.

Low-level overview​

Release testing​

We tested the 10.5 point releases. These contained only bug fixes, so previous test coverage largely applied. We focused on regression testing on local testnets and found no issues.

We also tested the 10.6.0 and 10.6.1 releases and produced a test report. During testing we found and reported two issues:

• BUG – cardano-cli queries hang indefinitely with the V1LMDB backend: issue #1787
• BUG – ObsoleteNode and no blocks forged on PV11: issue #6382

Our Node build test also uncovered an issue in the build documentation, and we submitted a fix PR.

Development​

Over the past two weeks, we enabled testing for Node 10.6. We updated our testnet setup for the new Dijkstra genesis file and disabled legacy and mixed topologies, as legacy topology is no longer supported in 10.6.

We also improved our test framework by adding timeouts for key operations:

• Obtaining a testnet cluster instance
• Calling cardano-cli
• Entire test run
• Network requests

Issue #1787 caused tests to hang indefinitely on cardano-cli commands, so adding these timeouts was essential.

Maintenance​

We continued analyzing and fixing nightly job failures. We are currently maintaining a 100% success rate with no flaky tests.

High level summary​

Since the last update (2023-03-26) we did more improvements on our Test Framework, added more automated tests, kept analyzing and fixing nightly jobs failures, tested the 1.35.7 and 8.0.0 releases.

Workstreams​

Full list of closed PRs

Framework improvements:​

• Make cardano-submit-api REST API service available and run corresponding regression tests by default
• Update cabal build testing for 8.0.0
• Add support for testing governance SPO poll
• Use Poetry for Python dependency management
• Check status of known GH issues during tests runtime and finish the test accordingly
• Generate topology files with both IP addresses and DNS names
• Log issues like failure to start a cluster instance and report errors during tests runtime

Testing improvements​

• Add rollback testing:
  1. global consensus is reached after rollback, in situation where less than securityParam blocks were produced since cluster split
  2. global consensus is not reached when more than securityParam blocks were produced, and the result is permanent fork
• Add tests to governance SPO poll commands

Node:​

• tested the 1.35.7 release
• tested the 8.0.0 release

DB-Sync:​

• improvements in db-sync sync tests
• improvements in db-sync functional tests

High level summary​

Since the last update (2023-1-13) we did more improvements on our Test Framework, added more automated tests, kept analyzing and fixing nightly jobs failures, tested the 1.35.5 and 1.35.6 releases (and their predecessing release candidates) and ran tests on branch with the UTxO-HD functionality.

Workstreams​

Framework improvements:​

• improvements in searching for unexpected error messages in log files during test run
• added support for UTxO-HD testing
• added support for the new P2P topology format
• added local cluster variant that has the same properties as mainnet (epoch length, fees, etc.)
• added preliminary support for the Conway era
• added support for running tests on the Preview testnet

Full list of closed PRs

Node:​

• ran tests on node branch with the UTxO-HD functionality
• tested the 1.35.5 release
• tested the 1.35.6 release

DB-Sync:​

• improvements in db-sync sync tests
• improvements in db-sync functional tests

High level summary​

During the last 2 weeks we did more improvements on our Test Framework, ran some sanity tests related to the P2P Single Relay functionality.

We also update the Node & DB-Sync sync tets to build with Nix as the prebuilt files are no longer available at PR level.

Workstreams​

Framework improvements:​

• extended the cardano-node-tests with the ability for anybody to fork the repo and run all our System Tests on GitHub Actions
• added 2 new nightly pipelines - nightly-mixed and nightly-p2p - details here
• some optimizations on how our regression tests are scheduled on pytest workers and how cluster instances are assigned to the tests;

=== 743 passed, 67 skipped, 24 xfailed in 9166.64s (2:32:46) === to === 753 passed, 67 skipped, 14 xfailed in 4654.80s (1:17:34) ===

Node:​

• ran a couple of sanity runs of CLI a& sync tests on a local branch with P2P Single Relay enabled
• started the preparations for testing the next tag - details here

DB-Sync:​

• some improvements on db-sync sync tests

High level summary​

During the last 2 weeks we did more improvements on our Test Framework, planned the testing of the P2P Single Relay functionality, and also tested some DB-Sync tags.

Workstreams​

Framework improvements:​

• moved the System Test CLI Pipelines from BuildKite to Github Actions
• improved the reporting tools to support the rerun of the failled tests and update of the reports
• added support for Github API in report-aggregator, so reports will be generated from the GitHub nightly jobs from now on
• added support for mixed topology - P2P, legacy, mixed topologies
• planned the P2P Single Relay system test activities
• added support to start regression tests with PV8 + better selection of tests

DB-Sync:​

• confirmed that DB-Sync release 13.0.5 is compatible with Node release 1.35.4 + Protocol Version 8 (on the Preview environment)
• tested a couple db-sync tags - 13.1.0.0-rc1, 13.1.0.0-rc2