During the lightly staffed holiday period for node SRE, the emphasis was on maintaining environment stability, tuning and resolving any noisey alerts.
Investigation into and testing around the following two topics also started during this period:
Ledger snapshots causing a small number of missed slots for forgers on mainnet: ouroboros-consensus-issue-868
A cardano-node rare file descriptor leak, with a more detailed description here
cardano-api consumers who are primarily concerned with the current mainnet era and the next era for an upcoming hardfork.cardano-cli, in particular to the babbage era commands where some Conway options had spilled. --output-format, like other commands, instead of --yamlquery pool-state default to returning information on all pools
This week, the Hydra team made significant progress, implementing an offline mode with associated refactoring. They enhanced user experience by detecting incompatible eras in hydra-node. Protocol changes were implemented for incremental decommits, addressing off-chain logic. Additionally, the team contributed fixes to cardano-ledger and coordinated with the Eternl team on enabling committing into a head from their wallet.
hydra-node and provide better UX
#1216cardano-ledger#3949 and
#3953
cardano-node-8.7.2In the current sprint, we received a bunch of reports from SPOs about nodes not
maintaining some connection when using cardano-node-8.7.2 (running in P2P
mode). Such regressions are very important to us since they can lead to lost
blocks. We were able to reproduce this issue. Every time there's a longer
pause of block production (due to the statistical nature of Ouroboros), there
is a chance that the bug will be armed. For this reason cardano-node-8.7.2
needs to be closely monitored.
We found the bug and developed a fix, ref. Karl Kntusson (CF) wasn't able to reproduce the bug with the patched version of the node for long enough (almost two weeks now) for us to belive that the fix is correct.
We created a release branch for 8.7.3. The advice from
the network team is to either downgrade to the previous release, e.g. 8.1.2
or use the above release branch (note that there were no benchmarks made or Q&A
tests yet).
We were also able to reproduce the bug using IOSim, ouroboros-network#4757.
However, the bug relies on a particular schedule of two threads which are
involved and we needed to artificailly modify IOSim schedule in production
code - something that we don't want to commit to the master branch. We also
experimented with a randomised scheduler for IOSim, but that did not lead to
finding the schedule which arms the bug: the search space grows exponentially
with the number of steps in the threads, partial order reduction techniques
implemented in IOSimPOR are more appropriate - unfortunatelly the simulation
test is too large to be executed in IOSimPOR even with large amounts of
RAM. To use IOSimPOR we need to implement a test which includes the two
interacting components:
connection-manageroutbound-governor (where the bug was located)which communicate through PeerStateActions, without including all the
diffusion components as we do in our sim-net tests. More in style of
outbound-governor tests where there is just a single outbound-governor,
unlike in the sim-net which runs multiple communicating diffusions.
We continued working on bootstrap peers, ouroboros-network#4555
We continued working on tx-submission decision logic, ouroboros-network#3311
The SRE team continues work on cardano environment improvements and general environment maintenance.
Some notable recent changes, updates or improvements include: