Skip to main content

58 posts tagged with "network"

View All Tags

· 2 min read
Marcin Szamotulski

High level overview of sprint 53​

We integrated bootstrap peers with cardano-node in an experimental branch ([cardano-node-5485], ouroboros-network-4615). We continued working on tx-submission: ouroboros-network-3311.

Karl Knutsson (CF) worked on peer sharing improvements: ouroboros-network-4778.

Erik de Castro Lopo made ouroboros-network packages compile with GHC-9.8: ouroboros-network-4727.

Nick Frisby (Tweag, Consensus Team), modified local-state-query mini-protocol so that one can acquire the immutable tip and thus get information which is not subject to rollbacks: ouroboros-network-4765.

Low level summary​

Other, less significant changes / improvements include:

· 2 min read
Marcin Szamotulski

High-level overview of sprint 52​

Happy New Year!

In this short sprint we analysed a failure which happened on a new large cluster that's run by IOG. The process exhausted all file handles and was left without any functional connections. The issues apparently is rare, and thus doesn't impose a high risk.

We also continued working on tx-submission: ouroboros-network-3311.

Detailed description​

It turned out that the process exhausted the number of file handles leaking multiple /proc/{PID}/stat files open. We suspect that the bug is caused by

  • using lazy IO in iohk-monitoring-framework, and
  • using a recent kernel version

With lazy IO file handles are read as long as the data is required and they are closed only when EOF is reached. We currently suspect that a new linux kernel added something at the end of the /proc/{PID}/stat which is not parsed by iohk-monitoring-framework, so whenever the file is read we leak it (it's never closed) and eventually, there are no file handles to be used by the network layer: the accept loop doesn't return any inbound connection, neither an outbound connection can be created. This issue will be addressed by the profiling team (which owns the logging subsystem).

The fix will be proposed in the future release, in the meantime we suggest to keep observing file handles used by the node.

I would like to thank John Lotoski (IOG), Karl Knutsson (CF), Neil Davies (PNSol) and Michael Karg (IOG) who all contributed to this analysis.

While analysing the log we also found a few smaller issues in the outbound governor which were fixed in [ouroboros-network-#4764].

The IO error indicating exhausting file handles is not currently visible. It is not re-thrown nor logged. This needs to be fixed in a future version. See ouroboros-network-4769.

· 2 min read
Marcin Szamotulski

High-level overview of sprint 51​

Outbound Governor Bug in cardano-node-8.7.2​

In the current sprint, we received a bunch of reports from SPOs about nodes not maintaining some connection when using cardano-node-8.7.2 (running in P2P mode). Such regressions are very important to us since they can lead to lost blocks. We were able to reproduce this issue. Every time there's a longer pause of block production (due to the statistical nature of Ouroboros), there is a chance that the bug will be armed. For this reason cardano-node-8.7.2 needs to be closely monitored.

We found the bug and developed a fix, ref. Karl Kntusson (CF) wasn't able to reproduce the bug with the patched version of the node for long enough (almost two weeks now) for us to belive that the fix is correct.

Advise for SPOs​

We created a release branch for 8.7.3. The advice from the network team is to either downgrade to the previous release, e.g. 8.1.2 or use the above release branch (note that there were no benchmarks made or Q&A tests yet).

Testing plans​

We were also able to reproduce the bug using IOSim, ouroboros-network#4757. However, the bug relies on a particular schedule of two threads which are involved and we needed to artificailly modify IOSim schedule in production code - something that we don't want to commit to the master branch. We also experimented with a randomised scheduler for IOSim, but that did not lead to finding the schedule which arms the bug: the search space grows exponentially with the number of steps in the threads, partial order reduction techniques implemented in IOSimPOR are more appropriate - unfortunatelly the simulation test is too large to be executed in IOSimPOR even with large amounts of RAM. To use IOSimPOR we need to implement a test which includes the two interacting components:

  • connection-manager
  • outbound-governor (where the bug was located)

which communicate through PeerStateActions, without including all the diffusion components as we do in our sim-net tests. More in style of outbound-governor tests where there is just a single outbound-governor, unlike in the sim-net which runs multiple communicating diffusions.

Bootstrap peers​

We continued working on bootstrap peers, ouroboros-network#4555

TxSubmission Decision Logic​

We continued working on tx-submission decision logic, ouroboros-network#3311

· 3 min read
Marcin Szamotulski

High-level overview of sprint 49 & sprint 50​

Fixed PeerSelection bug​

Karl Knutsson (Cardano Foundation (CF)) found a bug in the cardano-node-8.7.0 version used on the Sancho Net which was fixed in 8.7.1. It resulted in a node not being able to reconnect to an upstream peer once it was demoted by an asynchronous exception. This bug would be caught by Q&A in a mainet release, but for testnet releases, Q&A test suite is not used. We also developed a test which covers the bug in the ouroboros-network, we also identified a missing PeerSelection test which we need to port to our simulation network. See ouroboros-network#4734, ouroboros-network#4665.

Bootstrap Peers​

Still under review, ouroboros-network#4555. The consensus team is now implementing the API we need for bootstrap peers. Once consensus API is implemented we will integrate changes in an experimental branch of cardano-node.

Tx-Submission​

We started working on a new implementation of the tx-submission application. No tx-submission protocol changes are foreseen, but we want to be able to download each tx from just one upstream peer and share the results between different connections. We want to distribute the bandwidth between multiple clients. We also think that this work will prepare us for the future Ouroboros-Leios changes, which will contain various versions of tx-submission like mini-protocols. See ouroboros-network#4701.

Peer Sharing​

Various fixes and improvements were implemented:

  • ouroboros-network#4725

    • disabled peer sharing with initiator-only nodes: currently it's not possible to get peers from initiator-only nodes (edge nodes, e.g. wallets). In the future, we might change this, which will require running a server-side of the peer-sharing protocol by such nodes. See ouroboros-network#4726.
    • fixed peer-sharing codec
    • fixed a handshake bug which returned a wrong peer-sharing option

  • ouroboros-network#4728

    • disabled peer-sharing for NodeToNodeV_11 and NodeToNodeV_12

  • Karl Knutsson (CF) has been working on additional improvements, e.g. ouroboros-network#4735

With these fixes, Karl Knutsson (CF) was able to see that two peers on the mainnet can discover themselves through peer-sharing and keep being mutually useful and thus the connection surviving outbound-governor churn events.

IOSim​

We improved the memory footprint of IOSim in io-sim#126, see ouroboros-network#4721 for heap profile improvements on large test cases.

We are working on optimising the memory footprint of IOSimPOR. We are reimplementing VectorClocks using a trie, instead of a map which leads to significant improvements.

Cardano-Ping​

cardano-node-0.2.0.10 was released to CHaP, ouroboros-network#4746. This version exports more APIs which turned out to be useful in cardano-node test suite, see cardano-node#5536.

Technical Debt​

We addressed some small tech-debt issues in ouroboros-network#4722:

  • fixed some typos
  • using bracket instead of onException in withSnocket
  • improved haddocks
  • organised TracePeerSelection constructors

We improved the memory footprint of some of our tests in ouroboros-network#4721.

· One min read
Marcin Szamotulski

High-level overview of sprint 48​

Bootstrap Peers​

We continued reviewing bootstrap peers, ouroboros-network#4555.

IOClasses / IOSim​

We prepared slides for a Haskell meetup were we presented a talk on IOSimPOR. The recording will be availble on YouTube.

We also used the opportunity to do some refactoring of the IOSim code base: io-sim#117. We released io-sim-1.3.0.0 on Hackage: io-sim#119.

We also added forkFinally to MonadFork (not included in 1.3.0.0 release): io-sim#123.

Tech debt​

We refactored Resource used by the DNS subsystem: ouroboros-network#4707. We continued reviewing the ouroboros-network#4625 PR, which refactors RootPeersDNS module.