Skip to main content

50 posts tagged with "network"

View All Tags

· 2 min read
Marcin Szamotulski

High-level overview of sprint 55​

This sprint the networking was nominated for the Cardano Release Engineer role and we've been working on the integration of bootstrap peers with cardano-node. This is a work in progress: cardano-node-5485. cardano-node-8.9 will be the first node which supports the bootstrap peers feature, but we anticipate additional improvements needed before this is used at scale.

We disabled mean reward for new peers in the churn mechanism which was hurting connection in remote destinations. Thanks to Karl Knutsson (CF) for providing evidence and a patch. In future, this mechanism will be further changed, but this requires more analysis. ouroboros-network#4800

We continued working on tx-submission, ouroboros-network#3311

We welcome Marcin Wójtowicz as part of the Networking Team of Cardano Core Tech. He reviewed our documentation as part of his onboarding process: ouroboros-network#4802.

Andrew Westberg (DripDropz) corrected documentation regarding which node-to-node protocol version introduced P2P.

Low-level summary​

Open Source Contributions​

IOSim​

We released io-sim-1.4.1.0 with some new features & improvements:

· 2 min read
Marcin Szamotulski

High-level overview of sprint 54​

In this sprint, we merged the bootstrap peers (aka weak genesis) changes to the ouroboros-network repository (ouroboros-network#4615). We continued working on integration & testing it using an early development version of cardano-node (cardano-node#5485).

Andrew Westberg (DripDropz) noticed and fixed a mistake in the cddl spec which resulted in misleading documentation; ouroboros-network#4790. The Handshake protocol flag should be called initiatorOnlyDiffusionMode. Library authors providing connectivity with other nodes via the node-to-node protocol are advised to review their implementation.

We continued working on the tx-submission decision logic; ouroboros-network#3311.

Karl Knutsson (CF) implemented:

Open Source Contributions​

IOSim​

We released a new version of io-sim to Hackage:

NoThunks​

We released a new version of nothunks to Hackage:

Low-level summary​

· 2 min read
Marcin Szamotulski

High level overview of sprint 53​

We integrated bootstrap peers with cardano-node in an experimental branch ([cardano-node-5485], ouroboros-network-4615). We continued working on tx-submission: ouroboros-network-3311.

Karl Knutsson (CF) worked on peer sharing improvements: ouroboros-network-4778.

Erik de Castro Lopo made ouroboros-network packages compile with GHC-9.8: ouroboros-network-4727.

Nick Frisby (Tweag, Consensus Team), modified local-state-query mini-protocol so that one can acquire the immutable tip and thus get information which is not subject to rollbacks: ouroboros-network-4765.

Low level summary​

Other, less significant changes / improvements include:

· 2 min read
Marcin Szamotulski

High-level overview of sprint 52​

Happy New Year!

In this short sprint we analysed a failure which happened on a new large cluster that's run by IOG. The process exhausted all file handles and was left without any functional connections. The issues apparently is rare, and thus doesn't impose a high risk.

We also continued working on tx-submission: ouroboros-network-3311.

Detailed description​

It turned out that the process exhausted the number of file handles leaking multiple /proc/{PID}/stat files open. We suspect that the bug is caused by

  • using lazy IO in iohk-monitoring-framework, and
  • using a recent kernel version

With lazy IO file handles are read as long as the data is required and they are closed only when EOF is reached. We currently suspect that a new linux kernel added something at the end of the /proc/{PID}/stat which is not parsed by iohk-monitoring-framework, so whenever the file is read we leak it (it's never closed) and eventually, there are no file handles to be used by the network layer: the accept loop doesn't return any inbound connection, neither an outbound connection can be created. This issue will be addressed by the profiling team (which owns the logging subsystem).

The fix will be proposed in the future release, in the meantime we suggest to keep observing file handles used by the node.

I would like to thank John Lotoski (IOG), Karl Knutsson (CF), Neil Davies (PNSol) and Michael Karg (IOG) who all contributed to this analysis.

While analysing the log we also found a few smaller issues in the outbound governor which were fixed in [ouroboros-network-#4764].

The IO error indicating exhausting file handles is not currently visible. It is not re-thrown nor logged. This needs to be fixed in a future version. See ouroboros-network-4769.

· 2 min read
Marcin Szamotulski

High-level overview of sprint 51​

Outbound Governor Bug in cardano-node-8.7.2​

In the current sprint, we received a bunch of reports from SPOs about nodes not maintaining some connection when using cardano-node-8.7.2 (running in P2P mode). Such regressions are very important to us since they can lead to lost blocks. We were able to reproduce this issue. Every time there's a longer pause of block production (due to the statistical nature of Ouroboros), there is a chance that the bug will be armed. For this reason cardano-node-8.7.2 needs to be closely monitored.

We found the bug and developed a fix, ref. Karl Kntusson (CF) wasn't able to reproduce the bug with the patched version of the node for long enough (almost two weeks now) for us to belive that the fix is correct.

Advise for SPOs​

We created a release branch for 8.7.3. The advice from the network team is to either downgrade to the previous release, e.g. 8.1.2 or use the above release branch (note that there were no benchmarks made or Q&A tests yet).

Testing plans​

We were also able to reproduce the bug using IOSim, ouroboros-network#4757. However, the bug relies on a particular schedule of two threads which are involved and we needed to artificailly modify IOSim schedule in production code - something that we don't want to commit to the master branch. We also experimented with a randomised scheduler for IOSim, but that did not lead to finding the schedule which arms the bug: the search space grows exponentially with the number of steps in the threads, partial order reduction techniques implemented in IOSimPOR are more appropriate - unfortunatelly the simulation test is too large to be executed in IOSimPOR even with large amounts of RAM. To use IOSimPOR we need to implement a test which includes the two interacting components:

  • connection-manager
  • outbound-governor (where the bug was located)

which communicate through PeerStateActions, without including all the diffusion components as we do in our sim-net tests. More in style of outbound-governor tests where there is just a single outbound-governor, unlike in the sim-net which runs multiple communicating diffusions.

Bootstrap peers​

We continued working on bootstrap peers, ouroboros-network#4555

TxSubmission Decision Logic​

We continued working on tx-submission decision logic, ouroboros-network#3311