Why is your network so slow? Your switch should tell you.
Who hasn't cursed their network for being slow while waiting for that annoying little hour glass of pain to release all its grains of sand? But what's really going on? Is your network really slow? PacketPushers Show 45 – Arista – EOS Network Software Architecture has a good explanation of what may be really at fault (paraphrased):
Network operators get calls from application guys saying the network is slow, but the problem is usually dropped packets due to congestion. It's not usually latency, it's usually packet loss. Packet loss causes TCP to back off and retransmit, which causes applications to appear slow.
Packet loss can be caused by a flakey transceiver, but the problem is usually network congestion. Somewhere on the network there's fan-in, a bottleneck develops, queues build up to a certain point, and when a queue overflows it drops packets. Often the first sign of this happening is application slowness.
Queues get deeper and deeper because the network is getting more and more use over time. Traffic is continually being added.
Get your switch to tell you about your queue stats so you can take proactive action. Find out if your network is going to drop packets before it happens. Polling doesn't work. Polling for stats will miss problems because you can miss queue threshold crossings. You need alerts to be correct. Polling can't see the microbursts which cause loss. It doesn't take a lot of loss to cause problems.
An obvious fix is to reduce the traffic on your network. Perhaps applications are in tight loops and are flooding the network. Maybe your network needs upgrading. In some cases a WAN accelerator might help.
But the idea that your network infrastructure shouldn't be hiding problems, but should be telling you when you are having problems, is an overlooked one. From a code perspective it's just impossible to tell what's really happening on the underlying layers without help from the network.
The podcast also has a good discussion of modern embedded architecture practices, but I thought this take on network issues, and how your network infrastructure should be helping you debug root causes, was helpful and worth pulling out.
