« Stuff The Internet Says On Scalability For June 3, 2011 | Main | Awesome List of Advanced Distributed Systems Papers »

Why is your network so slow? Your switch should tell you.

Who hasn't cursed their network for being slow while waiting for that annoying little hour glass of pain to release all its grains of sand? But what's really going on? Is your network really slow? PacketPushers Show 45 – Arista – EOS Network Software Architecture has a good explanation of what may be really at fault (paraphrased):

Network operators get calls from application guys saying the network is slow, but the problem is usually dropped packets due to congestion. It's not usually latency, it's usually packet loss. Packet loss causes TCP to back off and retransmit, which causes applications to appear slow.

Packet loss can be caused by a flakey transceiver, but the problem is usually network congestion. Somewhere on the network there's fan-in, a bottleneck develops, queues build up to a certain point, and when a queue overflows it drops packets. Often the first sign of this happening is application slowness.

Queues get deeper and deeper because the network is getting more and more use over time. Traffic is continually being added.

Get your switch to tell you about your queue stats so you can take proactive action. Find out if your network is going to drop packets before it happens. Polling doesn't work. Polling for stats will miss problems because you can miss queue threshold crossings. You need alerts to be correct. Polling can't see the microbursts which cause loss. It doesn't take a lot of loss to cause problems.  

An obvious fix is to reduce the traffic on your network. Perhaps applications are in tight loops and are flooding the network. Maybe your network needs upgrading.  In some cases a WAN accelerator might help.

But the idea that your network infrastructure shouldn't be hiding problems, but should be telling you when you are having problems, is an overlooked one. From a code perspective it's just impossible to tell what's really happening on the underlying layers without help from the network.

The podcast also has a good discussion of modern embedded architecture practices, but I thought this take on network issues, and how your network infrastructure should be helping you debug root causes, was helpful and worth pulling out.

Related Articles

Reader Comments (2)

Polling for discards / errors should be able to tell you when your queues are exceeded. Of course not all hardware shows this on the interface you would think. Harder to poll for is when queues get so long that TCP sessions no longer get notice that there is congestion. I have read some stuff on buffer bloat that is interesting. Check out www.bufferbloat.net for some interesting info. Matching queue size to transmit rate gets harder when you have speed conversions from 10Gbps down to 10 Mbps. Auto tuning queue size and AQM seem pretty important.

June 2, 2011 | Unregistered Commenterj1010y

Very nice explanation. Laid our beautifully and easy to understand.

Short and sweet, but nice and effective.

Without knowledge of switches, you will never run a successful network.

June 10, 2011 | Unregistered CommenterDarren H

PostPost a New Comment

Enter your information below to add a new comment.
Author Email (optional):
Author URL (optional):
Some HTML allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>