Strategy

Save some bandwidth by turning off TCP Timestamps

High Scalability

Oct 14, 2015 — 3 min read

This is a guest post by Donatas Abraitis, System Engineer at Vinted, with an unusual approach for saving a little bandwidth.

Looking at https://tools.ietf.org/html/rfc1323 there is a nice title: 'TCP Extensions for High Performance'. It's worth to take a look at date May 1992. Timestamps option may appear in any data or ACK segment, adding 12 bytes to the 20-byte TCP header.

Using TCP options, the sender places a timestamp in each data segment, and the receiver reflects these timestamps back in ACK segments. Then a single subtract gives the sender an accurate RTT measurement for every ACK segment.

To prove this let's dig into kernel source:

./include/net/tcp.h:#define TCPOLEN_TSTAMP_ALIGNED    12
./net/ipv4/tcp_output.c:static void tcp_connect_init(struct sock *sk)
  ...
  tp->tcp_header_len = sizeof(struct tcphdr) +
    (sysctl_tcp_timestamps ? TCPOLEN_TSTAMP_ALIGNED : 0);

Some visualizations:

net.ipv4.tcp_timestamp=0
TCP Header length
value |-------------------------------------------------- count
   20 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@         256
   22 |                                                     0
   24 |                                                     0

This shows that TCP header length is 20-bytes for 256 samples.

net.ipv4.tcp_timestamp=1
TCP Header length
value |-------------------------------------------------- count
   28 |                                                     0
   30 |                                                     0
   32 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@         256
   34 |                                                     0

After turning on timestamps graph shows that TCP header is already 32-bytes length.

This code was used to make these graphs using Systemtap:

%{
#include <linux/tcp.h>
%}

global __tcp_header;

function get_tcp_header_len:long(sk:long)
%{
        struct tcp_sock *tp = tcp_sk((struct sock *)STAP_ARG_sk);
        THIS->__retvalue = tp->tcp_header_len;
%}

probe kernel.function("tcp_connect_init").return
{
        __tcp_header <<< get_tcp_header_len($sk);
}

probe timer.s(1)
{
        ansi_clear_screen();
        println("TCP Header length");
        print(@hist_linear(__tcp_header, 20, 34, 2));
}

Some real case benchmarks:

I used two tools to measure performance differences using Flent and qperf.

The first run was using qperf. This benchmark was used to measure TCP bandwidth with TCP timestamps.

~$ for x in $(seq 1 10); do qperf --use_bits_per_sec --time 2 192.168.1.1 tcp_bw | tail -n1 | awk '{print $3}' ; done
9.39
9.4
9.4
9.4
9.39
9.4
9.39
9.39
9.38
9.39

This - after turning them off. You would see increased bandwidth by ~1%.

~$ for x in $(seq 1 10); do qperf --use_bits_per_sec --time 2 192.168.1.1 tcp_bw | tail -n1 | awk '{print $3}' ; done
9.46
9.46
9.45
9.47
9.49
9.47
9.46
9.48
9.46
9.47

The second tool Flent is much more informative and gives more details including built-in graphing option. I won't cover in this post any configuration about Flent, because it's well documented here.

Summary

Results show that it's reasonable to turn off timestamps on 10GE interfaces, but keep in mind that it should be performed only in low latency networks.

Here are the graphs (Green - download) for TCP download with appropriate timestamp switch: