You might have heard about HTTP/3, which is used in on top of QUIC – a spanking-new transport layer protocol that replaces TCP. (If you’re thinking, “waitaminute, didn’t HTTP/2 come out just recently?!”, you’re not alone; its like suddenly there’s an urgency to develop on what was previously iterated slooowwwly and carefully).
The venerable TCP has been around for around 50 years now, but has had revisions throughout the years, some as recently as 2001. With changing patterns of usage (driven primarily by dominance of web traffic), further evolution was found necessary. The consensus view is that TCP today has been “ossified” in its current configurations by network middle-boxes. Hence QUIC, proposed and developed in large part by Google in the last decade, is intended to replace TCP, at least for web uses. So clearly, there was no other candidate protocol… right?
There was a choice developed in 90s, and standardised for around 20 years now. Stream Control Transport Protocol (SCTP) was originally intended for PSTN (telephone) signalling, but is a general-purpose modern transport protocol. It supports streams, one of the headline features of QUIC (and HTTP/2 as well), and multi-homing1, a feature that is not yet supported by QUIC. So why did SCTP not replace TCP?
For the same reason QUIC has to be tunnelled over UDP (it is defined as such in the QUIC specification, it’s not a choice). SCTP is one of the poster-children of protocol-ossification — network equipment that you don’t control sitting between you and your desired content, interfering with protocols in ways you can’t usually change such that you’re forced to use the lowest common-denominator protocol. The biggest of these ways is to only pass-through TCP (and, at best UDP), branding everything else as unusual and simply dropping it in the name of security.
IPv6: the game-changer
But what about IPv6? IPv6 does not need or want NA(P)T. The question arises — would SCTP face the same problem on IPv6? Sometimes, the right answer is to just try it out and see what happens and how bad it is…
To test whether SCTP works or not, I’ll be using iperf3. First step is to use SCTP in a local, private network – good things begin at home.
Using Fedora 38 on both ends, I had to do the following to get iperf to connect and test successfully:
- Enable kernel SCTP module with
modprobe sctp - Allow TCP and SCTP port 5201 in the firewall — the TCP port is to allow iperf to do some apparently necessary coordination b/w client and server before the actual bulk transfer test
Here’s how that looks
[milind@desktop-linux ~]$ iperf3 -c 192.168.1.212 --sctp
Connecting to host 192.168.1.212, port 5201
[ 5] local 192.168.1.107 port 58950 connected to 192.168.1.212 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 66.2 MBytes 556 Mbits/sec
[ 5] 1.00-2.00 sec 67.0 MBytes 562 Mbits/sec
[ 5] 2.00-3.00 sec 72.5 MBytes 608 Mbits/sec
[ 5] 3.00-4.00 sec 70.1 MBytes 588 Mbits/sec
[ 5] 4.00-5.00 sec 66.7 MBytes 560 Mbits/sec
[ 5] 5.00-6.00 sec 65.2 MBytes 547 Mbits/sec
[ 5] 6.00-7.00 sec 57.9 MBytes 486 Mbits/sec
[ 5] 7.00-8.00 sec 63.2 MBytes 530 Mbits/sec
[ 5] 8.00-9.00 sec 68.4 MBytes 574 Mbits/sec
[ 5] 9.00-10.00 sec 63.2 MBytes 531 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 661 MBytes 554 Mbits/sec sender
[ 5] 0.00-10.00 sec 660 MBytes 554 Mbits/sec receiver
iperf Done.Well it works, phew, but why such low transfer throughput? Here’s how it looks for TCP
[milind@desktop-linux ~]$ iperf3 -c 192.168.1.212
Connecting to host 192.168.1.212, port 5201
[ 5] local 192.168.1.107 port 33508 connected to 192.168.1.212 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 116 MBytes 970 Mbits/sec 0 662 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 944 Mbits/sec 0 662 KBytes
[ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec 0 696 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 944 Mbits/sec 0 696 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 944 Mbits/sec 0 757 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 944 Mbits/sec 0 757 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 0 757 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 944 Mbits/sec 0 757 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 944 Mbits/sec 0 757 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 944 Mbits/sec 0 757 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 944 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver
iperf Done.What gives? UDP?
[milind@desktop-linux ~]$ iperf3 -c 192.168.1.212 --udp
Connecting to host 192.168.1.212, port 5201
[ 5] local 192.168.1.107 port 51952 connected to 192.168.1.212 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 129 KBytes 1.05 Mbits/sec 91
[ 5] 1.00-2.00 sec 127 KBytes 1.04 Mbits/sec 90
[ 5] 2.00-3.00 sec 129 KBytes 1.05 Mbits/sec 91
[ 5] 3.00-4.00 sec 127 KBytes 1.04 Mbits/sec 90
[ 5] 4.00-5.00 sec 129 KBytes 1.05 Mbits/sec 91
[ 5] 5.00-6.00 sec 129 KBytes 1.05 Mbits/sec 91
[ 5] 6.00-7.00 sec 127 KBytes 1.04 Mbits/sec 90
[ 5] 7.00-8.00 sec 129 KBytes 1.05 Mbits/sec 91
[ 5] 8.00-9.00 sec 127 KBytes 1.04 Mbits/sec 90
[ 5] 9.00-10.00 sec 129 KBytes 1.05 Mbits/sec 91
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.000 ms 0/906 (0%) sender
[ 5] 0.00-10.00 sec 1.25 MBytes 1.05 Mbits/sec 0.027 ms 0/906 (0%) receiver
iperf Done.Uhm, so much for a gigabit link through a 2.5g switch. Something is clearly up…
Okay, so it works on my local network at home. Big deal. That isn’t the “internet” by any stretch of imagination. Alright then, onto the next step. I guess I’m lucky that my ISP gives me a single full /64 IPv6 subnet of addresses (although it’s “not official” and I had to figure out how to enable it myself). With that I venture forth with no interest in legacy technologies like IPv4 <flips non-existent long hair backward arrogantly>.
So I got a friend with a Raspberry Pi to hook it to an LTE (with an Airtel SIM) USB modem (usually called a “dongle”) and configured it to not bother with any intermediate shenanigans routing2. I SSHed into it (Yep, that was also over IPv6!) to control and observe both sides. I then tried the same experiment over IPv6 using SCTP. With bated breath
[milind@desktop-linux ~]$ iperf3 -c sctptest.sampleddns.net -6 --sctp -M 1000
Connecting to host sctptest.sampleddns.net, port 5201
[ 5] local 2406:7400:9f:xxxx:xxxx:xxxx:xxxx:xxxx port 39980 connected to 2401:4900:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 832 KBytes 6.81 Mbits/sec
[ 5] 1.00-2.00 sec 1.38 MBytes 11.5 Mbits/sec
[ 5] 2.00-3.00 sec 1.38 MBytes 11.5 Mbits/sec
[ 5] 3.00-4.00 sec 1.31 MBytes 11.0 Mbits/sec
[ 5] 4.00-5.00 sec 1.31 MBytes 11.0 Mbits/sec
[ 5] 5.00-6.00 sec 1.19 MBytes 9.96 Mbits/sec
[ 5] 6.00-7.00 sec 1.06 MBytes 8.91 Mbits/sec
[ 5] 7.00-8.00 sec 1.38 MBytes 11.5 Mbits/sec
[ 5] 8.00-9.00 sec 1.31 MBytes 11.0 Mbits/sec
[ 5] 9.00-10.00 sec 1.38 MBytes 11.5 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 12.5 MBytes 10.5 Mbits/sec sender
[ 5] 0.00-10.02 sec 12.4 MBytes 10.4 Mbits/sec receiver
So SCTP over the open internet works!! (Note: hostname shown is faked)
On one of the tries, I had to change the Maximum Segment Size (MSS) to 1000 bytes instead of 1424 that it assumed to begin with. Some experimentation showed the max MSS to be 1283 — significance is as yet unknown. However, on another try on a different day it worked without any such parameter. Further digging required, clearly.
Another trial, this time using a hotspot from my phone’s LTE (by Vodafone-Idea, or as they glibly call themselves, “Vi”) connection to this Raspberry Pi.
[milind@desktop-linux ~]$ iperf3 -c sctptest.sampleddns.net --sctp
Connecting to host sctptest.sampleddns.net, port 5201
[ 5] local 2402:8100:25xx:xxxx:xxxx:xxxx:xxxx:xxxx port 42110 connected to 2401:4900:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 128 KBytes 1.05 Mbits/sec
[ 5] 1.00-2.00 sec 64.0 KBytes 525 Kbits/sec
[ 5] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec
[ 5] 3.00-4.00 sec 256 KBytes 2.10 Mbits/sec
[ 5] 4.00-5.00 sec 192 KBytes 1.57 Mbits/sec
[ 5] 5.00-6.00 sec 192 KBytes 1.57 Mbits/sec
[ 5] 6.00-7.00 sec 192 KBytes 1.57 Mbits/sec
[ 5] 7.00-8.00 sec 192 KBytes 1.57 Mbits/sec
[ 5] 8.00-9.00 sec 128 KBytes 1.05 Mbits/sec
[ 5] 9.00-10.00 sec 192 KBytes 1.57 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 1.62 MBytes 1.36 Mbits/sec sender
[ 5] 0.00-10.26 sec 1.56 MBytes 1.28 Mbits/sec receiver
iperf Done.Trying it the other way round (from RPi to desktop) didn’t work at first, either because a firewall somewhere is “protecting” me, or because there truly are middle-boxes at play… ?
To complete the triangle, I also ran the test on its third leg, between my desktop on FTTN and laptop on the LTE-fed hotspot. I could not get it to work with desktop as server and laptop as client, until I tried opening the port on my trusty Ubiquiti EdgeRouterX firewall (I’m not going to show it here, both because it occupies space and because I’m embarrassed at the low speed of Wi-Fi on the laptop.)
Switching back to the RPi on Airtel as a client and my desktop as a server, it was successful, showing it was indeed my firewall blocking connections.
| ISP, Connection method | Client ↓ ＼ Server → | Desktop | Laptop | Raspberry Pi |
| ACT Fibernet, PPPoE | Desktop | No | Yes | |
| Vi, LTE hotspot hosted by phone | Laptop | Yes | Yes | |
| Airtel, LTE direct | Raspberry Pi | Yes | No |
Overall the score is 2/3 internet connections work okay, and one may or may not be affected by Android’s firewall.
So there’s something to this after all, eh? Other than the MSS, I encountered issues only with firewalls on my devices — hardly “middle-boxes”.
The next step is to of course verify that Vi LTE also really does support SCTP on IPv6. If that is the case, then a good test (which I hope to do “soon”) would be to verify among all the major mobile operators and the major wired ISPs which do support IPv6.
To be sure, this is a very small sample set, and IPv6 is not exactly dominant on the internet yet. But it does show that sometimes the Right Thing ™ happens even without explicit intent, and we should be attentive to such opportunities to grab them. The real lesson is to not give up before even trying.
- I must take a sentence out to forcefully recommend Avery Pennarun’s article on networking. Nominally it is about IPv6, but each digression grants an interested reader with deep insight; it is the very best on the topic. ↩︎
- Which meant that on IPv4 I got a single “public” address directly from the ISP, which was of course lovingly CGNATed. ↩︎
Reading through Nmap Reference Guide where SCTP is described as “relatively new alternative to the TCP and UDP protocols” I doubted it even worked in 2023. Glad to be proven wrong! Just a heads up, the poor performance you got on UDP was because of default settings that put limit on bandwidth to 1Mbps. Try again with “-b 1G” to check the full potential.
FWIW, I did not start off with much hope that SCTP would work anywhere either, just a vague intuition that US ISPs are perhaps more “interventionist” than average.
Thanks for the tip on iperf default UDP bandwidth, I suspected it was something like that!