So I’m back after a hiatus, but the good news is that I was finally able to do these network tests more systematically.
If you haven’t read Part 1, please do, it’s essential reading, but a TL;DR refresher is that I tried out SCTP over IPv6 and it worked for more than one of the internet connections where I have IPv6 available.
As a preamble, I must emphasise that somehow, a full 100% of the internet connections that I was able to lay my hands on do indeed support IPv6. This is without any far-sighted selection criterion for catching up to the new now 14-year-old version of the internet protocol, mind you. (And not super-ugly either, with separate IA_NA for the router and IA_PD for the LAN, etc. Albeit delegating only a single /64, but which is arguably not too bad.)
A neater setup
On this note, the story resumed from where I earlier tested connections only between machines I was physically present with. It, however, quickly became a little annoying:
Also, any odd behaviour I encountered always retained an uncertainty about whether it arose from the iperf client ISP or the server ISP. Isolating that required more than one test using different clients/servers, making each test double the work and half the insight.
So I bit the bullet and bought a really cheap IPv6-only VPS (thanks hostbrr!). For less than the price of some of even third-world country coffees, I got a quick, robust, and sophisticated setup experience, and most importantly with zero firewalls of any kind. The end-to-end internet is shocking and almost unnerving to experience first-hand. Just find the globally unique address of any other machine, anything at all in the wide world, and just send packets. There was absolutely nothing to futz around with. So obviously this became the reference machine and my device was acting as the client and then as the server for an iPerf test, allowing me to check internet connections one-by-one and one direction at a time.
Securing the borders a.k.a How to assume you’re valuable on the internet
But first, a few niggling issues had to be tackled, starting with security. While the number of SCTP-aware attackers on the internet is likely to be preeetty low, I was still uncomfortable leaving a default allow on my router firewall for it at all times (All my earlier tests were done by doing this temporarily. But enabling and disabling rules each time had quickly become a pain). Who knows what documented vulnerabilities and zero-days exist across my internal network… Unfortunately, my earlier router’s firewall (an EdgeRouter-X) did not support filtering SCTP traffic by ports, at least with stock firmware. This was because the required kernel module (nf_conntrack.ko) was not compiled for stateful filtering of SCTP out of the box, and needed to be recompiled with different config settings. Along with other issues relating to second-class support for IPv6, this prompted me to jump to OpenWRT which was happily fully supported on the same hardware.
It turned out that while stateful filtering still needed a recompile on OpenWRT as well, stateless filtering worked out of the box due to the move to nftables from iptables! I proceeded with it as an acceptable compromise for now, but ideally the stateful firewall is needed for a production setup. More details here and here on this matter, drop me a comment if you have further questions.
Blackholing a.k.a ghosting
The next big issue is something I referred to in the previous part when my local machine was the server: having to clamp MSS else I would lose packets entirely. This turned out to be pretty simply because of my machine connecting to the internet via PPPoE which added another 8 bytes of packet overhead compared to just plain ethernet. Thus the maximum IPv6 packet size (called the MTU), and in turn the SCTP packet size (the MSS), should have been that much smaller. This should not be a problem on sane networks, but this particular ISP was not informing my VPS, the sender, with the appropriate ICMP message. Instead the packets were simply dropped by the ISP routers on the inbound journey (aptly termed getting “blackholed”). Of course this is not SCTP-specific.
Too fast too soon
The last problem I had was ironically one of SCTP’s headline features, multihoming. You see, in a fit of optimism, I had multiple IPv6 addresses (from different GUA subnets!) available on my client due to both fiber connections being active and delegating their own prefixes to my home network (and both stable and privacy addresses enabled). And the SCTP association setup process by iperf on my local machine included all of them while handshaking (a 4-step process in contrast with TCP’s 3) with the VPS client. This was, for whatever reason, causing the expected next packet of the handshake to never be sent from the client. I haven’t yet investigated what the reason is, but since the packet was never even sent, the network wasn’t at fault. I worked around this issue by specifically asking iperf to bind to a single address, which worked around the problem successfully.
Opening the borders a.k.a how to get out of a cage that’s technically yours
During my tests with the mobile service providers, I used USB tethering (or Wi-Fi hotspot, no difference for our purposes). The Jio “SIM card” happened to be in an iPhone, and it turns out that the iPhone behaves significantly different from Android. The way it works is that the mobile has one IPv6 address but the tethered devices are on a different prefix entirely. And Android has been super-cooperative with me in that it leaves the tethered subnet entirely wide-open to the internet, no filtering. That, however is not the case with iPhone, sadly, which seems to put in some sort of firewall in place. I haven’t dug deep enough into this, for e.g. to deactivate/configure the firewall, I just popped the card to an Android phone temporarily.
The sweet fruit of success
After dealing with all these local device level software issues, I ran tests across 3 of the major mobile providers and 2 FTTH/FTTN wired internet connections available with me. This set of results essentially supersede my results in part 1. When we look at the broader internet i.e. the one we don’t have direct control over, the result is:
| Type | Local connection | As client | As server |
|---|---|---|---|
| Fixed-line | ACT (FTTN) | Success | Success |
| Fixed-line | BSNL (FTTH) | Success | Success |
| Mobile | Airtel (LTE) | Success | Success |
| Mobile | Jio (LTE) | Success | Success |
| Mobile | Vi (LTE) | Success | ISP-side stateful firewall drops packets |
ALL of them allowed iPerf SCTP outbound, and all but 1 allowed it inbound. AND, Vi (LTE) was blocking ALL unsolicited inbound traffic, so not an ossification problem!
This is a massively positive finding: no ossification and no middleboxes. (I’m quite happy to bet quite a bit that SCTP was not explicitly allowlisted in their firewalls 🙂). The Indian IPv6 internet is free and ripe for transport protocol (Layer 4) innovation!
Epilogue
This doesn’t mean I didn’t face challenges, far from it as just described above. Overcoming them required some tenacity, some ugly hacks/workarounds, as well as a measure of luck. But all of that was encountered at the end-points, in my own devices. This is exactly the opposite of what a lot of the IETF literature, internet enthusiasts and blogosphere claim.
What are the next steps? One important thing is to check if multihoming works, which gives us an immediate and attractive reason to keep interest in this. The speed is also significantly lower than what I was able to achieve via TCP, something necessary to fix if it is to be mainstreamed someday. Another thing is to find applications that would benefit from using SCTP directly. This is not an easy problem to tackle, as its double-sided — both a server and client have to support it to work. We will tackle these in part 3. Till then, Sayonara.