The joys of carrier-grade NAT
Neuland
Living in the capital of Neuland doesn't guarantee to have decent internet connectivity. Some 20 years ago I lived in a much smaller city and it really mattered in which street you lived in: either you'd only get ISDN or you were lucky enough to move into a building where DSL is available. And apparently nothing really changed in all these years, except now you can choose between DSL and cable, and if you're lucky FTTC. But at my address no DSL and no fiber connections are available, only cable. One can choose from two (2) companies, one is called Pÿur and o2. Rolled a dice and went with the former and was pretty satisfied at first: stable enough service, a real flat rate, almost as fast as advertised, and the hotline (needed to exchange a modem once) was easy to reach and even appeared to be somewhat competent.
But all that changed a few weeks ago when Pyur decided to switch me over to CGN, or Carrier-grade NAT, one of these abominations some providers feel necessary when trying to deal with the impending IPv4 address exhaustion. Yes, I've called up their hotline, but every time I call them, the less the person on the other end understands the topic, and they don't really know what they are doing. The case is open since October now with no solution in sight.
So, how do we deal with that? IPv6 of course, as the modem still receives a valid, reachable IPv6 address so as long as I have IPv6 connectivity I can reach my little home network just fine, even via DNS. But I may not have IPv6 connectivity everywhere I go and thus may not be able to reach my home network. Also, there are a couple remote machines that used to access my network but not have an IPv6 uplink and are limited to IPv4.
Solutions
Luckily I have this little hosted VM somewhere on the internet equipped with a static IPv4 (and IPv6) address. That will be my little helper host to forward traffic. Let's use some ASCII art to sketch out the scenario:
|-- IPv4, CGN ------- NOT accessible from the IPv4 world
|
|
|-- VPN -- Hosted VM --- accessible from the IPv4 world
| with IPv4
| connectivity
_______ |
|machine| |
| at .|----- CPE -----|
| home .| (FritzBox) |
|_______| |-- IPv6, public -------- accessible from the IPv6 world
DNS
To bring DNS into the game: we will update our dynamic DNS record separately for IPv4 and for IPv6. For example, with DuckDNS:
curl -s "https://www.duckdns.org/update?domains=${DOMAIN}&token=${TOKEN}&ip4=${IP4}"
curl -s "https://www.duckdns.org/update?domains=${DOMAIN}&token=${TOKEN}&ipv6=${IP6}"
That results in:
$ dig +short -t A example.duckdns.org 93.184.216.34 # The address of our hosted VM $ dig +short -t AAAA example.duckdns.org 2606:2800:220:1:248:1893:25c8:1946 # The address of our CPE at home
All requests over IPv6 will terminate directly on the CPE (that does the port-forwarding to the actual machine serving the request) and all requests over IPv4 will terminate at the hosted VM. Here, we'll need to use some tricks to get these packets forwarded to our home network after all.
VPN
The machine in the home network will be connected via VPN to the hosted VM, so packets can flow back and forth via that tunnel connection. I went with a small VPN daemon named fastd, as the setup was easy enough and worked quite well, and was already in place for other applications. There are some serious performance issues here, but these will be discussed below and in a following post. A switch to WireGuard (the obvious choice these days) may or may not make a difference here, this is yet to be seen.
HTTPS
The most obvious problem to be solved was the web server running on my home network. To make it accessible via IPv4 again we need to use some kind of reverse proxy on our hosted VM.
-
A lighttpd instance was already running here. However, we needed to proxy to a HTTPS server but apparently that's not possible with mod_proxy.
-
The next contender was Nginx of course, since this was running on the other end as well. But all the same, Nginx is not able to proxy to HTTPS servers just so. An out-of-tree http_proxy_connect_module would to the trick, but this might be an exercise for another day.
-
Finally we settled for Apache because it can proxy to HTTPS servers out-of-the box (with both mod_proxy_connect and mod_proxy_http enabled):
<VirtualHost *:80>
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
CustomLog /var/www/logs/access_notls.log combined
Protocols h2 http/1.1
</VirtualHost>
<VirtualHost *:443>
ServerName example.duckdns.org
ServerAlias sub.example.duckdns.org
Protocols h2 http/1.1
ErrorLog /var/www/logs/error_example.log
CustomLog /var/www/logs/access_example.log combined
SSLEngine on
SSLCertificateFile /etc/ssl/private/example.pem
SSLCertificateKeyFile /etc/ssl/private/example.key
SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerName off
# SSLProxyCheckPeerExpire off
ProxyPass / https://example.fastd/
ProxyPassReverse / https://example.fastd/
Here, example.fastd is basically just an /etc/hosts entry pointing to the
other end of that VPN connection, i.e. the server in my home network. Notice the
SSLProxyCheckPeerName
option set since the TLS certificate on the server will only be valid for
example.duckdns.org of course, not for example.fastd.
This works quite well and is fast enough for our little web server instance. It
even does the Right Thing™ for ACME clients
most of the time. For dehydrated we need to add
the --ipv6 switch so that the home network server will handle the
verification steps (i.e. placing a file into the .well-known/acme-challenge/ directory).
Loghost
Several external machines are configured to use our machine at home as a loghost. That is, they are sending syslog messages to a Syslog-ng instance over syslog-tls. But some of these machines can only use IPv4. Let's reconfigure these machines to use our hosted VM as a loghost instead, and then forward these packets through the VPN to the machine at home. Most of these external machines have Rsyslog installed, i.e. they are acting as the syslog client:
$ cat /etc/rsyslog.d/local.conf $ActionWriteAllMarkMessages on $PreserveFQDN on $DefaultNetstreamDriverCAFile /etc/ssl/certs/ISRG_Root_X1.pem $DefaultNetstreamDriver gtls $ActionSendStreamDriverMode 1 $ActionSendStreamDriverAuthMode 509/certvalid # Or use "anon" kern,mark.*;*.warning @@hosted-vm.example.net:6514
Similarly to the SSLProxyCheckPeerName option above, we need to set
ActionSendStreamDriverAuthMode to allow the TLS connection because
the subject name cannot be validated.
On the hosted VM we will use iptables nftables
to mangle our packets accordingly:
# Enable IP forwarding
$ sysctl net.ipv4.ip_forward=1
# Setup netfilter rules
nft add table ip nat
nft add chain ip nat prerouting '{ type nat hook prerouting priority -100; }'
nft add chain ip nat postrouting '{ type nat hook postrouting priority 100; }'
nft add rule ip nat prerouting tcp dport 6514 dnat to example.fastd
nft add rule ip nat postrouting ip daddr example.fastd masquerade
That wasn't quite enough and we still had to adjust the policy of our
forward
chain to accept:
$ cat /etc/nftables.conf
[...]
chain forward {
type filter hook forward priority 0; policy accept;
}
With all that in place, packets to our hosted VM on 6514/tcp will be sent to
our machine at home.
SSH
While our machine at home can be reached via IPv6 just fine, we may want to setup a backup route via IPv4, that is our hosted VM. For that we'll add another Netfilter rule and our configuration should now look something like this:
table ip nat {
chain prerouting {
type nat hook prerouting priority dstnat; policy accept;
tcp dport 2022 dnat to example.fastd:22
tcp dport 6514 dnat to example.fastd
}
chain postrouting {
type nat hook postrouting priority srcnat; policy accept;
ip daddr example.fastd masquerade
}
}
Performance
The setup above works quite well most of the time, but when trying to use a more network-intensive application, for example Nextcloud, the poor network performance really shows.
This will be covered in detail in the next post, but let's dump some benchmarks here:
# Running speedtest on the hosted VM, which located in a well # connected data center in Europe: $ speedtest RTT: 1.685 ms Testing download speed......................................................... Download: 4224.57 Mbit/s Testing upload speed........................................................... Upload: 3174.84 Mbit/s # Running speedtest on my machine at home, with an advertised 500 Mbps uplink: $ speedtest RTT: 30.911 ms Testing download speed......................................................... Download: 432.97 Mbit/s Testing upload speed........................................................... Upload: 26.90 Mbit/s
So, we are severely limited by our uplink at home of course. Perhaps not as much by its upload speed, but maybe more but the huge round round-trip time.
Use iperf3 to see how fast we can go between our hosts:
$ iperf3 -fM -c hosted-vm.example.net [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 4.37 MBytes 4.37 MBytes/sec 0 499 KBytes [ 5] 1.00-2.00 sec 3.63 MBytes 3.63 MBytes/sec 0 638 KBytes [ 5] 2.00-3.00 sec 2.50 MBytes 2.50 MBytes/sec 0 775 KBytes [ 5] 3.00-4.00 sec 2.50 MBytes 2.50 MBytes/sec 0 898 KBytes [ 5] 4.00-5.00 sec 2.50 MBytes 2.50 MBytes/sec 0 988 KBytes [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 MBytes/sec 6 2.83 KBytes [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 MBytes/sec 1 1.41 KBytes [ 5] 7.00-8.00 sec 0.00 Bytes 0.00 MBytes/sec 1 1.41 KBytes [ 5] 8.00-9.00 sec 0.00 Bytes 0.00 MBytes/sec 0 1.41 KBytes [ 5] 9.00-10.00 sec 0.00 Bytes 0.00 MBytes/sec 1 1.41 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 15.5 MBytes 1.55 MBytes/sec 9 sender [ 5] 0.00-10.06 sec 12.5 MBytes 1.24 MBytes/sec receiver
Sadly, this is repeatable: 5 transfers are completed just fine, then the transfer rate drops to zero and comes out at ~1.5 MB/s overall. Phew...
Let's try again over our VPN connection:
$ iperf3 -fM -c example.fastd [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 801 KBytes 0.78 MBytes/sec 0 59.3 KBytes [ 5] 1.00-2.00 sec 1.61 MBytes 1.61 MBytes/sec 0 121 KBytes [ 5] 2.00-3.00 sec 3.22 MBytes 3.22 MBytes/sec 0 248 KBytes [ 5] 3.00-4.00 sec 571 KBytes 0.56 MBytes/sec 12 193 KBytes [ 5] 4.00-5.00 sec 0.00 Bytes 0.00 MBytes/sec 15 193 KBytes [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 MBytes/sec 16 193 KBytes [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 MBytes/sec 15 193 KBytes [ 5] 7.00-8.00 sec 1.98 MBytes 1.98 MBytes/sec 8 214 KBytes [ 5] 8.00-9.00 sec 2.79 MBytes 2.79 MBytes/sec 0 245 KBytes [ 5] 9.00-10.00 sec 571 KBytes 0.56 MBytes/sec 12 174 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.5 MBytes 1.15 MBytes/sec 78 sender [ 5] 0.00-10.06 sec 10.0 MBytes 0.99 MBytes/sec receiver
Again, the transfer rate drops to zero after 5 transfers, then picks up speed again, but we are still slightly slower than without the VPN. Maybe it's time to employ some kind of traffic shaping again :-\