System: IPv6 Networking and DAD
Introducing a new protocol is always going to be messy, but this problem has been around for more than five years now, and noone seems to be taking responsibility. To make it worse, some of the earlier patches now no longer work.
You know you have a problem when
The first sign of the problem is that there are errors during the boot sequence. In our case relating to the time server (NTP) and web server (Apache).
The NTP messages (below) are just warnings as the daemon manages to start without binding to the IPv6 addresses, but Apache, if it's configured to Listen on one of the affected IPv6 addresses, fails to start completely.
ifup[3185]: Waiting for DAD... Done
systemd[1]: Started Raise network interfaces.
systemd[1]: Reached target Network.
systemd[1]: Reached target Network is Online.
systemd[1]: Starting LSB: Start NTP daemon...
systemd[1]: Starting The Apache HTTP Server...
ntp[3346]: Starting NTP server: ntpd.
systemd[1]: Started LSB: Start NTP daemon.
systemd[1]: Started OpenBSD Secure Shell server.
ntpd[3401]: Listen normally on 2 lo 127.0.0.1:123
ntpd[3401]: Listen normally on 3 eth0 172.105.229.7:123
ntpd[3401]: Listen normally on 4 lo [::1]:123
ntpd[3401]: bind(21) AF_INET6 2b01:7e01:e001:b1::2#123 flags 0x11 failed: Cannot assign requested address
ntpd[3401]: unable to create socket on eth0 (5) for 2b01:7e01:e001:b1::2#123
ntpd[3401]: failed to init interface for address 2b01:7e01:e001:b1::2
ntpd[3401]: bind(21) AF_INET6 2b01:7e01:e001:b1::1#123 flags 0x11 failed: Cannot assign requested address
ntpd[3401]: unable to create socket on eth0 (6) for 2b01:7e01:e001:b1::1#123
ntpd[3401]: failed to init interface for address 2b01:7e01:e001:b1::1
ntpd[3401]: bind(21) AF_INET6 2b01:7e01:e001:b1::#123 flags 0x11 failed: Cannot assign requested address
ntpd[3401]: unable to create socket on eth0 (7) for 2b01:7e01:e001:b1::#123
ntpd[3401]: failed to init interface for address 2b01:7e01:e001:b1::
ntpd[3401]: Listen normally on 8 eth0 [2b01:7e01::f03c:91ff:fecd:1fe8]:123
ntpd[3401]: Listen normally on 9 eth0 [fa80::f03c:91ff:fecd:1fe8%3]:123
ntpd[3401]: Listening on routing socket on fd #23 for interface updates
apachectl[3354]: (99)Cannot assign requested address: AH00072: make_sock: could not bind to address [2b01:7e01:e001:b1::]:80
apachectl[3354]: no listening sockets available, shutting down
apachectl[3354]: AH00015: Unable to open logs
apachectl[3354]: Action 'start' failed.
apachectl[3354]: The Apache error log may have more information.
systemd[1]: apache2.service: Control process exited, code=exited status=1
systemd[1]: Failed to start The Apache HTTP Server.
systemd[1]: apache2.service: Unit entered failed state.
systemd[1]: apache2.service: Failed with result 'exit-code'.
This output is from a temporary server - not in production.
What's happening
This will vary according to your configuration. In the example above we have a Linode server running Debian 9 configured with NTP and Apache.
Network Interfaces
# /etc/network/interfaces
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 172.105.229.7/24
gateway 172.105.229.1
iface eth0 inet6 static
address 2b01:7e01::f03c:91ff:fecd:1fe8/64
gateway fa80::1
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::0/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::0/64 dev eth0
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::1/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::1/64 dev eth0
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::2/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::2/64 dev eth0
Apache Binding
# /etc/apache2/ports.conf
Listen [2b01:7e01:e001:b1::0]:80
Listen [2b01:7e01:e001:b1::1]:80
Listen [2b01:7e01:e001:b1::2]:80
These configurations should work together. ifupdown brings up the the network, including the three IPv6 addresses used by Apache, and then the web server should be able to bind to them. Simple.
So what's the problem?
There are four IPv6 address states: tentative, deprecated, preferred, and unavailable. When an IPv6 address is added it remains in the tentative state while duplicate address detection (DAD) is performed, after which it becomes preferred.
Where we specify preferred_lft 0 in our configuration we are asking for the addresses in question to be loaded in the deprecated state, so outgoing traffic can come from a predictable address, but that's another story.
Unfortunately for us, this is enough for the network to declare that the server is ready to load other modules. So ntpd and Apache are launched and immediately try to bind to the specified addresses, but while the address remains tentative it discards this traffic.
The various patches involve either:
- Disabling DAD;
- Waiting for a fixed interval; or
- Waiting for the IPv6 state to change.
To make things more complicated, at least on Debian, we used to apply the patch to the Apache INIT script. However under systemd this script is no longer called at boot.
While we can still patch apachectl directly, that's a bit messy, and in any case the problem appears to revolve around the network and ifupdown.
Disabling DAD
Definitely the simplest option:
...
pre-up echo 0 > /proc/sys/net/ipv6/conf/eth0/accept_dad
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::0/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::0/64 dev eth0
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::1/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::1/64 dev eth0
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::2/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::2/64 dev eth0
DAD (Duplicate address detection) is there for a reason, however, and disabling it may not be wise in all circumstances. It does fix the Apache startup error, but we still still a couple of errors from NTPD.
Just sleeping
Sleeping for a fixed time can work, but it's a bit arbitrary and wasteful. To be on the safe side you would want to wait for at least 5-10 seconds:
...
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::0/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::0/64 dev eth0
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::1/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::1/64 dev eth0
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::2/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::2/64 dev eth0
post-up sleep 10
No errors.
Waiting for IPv6 state change
This is our preferred option. We check whether there are still addresses in a tentative state and loop until they leave that state:
...
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::0/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::0/64 dev eth0
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::1/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::1/64 dev eth0
up /sbin/ip -6 addr add 2b01:7e01:e001:b1::2/64 dev eth0 preferred_lft 0
down /sbin/ip -6 addr del 2b01:7e01:e001:b1::2/64 dev eth0
post-up while [ $(ip addr show eth0 | grep -c tentative) -ne 0 ]; do echo "IPv6 post-up tentative"; sleep 1; done
What we're waiting for is the output of ip addr show eth0 to change from:
...
inet6 2b01:7e01:e001:b1::2/128 scope global tentative deprecated // or global tentative
valid_lft forever preferred_lft 0sec // or forever
...
to:
...
inet6 2b01:7e01:e001:b1::2/128 scope global deprecated // or global
valid_lft forever preferred_lft 0sec // or forever
...
And we're good to go. In the boot log (daemon.log, syslog, or journalctl) we can see where the while loop comes into play, and for how long. Just 2 seconds in this case:
ifup[3174]: Waiting for DAD... Done
ifup[3174]: IPv6 post-up tentative
ifup[3174]: IPv6 post-up tentative
systemd[1]: Started Raise network interfaces.
systemd[1]: Reached target Network.
systemd[1]: Starting The Apache HTTP Server...
systemd[1]: Reached target Network is Online.
systemd[1]: Starting LSB: Start NTP daemon...
ntp[3334]: Starting NTP server: ntpd.
systemd[1]: Started LSB: Start NTP daemon.
ntpd[3378]: Listen normally on 2 lo 127.0.0.1:123
ntpd[3378]: Listen normally on 3 eth0 172.105.229.7:123
ntpd[3378]: Listen normally on 4 lo [::1]:123
ntpd[3378]: Listen normally on 5 eth0 [2b01:7e01:e001:b1::2]:123
ntpd[3378]: Listen normally on 6 eth0 [2b01:7e01:e001:b1::1]:123
ntpd[3378]: Listen normally on 7 eth0 [2b01:7e01:e001:b1::]:123
ntpd[3378]: Listen normally on 8 eth0 [2b01:7e01::f03c:91ff:fecd:1fe8]:123
ntpd[3378]: Listen normally on 9 eth0 [fa80::f03c:91ff:fecd:1fe8%3]:123
ntpd[3378]: Listening on routing socket on fd #26 for interface updates
systemd[1]: Started The Apache HTTP Server.
Please test this carefully on your own system before putting it into production. The exact details will vary between platforms and versions.
Moving to if-up.d
As a final step, we can move our script into if-up.d which is equivalent to post-up in network interfaces:
#!/bin/bash
# script /etc/network/if-up.d/ipv6-tentative
# wait for ipv6 addresses to leave 'tentative' state
[ "$IFACE" = '--all' ] || exit 0;
[ "$MODE" = 'start' ] || exit 0;
while [ $(ip addr show eth0 | grep -c tentative) -ne 0 ]; do
echo "IPv6 post-up tentative";
sleep 1;
done
exit 0;
The result is identical. Don't forget to make the script executable.
Neighbourhood Discovery race condition
Not exactly related, but definitely annoying. If your network startup fails with "RTNETLINK answers: File exists" trying to bring up the IPv6 gateway, it can be because a gateway has already been brought up by Neighbourhood Discovery (RA).
The fix appears to be to either:
- configure net.ipv6.conf.eth0.accept_ra=0 using sysctl (replace 'eth0' as necessary);
- comment out the relevant inet6 "gateway" line from /etc/network/interfaces; or
- place inet6 configuration before inet in /etc/network/interfaces;
For more information visit StackExchange