skip to content

System: IPv6 Networking and DAD

 Tweet Share0 Tweets

Introducing a new protocol is always going to be messy, but this problem has been around for more than five years now, and noone seems to be taking responsibility. To make it worse, some of the earlier patches now no longer work.

You know you have a problem when

The first sign of the problem is that there are errors during the boot sequence. In our case relating to the time server (NTP) and web server (Apache).

The NTP messages (below) are just warnings as the daemon manages to start without binding to the IPv6 addresses, but Apache, if it's configured to Listen on one of the affected IPv6 addresses, fails to start completely.

ifup[3185]: Waiting for DAD... Done systemd[1]: Started Raise network interfaces. systemd[1]: Reached target Network. systemd[1]: Reached target Network is Online. systemd[1]: Starting LSB: Start NTP daemon... systemd[1]: Starting The Apache HTTP Server... ntp[3346]: Starting NTP server: ntpd. systemd[1]: Started LSB: Start NTP daemon. systemd[1]: Started OpenBSD Secure Shell server. ntpd[3401]: Listen normally on 2 lo 127.0.0.1:123 ntpd[3401]: Listen normally on 3 eth0 172.105.229.7:123 ntpd[3401]: Listen normally on 4 lo [::1]:123 ntpd[3401]: bind(21) AF_INET6 2b01:7e01:e001:b1::2#123 flags 0x11 failed: Cannot assign requested address ntpd[3401]: unable to create socket on eth0 (5) for 2b01:7e01:e001:b1::2#123 ntpd[3401]: failed to init interface for address 2b01:7e01:e001:b1::2 ntpd[3401]: bind(21) AF_INET6 2b01:7e01:e001:b1::1#123 flags 0x11 failed: Cannot assign requested address ntpd[3401]: unable to create socket on eth0 (6) for 2b01:7e01:e001:b1::1#123 ntpd[3401]: failed to init interface for address 2b01:7e01:e001:b1::1 ntpd[3401]: bind(21) AF_INET6 2b01:7e01:e001:b1::#123 flags 0x11 failed: Cannot assign requested address ntpd[3401]: unable to create socket on eth0 (7) for 2b01:7e01:e001:b1::#123 ntpd[3401]: failed to init interface for address 2b01:7e01:e001:b1:: ntpd[3401]: Listen normally on 8 eth0 [2b01:7e01::f03c:91ff:fecd:1fe8]:123 ntpd[3401]: Listen normally on 9 eth0 [fa80::f03c:91ff:fecd:1fe8%3]:123 ntpd[3401]: Listening on routing socket on fd #23 for interface updates apachectl[3354]: (99)Cannot assign requested address: AH00072: make_sock: could not bind to address [2b01:7e01:e001:b1::]:80 apachectl[3354]: no listening sockets available, shutting down apachectl[3354]: AH00015: Unable to open logs apachectl[3354]: Action 'start' failed. apachectl[3354]: The Apache error log may have more information. systemd[1]: apache2.service: Control process exited, code=exited status=1 systemd[1]: Failed to start The Apache HTTP Server. systemd[1]: apache2.service: Unit entered failed state. systemd[1]: apache2.service: Failed with result 'exit-code'.

This output is from a temporary server - not in production.

What's happening

This will vary according to your configuration. In the example above we have a Linode server running Debian 9 configured with NTP and Apache.

Network Interfaces

# /etc/network/interfaces auto lo iface lo inet loopback auto eth0 iface eth0 inet static address 172.105.229.7/24 gateway 172.105.229.1 iface eth0 inet6 static address 2b01:7e01::f03c:91ff:fecd:1fe8/64 gateway fa80::1 up /sbin/ip -6 addr add 2b01:7e01:e001:b1::0/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::0/64 dev eth0 up /sbin/ip -6 addr add 2b01:7e01:e001:b1::1/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::1/64 dev eth0 up /sbin/ip -6 addr add 2b01:7e01:e001:b1::2/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::2/64 dev eth0

Apache Binding

# /etc/apache2/ports.conf Listen [2b01:7e01:e001:b1::0]:80 Listen [2b01:7e01:e001:b1::1]:80 Listen [2b01:7e01:e001:b1::2]:80

These configurations should work together. ifupdown brings up the the network, including the three IPv6 addresses used by Apache, and then the web server should be able to bind to them. Simple.

But there's a problem

There are four IPv6 address states: tentative, deprecated, preferred, and unavailable. When an IPv6 address is added it remains in the tentative state while duplicate address detection (DAD) is performed, after which it becomes preferred.

Where we specify preferred_lft 0 in our configuration we are asking for the addresses in question to be loaded in the deprecated state, so outgoing traffic can come from a predictable address, but that's another story.

Unfortunately for us, this is enough for the network to declare that the server is ready to load other modules. So NTPD and Apache are launched and immediately try to bind to the specified addresses, but while the address remains tentative it discards this traffic.

The various patches involve either:

  1. Disabling DAD;
  2. Waiting for a fixed interval; or
  3. Waiting for the IPv6 state to change.

To make things more complicated, at least on Debian, we used to apply the patch to the Apache INIT script. However under systemd this script is no longer called at boot.

While we can still patch apachectl directly, that's a bit messy, and in any case the problem appears to revolve around the network and ifupdown.

Disabling DAD

Definitely the simplest option:

... pre-up echo 0 > /proc/sys/net/ipv6/conf/eth0/accept_dad up /sbin/ip -6 addr add 2b01:7e01:e001:b1::0/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::0/64 dev eth0 up /sbin/ip -6 addr add 2b01:7e01:e001:b1::1/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::1/64 dev eth0 up /sbin/ip -6 addr add 2b01:7e01:e001:b1::2/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::2/64 dev eth0

DAD (Duplicate address detection) is there for a reason, however, and disabling it may not be wise in all circumstances. It does fix the Apache startup error, but we still still a couple of errors from NTPD.

Just sleeping

Sleeping for a fixed time can work, but it's a bit arbitrary and wasteful. To be on the safe side you would want to wait for at least 5-10 seconds:

... up /sbin/ip -6 addr add 2b01:7e01:e001:b1::0/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::0/64 dev eth0 up /sbin/ip -6 addr add 2b01:7e01:e001:b1::1/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::1/64 dev eth0 up /sbin/ip -6 addr add 2b01:7e01:e001:b1::2/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::2/64 dev eth0 post-up sleep 10

No errors.

Waiting for IPv6 state change

This is our preferred option. We check whether there are still addresses in a tentative state and loop until they leave that state:

... up /sbin/ip -6 addr add 2b01:7e01:e001:b1::0/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::0/64 dev eth0 up /sbin/ip -6 addr add 2b01:7e01:e001:b1::1/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::1/64 dev eth0 up /sbin/ip -6 addr add 2b01:7e01:e001:b1::2/64 dev eth0 preferred_lft 0 down /sbin/ip -6 addr del 2b01:7e01:e001:b1::2/64 dev eth0 post-up while [ $(ip addr show eth0 | grep -c tentative) -ne 0 ]; do echo "IPv6 post-up tentative"; sleep 1; done

What we're waiting for is the output of ip addr show eth0 to change from:

... inet6 2b01:7e01:e001:b1::2/128 scope global tentative deprecated // or global tentative valid_lft forever preferred_lft 0sec // or forever ...

to:

... inet6 2b01:7e01:e001:b1::2/128 scope global deprecated // or global valid_lft forever preferred_lft 0sec // or forever ...

And we're good to go. In the boot log (daemon.log, syslog, or journalctl) we can see where the while loop comes into play, and for how long. Just 2 seconds in this case:

ifup[3174]: Waiting for DAD... Done ifup[3174]: IPv6 post-up tentative ifup[3174]: IPv6 post-up tentative systemd[1]: Started Raise network interfaces. systemd[1]: Reached target Network. systemd[1]: Starting The Apache HTTP Server... systemd[1]: Reached target Network is Online. systemd[1]: Starting LSB: Start NTP daemon... ntp[3334]: Starting NTP server: ntpd. systemd[1]: Started LSB: Start NTP daemon. ntpd[3378]: Listen normally on 2 lo 127.0.0.1:123 ntpd[3378]: Listen normally on 3 eth0 172.105.229.7:123 ntpd[3378]: Listen normally on 4 lo [::1]:123 ntpd[3378]: Listen normally on 5 eth0 [2b01:7e01:e001:b1::2]:123 ntpd[3378]: Listen normally on 6 eth0 [2b01:7e01:e001:b1::1]:123 ntpd[3378]: Listen normally on 7 eth0 [2b01:7e01:e001:b1::]:123 ntpd[3378]: Listen normally on 8 eth0 [2b01:7e01::f03c:91ff:fecd:1fe8]:123 ntpd[3378]: Listen normally on 9 eth0 [fa80::f03c:91ff:fecd:1fe8%3]:123 ntpd[3378]: Listening on routing socket on fd #26 for interface updates systemd[1]: Started The Apache HTTP Server.

Please test this carefully on your own system before putting it into production. The exact details will vary between platforms and versions.

Moving to if-up.d

As a final step, we can move our script into if-up.d which is equivalent to post-up in network interfaces:

#!/bin/bash # script /etc/network/if-up.d/ipv6-tentative # wait for ipv6 addresses to leave 'tentative' state [ "$IFACE" = '--all' ] || exit 0; [ "$MODE" = 'start' ] || exit 0; while [ $(ip addr show eth0 | grep -c tentative) -ne 0 ]; do echo "IPv6 post-up tentative"; sleep 1; done exit 0;

The result is identical. Don't forget to make the script executable.

Neighbourhood Discovery race condition

Not exactly related, but definitely annoying. If your network startup fails with "RTNETLINK answers: File exists" trying to bring up the IPv6 gateway, it can be because a gateway has already been brought up by Neighbourhood Discovery (RA).

The fix appears to be to either:

  1. configure net.ipv6.conf.eth0.accept_ra=0 using sysctl (replace 'eth0' as necessary);
  2. comment out the relevant inet6 "gateway" line from /etc/network/interfaces; or
  3. place inet6 configuration before inet in /etc/network/interfaces;

For more information visit StackExchange

References

< System

Send a message to The Art of Web:


used only for us to reply, and to display your gravatar.

<- copy the digits from the image into this box

press <Esc> or click outside this box to close

Post your comment or question
top