Add dual-stack IPv6 networking support

README.md

@@ -240,7 +240,8 @@ RunCVM is free and open-source, licensed under the Apache Licence, Version 2.0.
 - Command-line and image-embedded options for customising the a container's VM specifications, devices, kernel
 - Intelligent kernel selection, according to the distribution used in the image being launched
 - No external dependencies, except for Docker/Podman and relevant Linux kernel modules (`kvm` and `tun`)
-- Support multiple Docker network interfaces attached to a created (but not yet running) container using `docker run --network=<network>` and `docker network connect` (excluding IPv6)
+- Support multiple Docker network interfaces attached to a created (but not yet running) container using `docker run --network=<network>` and `docker network connect`
+- Dual-stack (IPv4 and IPv6) networking, where the host and Docker are already configured to support it
 
 ## Project ambitions
 
@@ -297,6 +298,8 @@ At time of writing:
 
 We recommend `all/rp_filter` be set to 2, as this is the simplest change and provides a good balance of security.
 
+This requirement is IPv4-specific: Linux has no IPv6 equivalent of `rp_filter`, so no further host configuration is needed to support the IPv4-proxied DNS/exec helper channel used by dual-stack Container/VMs.
+
 ## Installation
 
 Run:
@@ -373,7 +376,9 @@ In the below summary of RunCVM's current main features and limitations, [+] is u
       - [+] `--hostname` (or `-h`) is supported
       - [-] `docker network connect` on a running container is not supported
       - [-] `--network=host` and `--network=container:name|id` are not supported
-      - [-] IPv6 is not supported
+      - [+] Dual-stack (IPv4 and IPv6) networking is supported, automatically, where the host and Docker are already configured for IPv6 (see `--env=RUNCVM_IPV6=<0|1>`)
+      - [-] IPv6-only networks (i.e. with no IPv4 connectivity at all) are not supported: a working IPv4 leg is required
+      - [-] `--dns=<IPv6 address>` is not supported: outgoing DNS requests are proxied via `dnsmasq` running in the container, which is only ever given an IPv4 address
    - Execution environment
       - [+] `--user` (or `-u`) is supported
       - [?] `--workdir` (or `-w`) is supported
@@ -594,6 +599,10 @@ By default SeaBIOS is used to boot the VM. Enable OVMF EFI boot with this option
 
 Enable use of [virtio vhost-net](https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net) (reliant on host `vhost_net` module and `/dev/vhost-net` device) to accelerate networking.
 
+### `--env=RUNCVM_IPV6=<0|1>`
+
+By default, IPv6 is enabled in the VM automatically if a global IPv6 address was assigned to the container (i.e. the Docker network is dual-stack), and disabled otherwise. Set to `1` to force IPv6 on regardless, or `0` to force it off (e.g. to keep a container IPv4-only despite being attached to a dual-stack network).
+
 ### `--env=RUNCVM_SYS_ADMIN=1`
 
 By default, `virtiofsd` is not launched with `-o modcaps=+sys_admin` (and containers are not granted `CAP_SYS_ADMIN`). Use this option if you need to change this.
@@ -731,7 +740,7 @@ In more detail, the RunCVM runtime `create` process:
 
 The `runcvm-ctr-entrypoint`:
 - Is always launched as PID1 within the standard Docker container.
-- Saves the container's originally-intended entrypoint and command line, environment variables and network configuration to files inside `/.runcvm`.
+- Saves the container's originally-intended entrypoint and command line, environment variables and network configuration (IPv4, and IPv6 if a global IPv6 address is present) to files inside `/.runcvm`.
 - Creates a bridge (acting as a hub) for each container network interface, to join that interface to a VM tap network interface.
 - Launches `virtiofsd` to serve the container's root filesystem.
 - Configures `/etc/resolv.conf` in the container.
@@ -745,12 +754,12 @@ The `runcvm-init` process:
 
 The `runcvm-ctr-qemu` script:
 - Prepares disk backing files as specified by `--env=RUNCVM_DISKS=<disks>`
-- Prepares network configuration as saved from the container (modifying the MAC address of each container interface)
+- Prepares network configuration as saved from the container (modifying the MAC address of each container interface), enabling IPv6 support in the VM's kernel command line if a global IPv6 address was saved for any interface (or if forced with `--env=RUNCVM_IPV6=1`)
 - Launches [QEMU](https://www.qemu.org/) with the required kernel, network interfaces, disks, display, and with a root filesystem mounted via virtiofs from the container and with `runcvm-vm-init` as the VM's init process.
 
 The `runcvm-vm-init` process:
 - Runs as PID1 within the VM.
-- Retrieves the container configuration - network, environment, disk and tmpfs mounts - saved by `runcvm-ctr-entrypoint` to `/.runcvm`, and reproduces it within the VM
+- Retrieves the container configuration - network (IPv4, and IPv6 if present), environment, disk and tmpfs mounts - saved by `runcvm-ctr-entrypoint` to `/.runcvm`, and reproduces it within the VM
 - Launches the container's pre-existing entrypoint, in one of two ways.
    1. If `RUNCVM_INIT` is `1` (i.e. the container was originally intended to be launched with Docker's own init process) then it configures and execs busybox `init`, which becomes the VM's PID1, to supervise `dropbear`, run `runcvm-vm-start` and `poweroff` the VM if signalled to do so.
    2. Else, it backgrounds `dropbear`, then execs (via `runcvm-init`, purely to create a controlling tty) `runcvm-vm-start`, which runs as the VM's PID1.

runcvm-scripts/runcvm-ctr-defaults

@@ -17,6 +17,7 @@ clean_env() {
   RUNCVM_BIOS_DEBUG RUNCVM_BIOS \
   RUNCVM_DISPLAY_MODE \
   RUNCVM_QEMU_DEBUG RUNCVM_QEMU_ARCH RUNCVM_QEMU_DISPLAY RUNCVM_QEMU_VGA RUNCVM_QEMU_VNC_DISPLAY RUNCVM_QEMU_USB RUNCVM_QEMU_NET_VHOST RUNCVM_QEMU_MEM_PREALLOC \
+  RUNCVM_IPV6 \
   RUNCVM_KERNEL RUNCVM_KERNEL_ROOT RUNCVM_KERNEL_APPEND RUNCVM_KERNEL_INITRAMFS_PATH RUNCVM_KERNEL_DEBUG RUNCVM_KERNEL_PATH \
   RUNCVM_DISKS \
   RUNCVM_UIDGID RUNCVM_VM_MOUNTPOINT RUNCVM_TMPFS \
@@ -36,6 +37,22 @@ load_network() {
   return 0
 }
 
+# Loads IPv6 addressing for interface $1 (or the default-route interface, if
+# omitted), if any was captured. Returns 1 (leaving DOCKER_IF6_* unset) for an
+# interface with no IPv6 address, or when the container has no IPv6 at all,
+# i.e. /.runcvm/network/devices6 does not exist.
+load_network6() {
+  local if="${1:-default}"
+  # Always reset first: unlike load_network(), this is routinely called for
+  # interfaces that turn out to have no IPv6, and callers rely on
+  # DOCKER_IF6_* being genuinely empty (not left over from a previous
+  # interface) when this returns 1.
+  DOCKER_IF6= DOCKER_IF6_IP= DOCKER_IF6_IP_NETPREFIX= DOCKER_IF6_IP_GW=
+  [ -d /.runcvm/network/devices6 ] && [ -s /.runcvm/network/devices6/$if ] || return 1
+  read -r DOCKER_IF6 DOCKER_IF6_IP DOCKER_IF6_IP_NETPREFIX DOCKER_IF6_IP_GW </.runcvm/network/devices6/$if
+  return 0
+}
+
 which() {
   local cmd="$1"
   local WHICH_PATH="${RUNCVM_PATH//:/ }" # Replace ':' with ' '

runcvm-scripts/runcvm-ctr-entrypoint

@@ -86,6 +86,11 @@ read -r DOCKER_GW_IF DOCKER_GW_IF_IP < \
   <(ip -json route show | jq -r '.[] | (select(.dst == "default") | [.dev, .gateway]) | @tsv')
 # e.g. eth0 172.25.10.1
 
+# Identify IPv6 default gateway device and IP address, if any.
+read -r DOCKER_GW_IF6 DOCKER_GW_IF6_IP < \
+  <(ip -6 -json route show default | jq -r '.[0] | [.dev, .gateway] | @tsv')
+# e.g. eth0 fd00:1::1
+
 QEMU_BRIDGE_IP=169.254.1.1
 RUNCVM_DNS_IP=169.254.169.254
 
@@ -94,15 +99,36 @@ mkdir -p /.runcvm/network/devices
 # Save non-link-scope non-default routes for later restoration in the running VM.
 ip -json route show | jq -r '.[] | select(.scope != "link" and .dst != "default") | "\(.dst) \(.gateway) \(.dev) \(.prefsrc)"' >/.runcvm/network/routes
 
+# Save non-kernel (i.e. non-connected, non-link-local) non-default IPv6 routes
+# for later restoration in the running VM. Kernel-derived on-link prefix routes
+# are handled per-interface (as DOCKER_NET6, below); fe80::/64 is link-local and
+# is recreated automatically by the kernel when the VM brings up its interface.
+ip -6 -json route show | jq -r '.[] | select(.protocol != "kernel" and .dst != "default" and .dst != "fe80::/64") | "\(.dst) \(.gateway) \(.dev)"' >/.runcvm/network/routes6
+
 for if in $(ip -json link show | jq -r '.[] | .ifname')
 do
 
   [ "$if" = "lo" ] && continue
 
   read -r DOCKER_IF_IP DOCKER_IF_IP_NETPREFIX DOCKER_IF_MAC DOCKER_IF_MTU < \
-    <(ip -json addr show "$if" | jq -r '.[0] | [.addr_info[0].local, .addr_info[0].prefixlen, .address, .mtu] | @tsv')
+    <(ip -json addr show "$if" | jq -r '.[0] as $l | first($l.addr_info[] | select(.family=="inet")) as $a | [$a.local, $a.prefixlen, $l.address, $l.mtu] | @tsv')
   # e.g. 172.25.10.2 24 52:54:00:b7:0b:b6 1500
 
+  # Capture the on-link (connected) route prefix for this interface, before its
+  # address (and the kernel route derived from it) is flushed below.
+  DOCKER_NET=$(ip -json route show dev "$if" scope link | jq -r '.[0].dst')
+
+  # Capture a global IPv6 address for this interface, if any. Absence of a
+  # devices6/<if> file for an interface is what signals "no IPv6 on this
+  # interface" downstream. Unlike IPv4, no container-side IPv6 address or
+  # route is restored below: the VM is the sole owner of this address.
+  # (Under `set -e`, this read legitimately fails, with nothing to abort
+  # for, whenever this interface has no global IPv6 address - the common
+  # case for an IPv4-only container - since the underlying jq select()
+  # then produces no output at all for read to consume.)
+  read -r DOCKER_IF6_IP DOCKER_IF6_IP_NETPREFIX < \
+    <(ip -json addr show "$if" | jq -r '.[0].addr_info[] | select(.family=="inet6" and .scope=="global" and ((.deprecated // false) | not)) | [.local, .prefixlen] | @tsv' | head -n 1) || true
+
   # Save container network parameters
   if [ "$if" = "$DOCKER_GW_IF" ]; then
     printf "%s %s %s %s %s %s\n" \
@@ -115,6 +141,20 @@ do
       >/.runcvm/network/devices/$if
   fi
 
+  if [ -n "$DOCKER_IF6_IP" ]; then
+    mkdir -p /.runcvm/network/devices6
+    if [ "$if" = "$DOCKER_GW_IF6" ]; then
+      printf "%s %s %s %s\n" \
+        "$if" "$DOCKER_IF6_IP" "$DOCKER_IF6_IP_NETPREFIX" "$DOCKER_GW_IF6_IP" \
+        >/.runcvm/network/devices6/$if
+      ln -s "$if" /.runcvm/network/devices6/default
+    else
+      printf "%s %s %s %s\n" \
+        "$if" "$DOCKER_IF6_IP" "$DOCKER_IF6_IP_NETPREFIX" "-" \
+        >/.runcvm/network/devices6/$if
+    fi
+  fi
+
   # RECONFIGURE CONTAINER NETWORK
   ip addr flush dev "$if"
 
@@ -132,7 +172,6 @@ do
   ip link set dev "$QEMU_BRIDGE" up
 
   # Restore network route via this bridge
-  DOCKER_NET=$(ip_prefix_to_network "$DOCKER_IF_IP" "$DOCKER_IF_IP_NETPREFIX")/"$DOCKER_IF_IP_NETPREFIX"
   ip route add "$DOCKER_NET" dev "$QEMU_BRIDGE"
 
   # If this interface is the default gateway interface, perform additional special steps.

runcvm-scripts/runcvm-ctr-qemu

@@ -125,6 +125,8 @@ do_networks() {
   local id=0 ifpath if mac vhost
   local DOCKER_IF DOCKER_IF_MAC DOCKER_IF_MTU DOCKER_IF_IP DOCKER_IF_IP_NETPREFIX DOCKER_IF_IP_GW
 
+  HAS_IPV6=0
+
   for ifpath in /.runcvm/network/devices/*
   do
     if=$(busybox basename "$ifpath")
@@ -133,6 +135,12 @@ do_networks() {
 
     load_network "$if"
 
+    # Set (not local): read by the caller to decide whether to enable IPv6
+    # on the VM's kernel command line.
+    if load_network6 "$if"; then
+      HAS_IPV6=1
+    fi
+
     mac=$(busybox sed -r 's/^..:..:../52:54:00/' <<<$DOCKER_IF_MAC)
 
     if [ "$RUNCVM_QEMU_NET_VHOST" = "1" ]; then
@@ -329,8 +337,17 @@ VIRTIOFS+=(
 # Experimental: Enable for a SCSI bus
 # OPTS+=(-device virtio-scsi-pci,id=scsi0,disable-modern=true)
 
-# Disable IPv6, which is currently unsupported, at kernel boot time
-APPEND+=(ipv6.disable=1 panic=-1)
+# Enable IPv6 in the VM only if a global IPv6 address was captured for at
+# least one interface (set by do_networks(), above), unless overridden with
+# --env=RUNCVM_IPV6=<0|1>. Otherwise, disable IPv6 entirely at kernel boot
+# time, as before.
+case "$RUNCVM_IPV6" in
+  1) HAS_IPV6=1 ;;
+  0) HAS_IPV6=0 ;;
+esac
+
+[ "$HAS_IPV6" = "1" ] || APPEND+=(ipv6.disable=1)
+APPEND+=(panic=-1)
 
 # Disable unneeded functionality
 APPEND+=(scsi_mod.scan=none tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k cryptomgr.notests pci=lastbus=0 selinux=0)

runcvm-scripts/runcvm-ip-functions

@@ -1,44 +1,5 @@
 #!/bin/bash
 
-cidr_to_int() {
-  echo "$(( 0xffffffff ^ ((1 << (32 - $1)) - 1) ))"
-}
-
-int_to_ip() {
-  local value="$1"
-  echo "$(( ($1 >> 24) & 0xff )).$(( ($1 >> 16) & 0xff )).$(( ($1 >> 8) & 0xff )).$(( $1 & 0xff ))"
-}
-
-cidr_to_netmask() {
-  local value=$(cidr_to_int "$1")
-  int_to_ip "$value"
-}
-
-ip_prefix_to_network() {
-  local IFS i1 i2 i3 i4 m1 m2 m3 m4
-  IFS=. read -r i1 i2 i3 i4 <<< "$1"
-
-  local mask=$(cidr_to_netmask "$2")
-  IFS=. read -r m1 m2 m3 m4 <<< "$mask"
-
-  printf "%d.%d.%d.%d\n" "$((i1 & m1))" "$((i2 & m2))" "$((i3 & m3))" "$((i4 & m4))"
-}
-
-cidr_to_bcastmask() {
-  local value=$(( (1 << 32) - $(cidr_to_int "$1") - 1 ))
-  int_to_ip "$value"
-}
-
-ip_prefix_to_bcast() {
-  local IFS i1 i2 i3 i4 m1 m2 m3 m4
-  IFS=. read -r i1 i2 i3 i4 <<< "$1"
-
-  local mask=$(cidr_to_bcastmask "$2")
-  IFS=. read -r m1 m2 m3 m4 <<< "$mask"
-
-  printf "%d.%d.%d.%d\n" "$((i1 | m1))" "$((i2 | m2))" "$((i3 | m3))" "$((i4 | m4))"
-}
-
 is_natural_int() {
     case $1 in
         ''|*[!0-9]*) return 1 ;;  # not numeric

runcvm-scripts/runcvm-vm-init

@@ -42,14 +42,16 @@ if ! [ -h /dev/fd ]; then
   ln -s /proc/self/fd /dev/fd
 fi
 
-# FIXME: This must be run early enough, otherwise other interfaces like docker0 might have started
-IF=$(ls /sys/class/net/ | grep -vE '^(lo|docker)' | head -n 1)
-
+# Disable IPv6 on every interface by default (both those that already exist,
+# and any that may appear later), before any of them are renamed or brought
+# up, so that none of them briefly performs SLAAC/DAD or joins a
+# solicited-node multicast group using its unwanted kernel-assigned EUI-64
+# address. Interfaces with a captured IPv6 address are selectively
+# re-enabled, by their real (post-rename) name, below.
 # https://bugzilla.redhat.com/show_bug.cgi?id=501934
-for i in all $IF
+for i in default $(ls /sys/class/net/ | grep -v '^lo$')
 do
-  # /sbin/sysctl -q -w -e net.ipv6.conf.$i.disable_ipv6=1 net.ipv6.conf.$i.autoconf=0 net.ipv6.conf.$i.accept_ra=0
-  sysctl -q -w -e net.ipv6.conf.$i.disable_ipv6=1 net.ipv6.conf.$i.autoconf=0 || true
+  sysctl -q -w -e net.ipv6.conf.$i.disable_ipv6=1 net.ipv6.conf.$i.autoconf=0 net.ipv6.conf.$i.accept_ra=0 || true
 done
 
 # Bring up local interface
@@ -82,11 +84,33 @@ do
   load_network "$if"
 
   ip link set $DOCKER_IF-tmp name $DOCKER_IF
+
+  if load_network6 "$if"; then
+    # Re-enable IPv6 on this interface only. accept_ra/autoconf are left off
+    # (as set for every interface above): Docker addresses IPv6 statically,
+    # like IPv4, and runs no RA daemon on its bridge networks, so the VM
+    # should behave deterministically rather than react to anything it can
+    # now see on the raw L2 segment.
+    sysctl -q -w -e net.ipv6.conf.$DOCKER_IF.disable_ipv6=0 net.ipv6.conf.$DOCKER_IF.autoconf=0 net.ipv6.conf.$DOCKER_IF.accept_ra=0 || true
+  fi
+
   ip addr add $DOCKER_IF_IP/$DOCKER_IF_IP_NETPREFIX broadcast + dev $DOCKER_IF
   ip link set $DOCKER_IF up mtu "${DOCKER_IF_MTU:=1500}"
 
   # If this is the default gateway interface, establish the default gateway
-  [ -n "$DOCKER_IF_IP_GW" ] && ip route add default via $DOCKER_IF_IP_GW
+  [ -n "$DOCKER_IF_IP_GW" ] && [ "$DOCKER_IF_IP_GW" != "-" ] && ip route add default via $DOCKER_IF_IP_GW
+
+  if [ -n "$DOCKER_IF6_IP" ]; then
+    # The address is definitionally ours: Docker allocated it, and the
+    # container released it (at entrypoint time, before enslaving this
+    # interface) well before this point, so nothing else can answer for it.
+    # Skip DAD so the address is usable immediately: with DAD enabled the
+    # address is 'tentative' for ~1s, and the default route below would
+    # otherwise intermittently fail ('Network is unreachable') against a
+    # tentative source address.
+    ip -6 addr add $DOCKER_IF6_IP/$DOCKER_IF6_IP_NETPREFIX dev $DOCKER_IF nodad
+    [ -n "$DOCKER_IF6_IP_GW" ] && [ "$DOCKER_IF6_IP_GW" != "-" ] && ip -6 route add default via $DOCKER_IF6_IP_GW dev $DOCKER_IF
+  fi
 done
 
 # Read and install any supplementary routes.
@@ -96,6 +120,13 @@ do
     ip route add "$DOCKER_RT_NET" via "$DOCKER_RT_GW" dev "$DOCKER_RT_DEV" || true
 done </.runcvm/network/routes
 
+# Read and install any supplementary IPv6 routes.
+while read -r DOCKER_RT6_NET DOCKER_RT6_GW DOCKER_RT6_DEV
+do
+  [ -n "$DOCKER_RT6_NET" ] && [ -n "$DOCKER_RT6_GW" ] && [ -n "$DOCKER_RT6_DEV" ] && \
+    ip -6 route add "$DOCKER_RT6_NET" via "$DOCKER_RT6_GW" dev "$DOCKER_RT6_DEV" || true
+done </.runcvm/network/routes6
+
 # TODO
 # - bind-mount or overwrite /etc/resolv.conf, /etc/hosts and /etc/hostname?