update docker version

[maxknee]

Updating latest docker in docker.
We are seeing an issue with the latest 6.18 linux kernel:

latest: Pulling from plugins/gcr
Digest: sha256:6d47e0f5c7f3e23018d9d9235129ed2df47f7ab67a2fa566aba33176fe52dd56
Status: Downloaded newer image for plugins/gcr:latest
+ /usr/local/bin/dockerd --data-root /var/lib/docker --host=unix:///var/run/docker.sock --dns 10.248.2.2
Unable to reach Docker Daemon after 15 attempts.

[Ompragash]

@maxknee Could you please share more details about the issue you encountered? Does upgrading the Docker dind image to this exact version 29.3.1 solve the issue, given that it's a major version bump?

[maxknee]

there's an issue with how the docker in docker behaves with IPTables in the latest 6.18 linux kernel because theres missing IPTable configs on the older version of docker.

[maxknee]

Diagnostic evidence
Collected on agent ip-10-248-2-61 (kernel 6.18.15-14.217.amzn2023.x86_64):

Host kernel config
$ grep -E 'IP_NF_NAT|IP_NF_FILTER|IP_NF_IPTABLES|NF_NAT_MASQUERADE' /boot/config-$(uname -r)
CONFIG_NF_NAT_MASQUERADE=y
CONFIG_IP_NF_IPTABLES=y

CONFIG_IP_NF_NAT and CONFIG_IP_NF_FILTER are completely absent

Host iptables works (nft backend)
$ iptables -L DOCKER-USER -n
Chain DOCKER-USER (1 references)
target prot opt source destination
ACCEPT all -- 172.18.255.253 169.254.169.254
ACCEPT all -- 172.18.255.252 169.254.169.254
REJECT all -- 0.0.0.0/0 169.254.169.254 reject-with icmp-port-unreachable
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Modules don't exist (built-in, not loadable)
$ find /lib/modules/$(uname -r) -name 'ip_tables' -o -name 'iptable_nat'

(no output)

$ modprobe ip_tables
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/6.18.15-14.217.amzn2023.x86_64

$ modprobe iptable_nat
modprobe: FATAL: Module iptable_nat not found in directory /lib/modules/6.18.15-14.217.amzn2023.x86_64
Inner dockerd fatal error (captured with --entrypoint override)
$ docker run --rm --privileged --entrypoint sh plugins/gcr -c '
/usr/local/bin/dockerd --data-root /var/lib/docker
--host=unix:///var/run/docker.sock --log-level=debug 2>&1 &
sleep 20; kill %1 2>/dev/null'
Key output:

iptables v1.8.7 (legacy)

time="..." level=warning msg="Running iptables --wait -t nat -L -n failed with message:
modprobe: can't change directory to '/lib/modules': No such file or directory iptables v1.8.7 (legacy): can't initialize iptables table nat': Table does not exist
(do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.`,
error: exit status 3"

failed to start daemon: Error initializing network controller: error obtaining controller
instance: failed to create NAT chain DOCKER: iptables failed:
iptables -t nat -N DOCKER: modprobe: can't change directory to '/lib/modules': No such
file or directory
iptables v1.8.7 (legacy): can't initialize iptables table `nat': Table does not exist
(do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)
Container Docker version
Docker Client 20.10.14 (March 2022) — uses legacy iptables exclusively
containerd v1.5.11