If you use FirewallD to manage your firewall rules, and you include a `forward-port` tag in your zone file that is missing the `to-port` or `to-addr` parameters, Docker will refuse to start.
For example:
/etc/firewalld/zones/public.xml
<?xml version="1.0" encoding="utf-8"?>
<zone target="DROP">
<description>public zone.</description>
<service name="ssh" />
<forward-port port="51820" protocol="udp" />
</zone>
Will cause this output when you try to start Docker:
From `journalctl -l -u docker`
Dec 27 21:43:38 test systemd[1]: Starting Docker Application Container Engine...
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.939029977Z" level=info msg="Starting up"
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.941435919Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.943581501Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.943633881Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.943665051Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.943696801Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.945747572Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.945763802Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.945781072Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.945794012Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.961351463Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.964738675Z" level=warning msg="Your kernel does not support swap memory limit"
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.964752445Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.964758455Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.964763475Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.964896935Z" level=info msg="Loading containers: start."
Dec 27 21:43:38 test dockerd[7539]: time="2020-12-27T21:43:38.971881740Z" level=info msg="Firewalld: docker zone already exists, returning"
Dec 27 21:43:39 test dockerd[7539]: time="2020-12-27T21:43:39.111947329Z" level=warning msg="could not create bridge network for id acf7f168cd7aca8a45945338e5b60aa281b6c9f175c38adcd0bf854b25f4e01c bridge name docker0 while booting up from persistent state: Failed to program NAT chain: COMMAND_FAILED: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore v1.8.4 (legacy): goto 'PRE_docker' is not a chain\n\nError occurred at line: 2\nTry `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.\n"
Dec 27 21:43:39 test dockerd[7539]: time="2020-12-27T21:43:39.118342533Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Dec 27 21:43:39 test dockerd[7539]: time="2020-12-27T21:43:39.144437152Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Dec 27 21:43:39 test dockerd[7539]: failed to start daemon: Error initializing network controller: Error creating default "bridge" network: Failed to program NAT chain: COMMAND_FAILED: '/usr/sbin/ip6tables-restore -w -n' failed: ip6tables-restore v1.8.4 (legacy): goto 'PRE_docker' is not a chain
Dec 27 21:43:39 test dockerd[7539]: Error occurred at line: 2
Dec 27 21:43:39 test dockerd[7539]: Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
Dec 27 21:43:39 test systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Dec 27 21:43:39 test systemd[1]: docker.service: Failed with result 'exit-code'.
Dec 27 21:43:39 test systemd[1]: Failed to start Docker Application Container Engine.
Dec 27 21:43:41 test systemd[1]: docker.service: Scheduled restart job, restart counter is at 1.
Dec 27 21:43:41 test systemd[1]: Stopped Docker Application Container Engine.
If you set your forward-port like:
/etc/firewalld/zones/public.xml
<?xml version="1.0" encoding="utf-8"?>
<zone target="DROP">
<description>public zone.</description>
<service name="ssh" />
<forward-port port="51820" protocol="udp" to-port="51820" />
</zone>
Docker will start normally.
I ran into this because FirewallD's site says (as of writing this) that `to-port` and `to-addr` are optional.
I also would have figured this out more quickly if I had thought to look at the FirewallD logs sooner.
Anyway, here's hoping someone, someday, has some time saved by this post.