Troubleshooting
Why Docker published ports fail after an edge PC reboot
Short answer
A container that reports running after a reboot has proved nothing about its published port. Treat unreachable ports as a boot-sequencing fault: probe each port from a client, follow the packets with iptables counters, then start the stack in dependency order instead of tuning the applications. Wait delays are installation-specific; readiness is proven per port.
On this page
- Why the usual fixes miss
- The path a published port depends on
- Probe every published port separately
- Follow the packets with the rule counters
- Start the stack in dependency order
- When the bridge, the modem or the firmware is at fault
- Worked example: three published ports, one answering, after a reboot
- Prove the repair survives the next cycle
- Questions asked
After a full power cycle, Node-RED, InfluxDB and Grafana can all report running while one or two of their published ports stay dead. That combination is not an application fault. The containers, the host interfaces, the bridge and Docker's forwarding rules do not become ready at the same instant, and a restart policy brings back processes, not verified network paths. Probe every published port separately, follow the packets with the rule counters, then start the stack in dependency order. Tuning the applications before packets provably reach them fixes nothing.
Why the usual fixes miss #
--restart=unless-stopped restarts a container's process. It proves nothing about the published host port, the forwarding rules, the bridge path or the peer service behind it, which is why all three containers can show green while two ports stay unreachable. If a container is crash-looping rather than running, that is a different fault, covered in auto-restarting Node-RED in Docker after an exit-code crash.
Moving the containers from the default bridge to a user-defined bridge was tested on an affected stack and did not restore access. A user-defined bridge is the right tool for name resolution between containers, not for a host forwarding rule built at the wrong moment during boot.
Reserving container addresses by MAC address targets the wrong layer unless probing proves that address reassignment is the failure. Docker rebuilds the published-port mapping for the container's current address at each start, so a pinned address can mask a rule-lifecycle fault while the boot race stays in place.
Retuning Node-RED, InfluxDB or Grafana cannot repair traffic that never arrives at the container. Look at the path first; application changes begin only after packets reach the correct listener.
The path a published port depends on #
A client packet must clear six stages, and the response must clear them in reverse. One break anywhere leaves the container running and the port dead.
| Stage | What carries it | What a break looks like |
|---|---|---|
| Client to host port | The edge PC's client-facing interface and the published host port | Probe returns filtered, no usable answer |
| Forwarding decision | Docker-managed rules in the host firewall | Rule counters do not move while you probe |
| Translation target | The container's current address | Rule entry aims at an address that no longer exists |
| Bridge | Default bridge, user-defined bridge, or a configured br0 | Listener works internally, traffic never crosses |
| Listener | The process inside the container | Packets arrive, nothing accepts the connection |
| Startup order | Interfaces and peer services ready before consumers | Port answers only after a delayed start |
Probe every published port separately #
One reachable service proves nothing about the other mappings, so test each port on its own.
- Before the reboot, record every published host port, its container, the container's current address and its bridge, and confirm each port answers.
- Reboot the edge PC, wait for the containers to report running, then probe each published port from a client machine. For Grafana on host port 3000, the probe used on the affected stack was
nmap -p 3000 IP_ADDR, with IP_ADDR the edge PC's address. - Record open or filtered from the probe, not from the browser. Filtered means the probe drew no usable answer through the path; it does not by itself prove the application is down.
- Pick one failing port, fire the probe at it several times in a row, and read the rule counters in the next section while you do.
- Compare each container's current address with the destination its forwarding entry uses.
Follow the packets with the rule counters #
iptables -L -v -n lists the filter table, including the FORWARD, DOCKER and DOCKER-USER chains, with a live packet counter on every rule. Re-run the probe and re-read; the counter delta is the decision.
- A counter that moves proves the host accepted and forwarded your probes. If the port still reads filtered, the loss sits past that rule: compare the entry's target address with the container's current address. In the capture from the affected stack, the same published port appeared in two entries aimed at two different container addresses. That is stale forwarding state, and restarting the Docker daemon rebuilds it.
- A counter that never moves points upstream of Docker's rules: the client-facing interface was not ready, or another firewall tool flushed Docker's rules after they were built. Docker does not recreate them while running; a daemon restart does.
Two details keep the reading honest. Packets reach the DOCKER-USER chain already translated, so that chain judges the container address, not the published port. And two visually similar DOCKER entries are not automatically a fault: condemn an entry only when its counters or its destination disagree with the live container.
Start the stack in dependency order #
The mechanism behind most of these faults is initialization order. Under heavy startup load the gap between a container starting and usable networking stretches, and a consumer can start before the service it reads. On a Telegraf, InfluxDB and Grafana stack the working order was database first, dashboard second, telemetry collector last:
docker start influx
sleep 10
docker start grafana
sleep 60
docker start telegraf
Make the script executable, place it under /etc/init.d, and link it into startup:
ln -s /etc/init.d/SCRIPTNAME /etc/rc.d/SYMLINKNAME
The 10 s and 60 s waits are values from one loaded installation, not readiness specifications; confirm from port behavior that each dependency answers before the next consumer starts. If Node-RED stands in for Telegraf, place it by what it consumes. For what that stack asks of the host, see sizing a self-hosted Node-RED, InfluxDB and Grafana historian.
When the bridge, the modem or the firmware is at fault #
Three cases sit beside the sequencing fault, each with its own trigger.
Split interfaces: on hosts whose clients arrive over a second interface, one working fix is to bind Docker to the host bridge. Stop the daemon, add "bridge": "br0" to /etc/docker/daemon.json, start it again:
/etc/init.d/dockerd stop
nano /etc/docker/daemon.json
/etc/init.d/dockerd start
Stop the daemon from a session that survives it, a physical console or host SSH, never from inside a container. Apply the change only when br0 is the intended host-side bridge and the counters place the loss between the interface and Docker's network.
Cellular WAN: a modem assigns its address some time after boot, and a container that starts before that loses the same way. Delay the stack until the interface holds its address.
Firmware-triggered loss: when every container loses network immediately after a firmware upgrade applied with the vendor's upload tool, the upgrade is the trigger, not boot order. One installation recovered by reapplying the same firmware from an SD card.
Worked example: three published ports, one answering, after a reboot #
Inputs, each from the affected installation: three containers with one published port each, Grafana on host port 3000, the port its probe targeted, and Node-RED on 1880 and InfluxDB on 8086, their documented default listening ports. All three run --restart=unless-stopped on the default bridge. After a full reboot, 3 of 3 containers report running and 1 of 3 ports answers. The fix under test uses waits of 10 s and 60 s.
Rule in words: a published port is proven only when a remote probe answers open, and every probe must move a forwarding-rule counter. Count answering ports against published ports. Any port that answers after an ordered start but reads filtered after a plain reboot convicts the boot order, not the application.
The arithmetic:
- Unreachable ports after the plain reboot: 3 published ports − 1 answering = 2 unreachable.
- Five probes at one dead port move its DOCKER entry 5 − 0 = 5 packets while the port still reads filtered, so the host forwards to a destination that does not answer. That entry's target address, checked against the container's current address, is the stale one.
- The ordered start adds 10 s + 60 s = 70 s of startup delay and yields 3 − 3 = 0 unreachable ports.
Result: after the ordered start, all three probes read open.
Decision: a stack that loses ports on a plain reboot and loses none on the ordered start has a boot-sequencing fault. Keep the dependency order, accept the 70 s as the price on a box that reboots a few times a year, and stop tuning the containers. A port that still fails after the ordered start hands you its counter delta, which names the stage to repair.
Prove the repair survives the next cycle #
- Power-cycle the host fully; a Docker service restart is not the same test.
- Probe all three published ports independently once the containers report running.
- Re-read
iptables -L -v -nonce and confirm each port's entry targets the containers' current addresses. - Record how long boot-to-all-ports-open takes; that figure is your readiness baseline for the next change.
If a port fails again, the counter delta tells you whether to look upstream of Docker's rules or at the rule targets, before anyone touches the containers.
Questions asked
What is the difference between filtered and closed in a port scan?
Closed means the host answered the probe with an explicit refusal: the route to the machine works and no process is listening there, so the network path itself is proven. Filtered means the probe drew no answer at all, because something dropped it on the way, which is exactly what a broken forwarding rule or an unready bridge produces. A filtered result on a published port points at the path, not at the application.
Should every stack copy the 10 and 60 second delays?
No. Those two waits came from one heavily loaded installation and are installation-specific values, not readiness figures. Replace fixed sleeps with a readiness poll: loop a port probe or service check against the dependency and start the next container only when it answers. The delay then adapts to each boot, and a slow start costs time instead of reachability.
Why does iptables -L -v -n not show my port mapping?
That command reads the filter table, where the forwarding chains and their counters live. The translation that maps a published port to a container address lives in the nat table. Check it with iptables -t nat -L DOCKER: if that chain is empty or missing your port, Docker's network setup failed, classically after another firewall tool flushed its rules, and a Docker daemon restart rebuilds it.
Does Docker Compose remove the need for the ordered startup?
It gives you the mechanism, not the guarantee. Plain depends_on fixes start order, not readiness. Use the long form with a healthcheck condition so a container waits until its dependency reports healthy, and give each service a probe it can actually pass. On a host where the plain docker start sequence already works, Compose is optional rather than required.