My database was on the internet

and other things you learn by asking for a one-liner

What this is about. Docker publishes container ports on every network interface by default, and it does so in a way your host firewall never sees. Mine had done that with the database of every site I run, for months. This is how I found out, how I fixed it without breaking zero-downtime deploys, and the pile of smaller problems that came loose on the way.

Why. If you self-host with Docker Compose, there's a fair chance the same is true for you right now, and the obvious one-line fix might take your sites down. It nearly did mine.

It started innocently. A form wanted me to describe my hosting setup "in one line, e.g. Caddy in front, Docker Compose per project, deploy via git pull + rebuild." I have an AI pair programmer. I am, as the LinkedIn posts keep telling me, an expert with AI now. So I asked it to read my repo and write the line for me.

It wrote a lovely line. Then I asked it to shorten it for the web, and it quietly took out the port numbering scheme, adding, almost as an aside:

"…advertising predictable ports while that gap is open would be giving directions."

What gap?

The gap was that the Postgres database of every one of my seven sites was published on 0.0.0.0: every network interface, including the public one. It had been like that for months. There was even a comment in docker-compose.yml that said, with great confidence:

# Server firewall handles external access control.

Reader, the server firewall did not handle external access control.

Part 1: Docker's default, and why your firewall isn't looking

This is the part everyone learns once, usually the hard way.

services:
  db:
    image: postgres:16-alpine
    ports:
      - "${DB_PORT}:5432"

A port mapping without an IP in front of it binds to all interfaces. That's the default, and in plenty of tutorials it's simply what you type.

"Fine," you say, "I have ufw." Docker doesn't care. It publishes ports by writing its own NAT rules, and that traffic goes through the kernel's FORWARD path and Docker's own chains. It never reaches the INPUT chain where ufw and friends do their filtering. So your firewall can say "deny everything" and Docker will still forward port 5432 to your container, without a word.

What actually kept my databases off the public internet was my hosting provider's edge firewall, which happened to block those ports. That's a defence I didn't configure, can't see, and would lose the day I changed provider, changed network plan, or someone at the provider changed a rule. Behind it sat exactly one more barrier: the database password.

That's not defence in depth. That's defence in luck.

Part 2: The obvious fix, and why it would have broken everything

Every blog post (including, apparently, a note I wrote myself months ago) says the same thing:

ports:
  - "127.0.0.1:${DB_PORT}:5432"   # loopback only, done!

On a normal single-stack setup, yes, do this, you're done, go to bed.

My setup isn't normal. I do blue-green deploys: the database lives in a "blue" Compose project, and the app containers run in a separate "green" project that reaches the database through host.docker.internal:<port>. From inside a container, host.docker.internal resolves to the Docker bridge gateway (something like 172.17.0.1), not 127.0.0.1. Bind the port to loopback and green can't reach the DB any more. Every site goes down on the next deploy.

I know this because past me already did it once and lost four hours. That's where the confident "server firewall handles this" comment came from: someone (me) switched 127.0.0.1 back to 0.0.0.0 to make the outage stop, and wrote down a reassuring lie so future me wouldn't touch it.

The AI laid out three options:

1. Firewall rule in Docker's DOCKER-USER chain that drops public traffic to container ports (a script for this already existed in the repo, naturally).

2. Bind to the bridge gateway IP instead of loopback: reachable from containers, not from the internet.

3. Don't publish the port at all. Put the DB and the green stack on a shared private Docker network and connect by name.

It recommended 1+2. I picked 3, because if a door is a problem, I'd rather have a wall than a better lock.

Part 3: The actual fix

Each site gets its own private network. The database joins it with a stable alias; the green services join it too; the ports: entry goes away entirely.

# docker-compose.yml (shared by blue and green)
services:
  db:
    networks:
      default:
      sitedb:
        aliases:
          - ${BLUE_PROJECT:-${COMPOSE_PROJECT_NAME}}-db

networks:
  sitedb:
    external: true
    name: ${BLUE_PROJECT:-${COMPOSE_PROJECT_NAME}}_sitedb

# docker-compose.green.yml
services:
  api:
    networks: [default, sitedb]
    environment:
      DATABASE_URL: postgresql+psycopg2://…@${BLUE_PROJECT}-db:5432/…

Three details mattered more than the YAML:

1. Compose validates every external network in the file, even for services you aren't starting. Testing locally, a UI-only deploy (which never touches the DB) failed with network … declared as external, but could not be found. Had that shipped, the normal everyday deploy for every not-yet-migrated site would have broken the moment I merged. The fix is the boring pattern my repo already used for another shared network: declare it external everywhere, and run an idempotent docker network create … || true before anything starts.

2. Compose won't adopt a network it didn't create. If you docker network create a network and then declare it as compose-owned, you get network … was found but has incorrect label. Hence "external everywhere", not "compose creates it in blue, green joins it."

3. Timing. Recreating the DB container without its published port instantly cuts off the old green stack, which still connects through that port. My deploy already has a short "swap" window where the old release is stopped and the new one isn't started yet. Moving the DB there costs no extra downtime. Anywhere else, it's an outage with extra steps.

All three of these were caught by a throwaway local rehearsal with real Docker: an "old-style" database with a published port and a canary row, migrated by the new script, then queried from a green container by alias. Not one of them would have been caught by reading the diff and nodding.

Part 4: The rollout, featuring a robot being told "no"

Plan: canary on the least-used site, then one more, then the rest, each via a full blue-green deploy.

Before each one: who is actually connected to this database? My first instinct was ss -tn on the host. It showed zero connections, while Postgres had 24 open. Docker's NAT forwarding happens in the kernel; no host process owns those sockets. The tool that tells the truth is Postgres itself:

SELECT client_addr, count(*) FROM pg_stat_activity
WHERE client_addr IS NOT NULL GROUP BY 1;

Before: every connection came from the Docker gateway (that's the NAT path). After: from the containers' own addresses on the private network. And crucially, never anything from outside.

After two sites went fine, my AI, flush with confidence, wrote a neat script to deploy the remaining five production sites in one go. The safety layer in the tooling looked at "AI runs a script that deploys five production sites" and said no. Production deploys need a human saying yes, per site.

Annoying? Briefly. Correct? Completely. It turns out "you're an expert with AI" really means "you're an expert at reading what the AI proposes and deciding whether you'd sign it." So I typed "ship ", five times, and each one got checked before the next started. Seven sites, seven clean deploys, zero databases left on public interfaces.

Part 5: The loose threads you find when you start pulling

Once you're looking, you keep finding things:

• Secrets files. The live .env files were already owner-only. Some old backup copies of secrets files were not. chmod 600, and the backup copies are a reminder that every copy of a secret is a secret.

• Database dumps. Some were world-readable, and so were a couple of backup directories. On any box with more than one account, that's a full copy of a customer database for whoever looks. chmod 750 on the directories, 600 on the dumps.

• Retention. 30 of those dumps were stale, uncompressed, months-old copies from an old deploy path. Disk wasn't the issue (60 MB). The issue is that a months-old dump can still contain the data of someone who has since asked to be deleted. We kept the newest one per site as a long-term fallback and deleted 24.

• One site had no nightly backup at all. Its newest backup was five months old. It has one now. I checked the first one: valid gzip, every table present, and Postgres's -- dump complete trailer at the end, because a backup you haven't checked is just a hopeful file name. (Checked is still not restored, which is a different post.)

And one of the best moments of the day: I asked for umask 077 at the top of the deploy script. The AI pushed back. My nginx serves the branded error pages and some static files from inside the deploy directory, as the www-data user. A script-wide umask 077 means every file git pull writes during a deploy becomes owner-only, and the next time the "we'll be right back" page changed, nginx couldn't read it. Visitors would get a bare error precisely when the friendly one was supposed to appear. So the umask is scoped to the one subshell that writes .env. Same protection, no landmine.

Then the API ports. Same story, smaller stakes: every backend service and the frontend had their ports published on all interfaces too. With the database on its own private network, the reason they "had" to stay public was gone: the services were calling each other through the host's published ports, when they all live in the same Compose project and can simply use each other's service names. So the ports went to 127.0.0.1 (where nginx and my health checks already looked for them), the cross-service addresses went back to plain service names, and the AI checked from inside a running container that those names only ever resolve to that site's own containers, even on the one network all sites share. Seven more deploys, one at a time.

The final check of what was still listening on the public interface found one more thing, and it wasn't Docker at all: Ubuntu's resolver answering a Windows-style local-network name protocol (LLMNR) on every interface. Nothing on a VPS uses it. One drop-in config file, off. Plus three cron jobs for a service I'd retired a week earlier, quietly failing every five minutes.

Hardening has side effects. That's the whole point of this post.

Part 6: Hardening vs. not hardening: what actually happens

If you don't harden

If you harden carelessly

The pattern in the second table: every hardening step changes behaviour, and behaviour changes break things that depended on the old behaviour. The cure isn't to skip hardening. It's to treat a security fix like any other change: understand who depends on the thing you're closing, rehearse it, canary it, verify it with the system's own view of the truth.

The checklist I wish I'd had

1. docker ps --format '{{.Names}} {{.Ports}}'. Anything showing 0.0.0.0: that isn't your reverse proxy deserves a reason.

2. Don't trust ufw/iptables INPUT rules to cover Docker-published ports. Use DOCKER-USER, bind to a specific IP, or don't publish at all.

3. Prefer private Docker networks over published ports for anything that only containers need to reach.

4. Ask the database who's connected (pg_stat_activity), not the host.

5. find / -name '*.sql*' -perm -o=r and find / -name '.env*' -perm -o=r. Be prepared to feel things.

6. Every backup needs: a schedule, a retention limit, restrictive permissions, a copy somewhere else, and a test that it actually restores.

7. Rehearse the change locally with real containers. The bugs live in the transitions, not the end state.

8. Canary one site, verify, then the next. Let something say "no" to the bulk version.

What happened, how it was solved, and how else it could be done

Other ways to do it, and what they cost:

For a small fleet like mine, "don't publish what only containers need, and bind the rest to loopback" was the cheapest wall. If you run many hosts, a DOCKER-USER rule set as a second layer is worth it on top.

As for being an expert with AI: the AI found the problem because I asked a lazy question, and wrote most of the fix. The expertise was in the bits it couldn't decide alone: choosing the wall over the lock, approving deploys one at a time, and the moment it said "that umask will break your error pages" and I said, "…yes, fair."

Which, now that I write it down, is just what being an engineer always was. With better autocomplete.