Idunworks Issue #01

Bypassing the Docker Bridge

A tale about your Host, Macvlan/Ipvlan, and Overlay Networking. Because we can.

Why This Matters

Most of us run docker run and never think about the network again. That works fine until it stops working:

  • A latency-sensitive service is slower in a container than on bare metal, and nobody can say why.

  • A legacy application expects its own IP address on the office LAN and refuses to cooperate with NAT.

  • Containers on a cloud VM can't reach anything after you switch drivers, because the cloud network silently drops unknown MAC addresses.

  • A Swarm cluster mostly works, but large requests hang during TLS handshakes.

Every one of these comes down to how packets actually move through the Linux kernel. The default bridge is a sensible general-purpose choice, but it's a compromise: it adds processing steps on every packet in exchange for isolation and convenience. Once you know what the alternative drivers do under the hood, you can pick the right one on purpose, predict where it will cause trouble, and debug it when it does.

That's the purpose of this article: to explain the mechanics clearly enough that the next networking problem makes sense instead of feeling random.


Background: What the Default Bridge Does

By default, Docker attaches containers to the Linux bridge driver. Each container gets its own network namespace and a virtual Ethernet (veth) pair connected to the docker0 bridge, and iptables NAT rules handle traffic in and out.

People leave the default bridge to improve performance, cut latency, put containers directly on the physical network, or connect containers across multiple hosts. Each alternative driver uses a different part of the Linux kernel network stack. Below is how traffic flows under three of them: host, macvlan / ipvlan, and overlay (VXLAN).


1. The host Driver: No Separate Network Namespace

With --network host, Docker skips network virtualization entirely.

  • Shared network namespace. Docker does not create a new network namespace (netns) for the container. Its processes run in the host's root network namespace (init_net) and see the same interfaces, routing tables and sockets as any host process.

  • No virtualization overhead. Traffic between two host-networked containers, or between a container and a host process, goes through the host's loopback (lo) interface or normal routing tables. There are no veth pairs, no bridge processing and no NAT translation.

  • The catch: port collisions. All host-networked containers share one socket space, so two containers cannot both bind port 8080. Port management moves into application configuration, and that becomes your job.

  • Platform note. Host networking is a Linux kernel feature. On Docker Desktop (macOS/Windows), "the host" is the Linux VM Docker runs in, not your actual machine, so behavior differs.


2. macvlan and ipvlan: Containers Directly on the Physical Network

These drivers attach containers directly to a physical host interface (the parent, e.g. eth0) and bypass the host bridge entirely. The upstream switch or router then handles Layer 2/Layer 3 addressing for the containers as it would for any other device on the network.

macvlan (Bridge Mode)

  • One MAC and IP per container. Docker creates a kernel macvlan sub-interface on the parent interface for each container. Each one gets its own MAC address and an IP from the physical network's subnet, so on the LAN every container looks like a separate physical machine.

  • VLAN trunking (optional). If the parent is written with dot notation (e.g. -o parent=eth0.10), Docker creates an 802.1Q VLAN sub-interface and tags the containers' traffic with that VLAN ID. Without it, macvlan attaches directly to eth0.

  • Switching inside the kernel. When Container A talks to Container B on the same host, the kernel matches the destination MAC against its macvlan sub-interfaces and delivers the frame straight into B's namespace. The packet never reaches the physical wire or a virtual bridge.

  • The host can't reach its own containers. By design, the kernel blocks traffic between the parent interface (eth0) and its own macvlan sub-interfaces. If the host needs to reach its containers, create a separate macvlan "shim" interface on the host and add routes for the container IPs through it.

  • Environment limits. Many public clouds and enterprise switches drop traffic from unknown MAC addresses or limit how many MACs one port may use. Macvlan often fails there.

ipvlan (L2 and L3 Modes)

  • Shared MAC, separate IPs. Unlike macvlan, ipvlan containers all use the parent interface's MAC address but get their own IP addresses. That makes ipvlan the usual choice where upstream switches enforce MAC security policies or where cloud networks only accept the host's MAC.

  • L2 mode (the default). Works much like macvlan. The kernel switches traffic based on IP lookups, containers sit in the parent's subnet, and ARP and broadcast work normally. Traffic between containers on the same host never touches external hardware.

  • L3 mode. The host acts purely as a Layer 3 router. There is no Layer 2 domain for the containers: ARP is not used and broadcast and multicast don't pass. Packets to other containers or outside networks go straight to the host kernel's routing table, which forwards them. Containers on different subnets on the same parent can reach each other, but the upstream router needs static routes to the container subnets via the host's IP, because nothing announces them.


3. The overlay Driver: Networks Across Multiple Hosts

When containers run on separate machines (a Docker Swarm cluster, for example), the overlay driver builds one logical Layer 2 network on top of the physical Layer 3 network between the hosts.

Two Network Interfaces per Container

An overlay container is attached to two networks:

  1. The overlay network (data path). The container's overlay interface is a veth pair into a hidden, per-network overlay namespace on each host. Inside that namespace, a Linux bridge (br0) connects the container interfaces to a VXLAN device, which carries traffic to other hosts.

  2. docker_gwbridge. A local bridge that gives the container outbound access to external networks, and inbound access for published ports.

VXLAN Encapsulation (RFC 7348)

When Container A on Host 1 sends an Ethernet frame to Container B on Host 2:

  1. The frame crosses br0 in Host 1's overlay namespace and reaches the VXLAN device.

  2. The kernel adds a VXLAN header containing a VNI (VXLAN Network Identifier) that identifies which overlay network the frame belongs to.

  3. It puts that inside an outer UDP packet (destination port 4789) addressed to Host 2's physical IP.

  4. Host 2 receives the packet, its VXLAN device strips the outer headers, and the original Layer 2 frame is delivered into Container B's network namespace unchanged.

The Control Plane: Raft and Gossip Are Separate

Two different mechanisms keep the cluster consistent:

  • Raft (Swarm managers, TCP 2377). Manager nodes use the Raft consensus algorithm to keep a consistent cluster state: services, tasks, network definitions, and IP address allocation.

  • Gossip (all nodes, TCP/UDP 7946). Network endpoint information is spread peer-to-peer between the nodes that share a network: which container IP and MAC lives behind which host. Each host uses it to fill in the VXLAN forwarding (FDB) and ARP tables in advance, so the overlay doesn't need to flood the network to find where containers are.

Encryption

  • Control plane: Swarm management traffic and gossip are encrypted by default.

  • Data plane: VXLAN traffic between containers is not encrypted by default. Create the network with --opt encrypted to turn on IPsec between hosts. This costs extra CPU.

Operational Notes

  • MTU overhead. VXLAN adds about 50 bytes of headers per packet. If the overlay MTU isn't reduced to fit (e.g. 1450 on a 1500-byte network), large packets get dropped. The usual symptom is connections that hang during TLS handshakes or big transfers.

  • Firewall ports. Open 2377/tcp (managers), 7946/tcp+udp (gossip), and 4789/udp (VXLAN data) between all nodes.


Driver Comparison

Driver

How it works

Main benefit

Main cost

host

No separate network namespace; the container uses the host's network stack

No virtualization overhead: no veth, bridge or NAT

Shared port space; port conflicts are yours to manage

macvlan

Each container gets its own MAC and IP on the physical LAN

Containers look like real devices on the network

Host can't reach its own containers without a shim; often blocked by clouds and strict switches

ipvlan L2

Shared host MAC, separate IPs, switched in the kernel

Works where unknown MACs are blocked

Containers must fit in the parent's subnet

ipvlan L3

Shared host MAC; the host routes container traffic

Scales well; no ARP or broadcast traffic

Needs static routes upstream; no broadcast or multicast

overlay

L2 frames wrapped in VXLAN/UDP 4789 across hosts

One logical network across many hosts

About 50 bytes per packet, MTU tuning, a control plane to run, data not encrypted by default


What We Learn

  1. Bypassing the bridge moves the work somewhere else. Each driver removes some processing, but the job still has to be done by someone. With host you manage ports. With macvlan/ipvlan the physical network handles addressing. With overlay the kernel pays for encapsulation and the cluster runs a control plane.

  2. Choose the driver based on your network:

    • host for maximum speed on a single host, when you control port assignments.

    • macvlan when containers must appear as separate devices on the LAN (legacy apps, network appliances, DHCP- or MAC-dependent software).

    • ipvlan when the network limits MAC addresses, which is typical of public clouds and locked-down enterprise switches.

    • overlay when containers on different hosts need to share one network, as in Swarm.

  3. With macvlan/ipvlan, traffic between containers on the same host never reaches the wire. The kernel switches it internally, which is part of why these drivers are fast.

  4. The host-to-container block on macvlan is intentional. Plan for a shim interface if the host needs to reach its containers.

  5. Overlay is "Layer 2 over Layer 3". Expect encapsulation overhead and set the MTU correctly. Treat the VXLAN data path as plaintext unless you turn on encryption.

  6. Raft and gossip do different jobs. Raft keeps the cluster's desired state consistent on managers. Gossip tells every node where each container is. Both need their firewall ports open.


Outro: Choose Your Packet's Adventure

Think of the network drivers as different ways for your packet to get to work:

  • host walks out the front door. No wrapping, no detours, very fast. It does have to share one mailbox with every other process in the building, and if someone else already took port 8080, too bad.

  • macvlan gets its own house on the street, with its own MAC address and IP. The neighbors think it's a real machine. Its own parent host, oddly, isn't allowed to visit.

  • ipvlan lives in the same house as the host but has a separate apartment number. The switch sees one MAC and lets everyone in. In L3 mode nobody gossips over the fence, because ARP is gone.

  • overlay gets put in a box, inside another box, and shipped across the network with a VNI label on it. It arrives unchanged, but it's about 50 bytes heavier and occasionally gets stuck in a tunnel that's too small (check your MTU).

Phew! Lengthy text. Anyway, none of them is the "best" driver. Each is a different trade-off, and now you know what each one costs.

So the next time someone says "it's always DNS," you can reply: "Sometimes it's the MTU." 🐳📦

Here's your chance to make this into a t-shirt!

Log in to like this article, or create an account .
0 reads

Comments

Log in to post comments. Log in
No comments yet. Be the first to comment.