What this is about. While working out why my new robot counter had such a flattering opinion of my readership, I read the raw server log and found that the most requested page on the whole machine was one that has never existed. This is what I did about it, and the mistake that nearly took all my sites offline.

Why. Building the trap took an afternoon. Working out what must never be trapped took longer, and that is the part worth writing down.

Fifteen thousand requests for nothing

/wordpress was the most requested path on the server for my newly deployed site. More than the homepage. Fifteen thousand requests in a single log file.

I have never run WordPress there (!). I once had a React front end that read content from WordPress on a different machine entirely, which is not the same thing and would not leave this trail.

Every one of those requests forged a browser user-agent, rotating through Chrome, Edge, Safari, Firefox and Opera in almost exactly equal numbers. That is one scanner with a costume box, not many visitors. It is also the kind of sneaky little robot my shiny new counter cannot see, because it is pretending to be a person. Once identified, never doubted. No amount of counting fixes that.

What it can see is that the requests came from a couple of dozen addresses, two of which accounted for 87 percent of them.

Twenty bytes a second

A tarpit is an old idea (a pit and a load of tar): answer, but answer so irritatingly slow that the other side wastes real time. Ha! So I gave them something to download. A file called wordpress_backup.sql, served at twenty bytes per second. The whole megabyte takes just over fourteen hours.

The sketch I started from bolted on a Python service whose entire job was to dribble ones and zeros forever. The web server does it alone, sulkily, for nothing: a finite file served slowly enough is indistinguishable from an infinite one, so there is no extra process, no extra port and nothing extra to keep patched. I also dropped a (for me) clever header containing a JavaScript infinite loop, on the grounds that response headers are not executed by anything.

Then the web server and I fell out over what "slow" means.

The rate limit throttles the headers too. A client spends eighteen seconds receiving about 360 bytes of response headers, so it is stuck before it even learns the filename it is being offered. I spent a while convinced my header configuration was broken, when in fact it was still in flight. Stained by tar up to my knees already...

The defaults make a worse trap than small buffers do. Out of the box the server hands over one four-kilobyte chunk immediately and then goes quiet for about three and a half minutes, which any scanner with an inactivity timeout treats as a dead connection and abandons. Shrinking the buffers turns that into a steady dribble, so the connection stays interesting enough to sit in. Smaller buffers, stickier trap.

Up to my bollocks in tar

Then I widened it to the other things robots ask for. /.env, /.git/config, the files that hand over credentials when they exist. All of those begin with a dot, so a rule for dot paths looked obvious.

So does /.well-known/acme-challenge/, which is how my TLS certificates prove they are mine when they renew.

A rule that trapped every dot path would have tarpitted my own certificate renewals. Nothing would have broken that day, or that week. It would have surfaced a month or two later as expired certificates on every site at once, and I would never have connected it to the "clever" thing I did in October.

There is a second exclusion of the same shape. Search engines speculatively ask every site for /wp-json/ and /wp-sitemap.xml, just in case WordPress is hiding there. A blanket rule for anything starting with wp would have put Googlebot in the tarpit for fourteen hours at a time. I first assumed those requests were leftovers from some old setup and would fade. They are not. They are standard crawler behaviour and they will keep arriving forever (or until Mullenweg goes crazy again), so the exclusion is permanent. The note in the config now says so. The version where it looks temporary is the version somebody deletes at the Spring cleanout.

What it has caught

So far the trap has held 91 requests for a combined 25.6 hours. The average stay is seventeen minutes. Twenty lasted more than ten. The longest single connection sat there for eight hours and ten minutes.

The funniest part is that it is barely catching WordPress scanners at all. The top catches are /.env and /.git/config. I built a little black sticky goo for one kind of robot and it filled up with another. Never thought slow connections could bring me so much joy :)

What happened, and what else I could have done

A page that does not exist was my most popular page. It now answers, very slowly, to everyone except the three kinds of visitor that must never be slowed down: real people, search engines doing their normal speculative sniffing, and my own certificates renewing themselves.

A tarpit is not the only answer, and it is not obviously the best one:

The honest caveat is that a tarpit costs me something too. I hold a connection so that they hold one, and if the scanning came from thousands of addresses instead of a couple of dozen, that trade would stop favouring me. Hence a hard ceiling on how many of these connections can exist at once, and a refused one being closed rather than answered.

And a small confession about measurement, which is becoming a theme. My first instinct was to keep these requests out of the logs entirely, since fifteen thousand lines a day is noise. I did exactly that, and then realised I had built a trap with no way of telling whether it caught anything. They now go to their own log file, which is how I know about the eight hours.

Hit me in the comments if you think my reasoning is flawed, or if you think bots should be treated as equals.