What this is about. I asked my AI for a full maintenance audit of my CMS, and we spent an afternoon closing what it found. Then the first deploy failed, and one of my own fixes had caused it.
Why. Every fix changes something somebody depends on, and sometimes that somebody is your own tooling. It's better to find out in a rollback than in an outage.
The audit
My brief was short:
i want a full audit with a report at each step.
It came back in parts: UX, marketing, product, and security. I said "let's concentrate on the sec. items first", and we did them in a few rounds. The ones worth telling:
The quiz answered itself. My course pages fetched each quiz with the right answer included, unscrambled, and the page just didn't show it. Anyone opening the browser's network tab with a beautiful or curious mind could get a perfect score. Students now get the questions, and only course editors get the answers.
The browser named its own price. When you bought a course, the checkout trusted the amount the browser sent. Now the server looks up the price itself, and when the payment comes back it checks that what was paid matches what was owed. If it doesn't, nothing gets unlocked and I get a loud log line. In a webshop this is one of the building blocks. I started the courses out of a need for a small course platform, no payment. Easy to forget the nuances.
A door between my services had a wall, but no lock. The route that grants course access after a payment can only be reached from inside my own network. It still trusted anything that reached it. It now wants a shared secret, and it refuses when the secret isn't set, instead of letting everyone in.
Logging out didn't log you out. The button cleared your cookies, but the session behind them stayed valid until it expired. Logout now ends the session on the server too. A deactivated account, or a changed password, ends every session that account has. Easy to miss when you're up to your elbows in code-grease creating a course platform.
Each fix came with a test, and the test suites ran green: about 620 tests across three services. I've seen worse.
The deploy
Another fix from the same evening hid the API's documentation pages in production. FastAPI publishes them at /docs by default, a guided tour of every endpoint you have. Strangers don't need that tour.
The first site went out, and the deploy stopped with:
Health checks timed out after 120sMy deploys are of the blue-green kind: The new version starts next to the old one, a script checks that it's healthy, and only then does traffic move over. The script's idea of "healthy" was asking for /docs and getting a page back. I had just made sure it would never get one again.
So for two minutes the new version stood there, perfectly fine, knocking on a door I had locked an hour earlier. Then the script gave up, the old version kept serving, and nobody visiting the site noticed a thing.
The fix was one word in three scripts: ask /health instead. That route exists precisely to say "I'm alive". Nobody will ever hide it, because hiding it would be pointless. Go there and it will kick your ass to 404.
Then the evening ran long. Halfway through the fleet I said "hold off the rest for tomorrow", and the AI started stopping a deploy that was already halfway out. Yeah, what does "hold off" mean really? I corrected it:
i meant to not start a new deploy. Finish what is started as agreed
Fair. A half-finished deploy is worse than either a finished one or one that never started. The last site went out the next morning, and every site passed the same checks: no docs, logout only by POST, internal routes closed to the public, and a quiz with no answers in it.
What happened, how we fixed it, and how else it could be done
Quiz answers in the response: a separate public schema without the answer field.
Client-set prices: the server prices the order and checks the paid amount before it unlocks anything.
Unlocked internal route: a shared secret, with the route closed if the secret is missing.
Logout that didn't: sessions are revoked on the server at logout, deactivation and password change.
The failed deploy: health checks ask a dedicated
/healthroute, not the docs.
Other ways to handle the docs:
Option | Better at | Costs |
|---|---|---|
Turn them off in production (what we did) | Nothing to find, nothing to maintain | You lose the docs in production, so you read them locally |
Keep them behind an admin login | Docs are still there when you need them live | Another protected route to get right |
Block them at the web server | One rule covers every service | The app still serves them to anything inside the network |
Leave them on | Easy for integrators | A free map of your API for anyone who's curious |
Before you lock something
Search your deploy, monitoring and health scripts for the URL you're closing.
Give every service a
/healthroute, and point every probe at it.Make sure a failed health check rolls back, before you ever need it to.
When someone says "stop", ask if "don't start" or "abort" is what they mean. Or right?
The rollback did exactly what I built it for, on the first night it was needed. More from the workshop is on the Ship Log.