From prototype to proposal

Category: AI Engineering/Benchmarking

Docker, GPUs, a rented graphics card by the hour, a polite "no thanks", and what I'd do differently.

Five posts in, I had a measured, tuned and slightly over-documented prototype running on a laptop. This last post covers the step every side project eventually meets: turning it into something a real company could run, and putting a price on it. One of those went well.

What the client actually needed

The interview with Company A changed my picture of the project more than any benchmark had.

I'd assumed the assistant would answer questions from the company's customer content. It wouldn't. It was meant to help the people who use Company A's software: busy, often non-technical editors who land on a screen with a red warning and think "right, and now what?". They frequently don't dare ask basic questions in front of colleagues, or expect to get a support person helping them in minutes. Maybe both. A private assistant that never sighs is exactly the point.

That had consequences:

The knowledge base was small and not sensitive: the company's own help content, plus a battery of real user questions with answers. That was the human-verified test set I'd been wishing for since post 3.
About 80 % of questions repeat, which made approved answers (post 4) the centrepiece.
The answer should depend on where the user is in the tool: the same question means different things on different screens.
An earlier attempt to create a private AI had stalled because nobody could tell why its answers were bad. My whole method, measuring every step, was the answer to their actual pain. (Remember that one. It comes back.)

Their technical requirements were sensible and specific: everything in Docker, nothing on external servers, Postgres with pgvector (post 5), production inference on vLLM, an authentication step on every request using their existing logins, and one RTX 5090 with 32 GB of memory.

Building the boring bits

An authentication step that fails closed. Every request carries the user's session token, and the assistant asks the company's own login service who that is. Until their service is connected, a stub lets everyone in. Three rules make it safe:

The token itself is never stored, only a short fingerprint.
A missing or rejected token gets "not authenticated".
If the login service is down, the request is refused, never waved through.

A backend switch. In development I use Ollama on a Mac; production uses vLLM on an NVIDIA GPU. Both speak the same OpenAI-style API, so one setting switches between them, for answers and embeddings alike.

A Docker image with nothing left to download. The reranker's weights are baked into the image and downloads are switched off at runtime, so nothing reaches out to the internet. The image is about 8 GB, of which about 5 GB is the GPU libraries and 2.2 GB the reranker. It answered correctly in a container on the first try. In about 35 seconds.

That number deserves a second look. Docker on a Mac can't reach the Mac's GPU, so the reranker ran on CPU: 8.7 seconds per 30 passages, measured. Nothing in the code was wrong. Only the furniture was, and you could hear the cooling fans spinning up.

The Mac-to-GPU gap

Developing on Apple silicon and deploying on NVIDIA is a common setup and a quiet source of friction. The code is identical, but speed, the vLLM software and the exact model build can only be checked on the real kind of hardware. And remote access to the client's GPU for staging wasn't on offer.

So I built one command that turns any fresh Linux GPU machine into a test bench:

Preflight checks, each stopping with a plain-language message: GPU driver, Docker's GPU support, disk space, config. For example: the GPU driver is too old for the software, so install a current one, "otherwise the GPU sits unused". That one is very easy to miss otherwise: everything still runs, just painfully slowly on the CPU.
Model downloads and startup, then waiting until every service answers.
A test run: indexing, questions and a load test, written to one report that records the GPU, software versions and git commit, so any two reports can be compared question by question.

Two design choices made it safe to run on a rented machine:

No personal or client data. The test knowledge base is the project's own documentation, with its own question set.
A memory budget. One setting (GPU_BUDGET_GB=32) makes a bigger card behave like the client's 32 GB one. Anything that runs there also fits in production.

Then the pleasant surprise: a small Swedish provider rents an NVIDIA L40S (48 GB) by the hour for about 7 kronor, roughly 60 euro cents. A full check run costs less than a coffee. Their "monthly cap" turned out to be exactly 720 hours times the hourly price, so it isn't a discount; it's what running the card non-stop costs, with a lid on it. Read the small print, then set an alarm to switch the machine off.

The first test run on the demo data even found a real issue: the "I don't know" threshold, calibrated on my chatty email archive, was too strict for documentation-style text. It refused a question whose answer was right there. Thresholds get calibrated per knowledge base, which is now in the deployment checklist.

The proposal

Company A wrote before our first live meeting: they wanted a fixed price for the whole project, three weeks sounded right to them, and they had told another candidate the same. They needed numbers for an internal meeting the next day.

I sent a careful proposal:

A fixed price of 135,000 SEK (roughly €12,000) excluding VAT, for about three weeks.
Two steps with a checkpoint: week 1 (knowledge base in, first measured quality report, list of gaps) for 40,000; weeks 2–3 (admin interface, approved answers, context, login, deployment) for 95,000. They could stop after week 1 and pay only that.
A clear minimum delivery, a 30-day bug-fix warranty and an explicit list of what wasn't included.
Quality promised as a measurement, not a percentage, because quality depends mostly on their knowledge base.
Rights: they could use and develop everything; I kept the right to reuse the generic platform.

One more thing happened before it went out, and it's worth admitting. My first draft said the login and vLLM support were "built and tested". They were built. They had been tested against simulations and a stand-in server, not against Company A's real login service or a real vLLM server. A fact check caught it, and the wording became "prepared, tested live during the project". Lesson filed: fact-check every "done" in a proposal before a customer reads it.

The answer

"Thank you for your proposal. We went through it, and the price is far above what we consider reasonable for a project like this, so unfortunately we have to decline."

Polite, kind and final. They had been warm and positive throughout, about my person and the method. It failed on price, not on trust (I think).

What went wrong

With some distance, six things:

I never asked about their budget. Their email had two signals: they wanted to agree on what it "may cost", and there was a competing offer. I anchored blind instead of asking what they'd set aside and what they were comparing against.
"Three weeks" meant different things. I heard about 120 hours of full-time specialist work. They probably heard "a small project", with a budget to match. "Far above" suggests a half or a third of my number.
One package, take it or leave it. With a single price, the only possible answer to "too expensive" is no.
No invitation to counter. Nothing in my proposal said "if this is above your budget, tell me your level and I'll shape the scope to it".
The price arrived before the relationship. A written number, judged in an internal meeting without me in the room, before we'd even met live.
The strongest argument was buried. Their previous attempt failed because nobody could see why answers were bad. Measurement was the cure. In my proposal it was one bullet point, not the reason for the price.

What wasn't wrong

The rate itself. Three weeks of senior specialist work for less than my car mechanic charges for three weeks isn't a market rate. A client whose ceiling sits far below that isn't one to win by cutting the price. So asking about budget isn't about lowering your number. It's about qualifying: finding out early, and kindly, whether there's a deal at all, before anyone spends days measuring and an stressed-out morning trying to write a proposal before their (board?) meeting.

What I'd do next time

Ask about budget before writing anything: "What have you set aside, and what are you comparing with?"
Offer three levels, not one price: a measured proof on their own data (about a week), a working assistant in Docker, and the full product. The client picks a level instead of saying no.
Lead with the value that's unique to them: here, "you'll always know why an answer is wrong".
Present the price in a conversation, and send the document afterwards as a summary.
End every proposal with an opening for a counter-offer.
Decide your floor beforehand, so a counter-offer doesn't talk you below it.
Fact-check every "built and tested".

Nothing was wasted

This is where the "empty box" principle from post 1 turned out to apply to the business as well. The platform, the test battery, the measuring tools, the pgvector store, the Docker and GPU setup and the decision log were all built to be reused. None of it was tailored so tightly to one client that it leaves with them. The next client, and I think I know which one, doesn't start from zero.

Oh, and I can now get crazy precise intel from my 30K+ emails, in a private browser interface which gets even better over time.

The series in six sentences

A newer small model beat a bigger one: size isn't quality, and quantization wasn't the excuse.
You can test an assistant on documents you've never read by writing the answer key first.
Even the yardstick needs calibrating: measure the noise, and don't trust a single judge's absolute number.
"Similar" isn't "the same": a semantic cache can serve the right answer to the wrong question very quickly.
A database swap is never just a swap: same vectors, 20 fewer correct sources, until it was measured and fixed.
A good method doesn't sell itself; ask about the budget first.

Thanks for reading. If you're building something similar: measure first, then believe.

//T