Slow is the same as broken.

You can run a good coding model on your own hardware today. No account, no bill, nothing leaving the room. That part is solved.

A tool you stop reaching for is not a tool you own.

Speed is not a nice-to-have here. It decides whether the thing gets used at all. So tonight is not about running a model locally, it is about making the one you already have quick enough to live on, on the same hardware, for the same nothing per month.

If you were at Bare Metal in July you built exactly this rig and watched it crawl. If you were not: it is a model file on your own machine, answering through a small server you run yourself. That is the whole prerequisite, and step one tonight sets it up from scratch.

"Slow" is three different problems.

Nobody can fix a number they have not split up. One reply is made of three separate waits, and only one of them is usually the reason you are suffering.

So the order is fixed: measure first, change one thing, measure again. Everything after this slide is that loop. You cannot tune what you have not timed, and you cannot credit a change you made three of at once.

Make the model smaller so it can move faster.

A model is a huge pile of numbers. To write one word it has to haul all of them out of memory. Store those numbers with less precision and there is physically less to haul, so each word comes out sooner. That is called quantizing, and its real job is to make the model fit on your graphics card, which is where the big win actually comes from.

So the rule is: take the smallest version that still passes a real task of yours. Not a benchmark of somebody else's, and not the one a comment thread recommended. You will do exactly this in step two.

Two servers. Pick by what you own.

Something has to sit in front of the model and answer requests. There are two serious options and the choice is made by your hardware, not by taste.

both come out as the same socket http://your-machine:PORT/v1

Which is why the choice is reversible. Your coding tool is pointed at an address, not at a technology, so you can swap the thing behind it tonight without touching your editor again.

Five knobs. In this order.

Ranked by what they actually did on a real card, not by what sounds biggest. The first two we measured ourselves the day of this session; the rest are what people typically see, which is exactly why you re-time after every single one.

One knob at a time, re-time, keep what won. Turn three at once and you have learned nothing except that the pile of three did something. This is the most commonly skipped step of the night and the reason people leave unable to reproduce their own result.

Time it before you change it.

Nothing you do after this is worth anything without this number. Get a baseline reading from the model exactly as it is right now, so every later change can be credited or blamed.

# built-in benchmark: reading speed and writing speed, separately
./llama-bench -m models/coder-7b-q8_0.gguf -p 512 -n 128

# built-in benchmark: reading speed and writing speed, separately
./llama-bench -m models/coder-7b-q8_0.gguf -p 512 -n 128

# same tool, .exe, and use a full path if it is not on PATH
llama-bench.exe -m models\coder-7b-q8_0.gguf -p 512 -n 128

# on the Nebius box, same tool. -ngl 99 pushes it onto the card.
./llama-bench -m models/coder-7b-q8_0.gguf -p 512 -n 128 -ngl 99

# start the server
./llama-server -m models/coder-7b-q8_0.gguf --host 0.0.0.0 --port 8080

# in another window: ask it something and time the whole thing
time curl -s localhost:8080/v1/chat/completions \n  -H 'Content-Type: application/json' \n  -d '{"model":"local","messages":[{"role":"user",
       "content":"write a python lru cache"}],"max_tokens":128}'

# start the server
./llama-server -m models/coder-7b-q8_0.gguf --host 0.0.0.0 --port 8080

# in another window: ask it something and time the whole thing
time curl -s localhost:8080/v1/chat/completions \n  -H 'Content-Type: application/json' \n  -d '{"model":"local","messages":[{"role":"user",
       "content":"write a python lru cache"}],"max_tokens":128}'

# start the server
llama-server.exe -m models\coder-7b-q8_0.gguf --host 0.0.0.0 --port 8080

# PowerShell EATS the quotes in an inline JSON body. The server
# answers: parse error ... last read '{m'. Put the body in a file.
'{"model":"local","messages":[{"role":"user","content":"write a python lru cache"}],"max_tokens":128}' | Out-File req.json -Encoding ascii
Measure-Command { curl.exe -s -X POST localhost:8080/v1/chat/completions `
  -H "Content-Type: application/json" -d "@req.json" }

# start the server
./llama-server -m models/coder-7b-q8_0.gguf --host 0.0.0.0 --port 8080

# in another window: ask it something and time the whole thing
time curl -s localhost:8080/v1/chat/completions \n  -H 'Content-Type: application/json' \n  -d '{"model":"local","messages":[{"role":"user",
       "content":"write a python lru cache"}],"max_tokens":128}'

Write these three down now. They are your "before", and you will need them at 21:15 whether or not the rest of the hour goes to plan. words per second  ·  seconds until the first word  ·  memory used

Shrink it and re-time it.

Turn the model into a smaller version of itself, then run the exact same benchmark from step one. One change, one measurement. This is the biggest single jump you will see tonight.

Short on time or on disk: plenty of models are published already shrunk, so you can download a ready-made Q4_K_M and skip steps 1 and 2. You will get the speed either way. Doing the conversion yourself is what lets you do it to a model nobody has published, which is most of the interesting ones.

Re-time, and write the new numbers under the old ones. Expect the speed up and the memory down. Do not trust the speed on its own yet - whether it still writes working code is step five's job, and it is the step people skip.

Put a real server in front of it.

Same shrunken model, now behind an address your coding tool can talk to. Take the left column if you are on a laptop or one card, the right if you have a serious card or the Nebius credits.

Flag names on both of these move between releases, and a wrong flag fails loudly at startup rather than quietly. If one is rejected, --help on the binary you actually installed is the truth, not this slide and not a blog post from last year.

One knob. Re-time. Keep or revert.

Take these in order, stop when you run out of hardware, and after every single one re-run the benchmark and write the number next to the flag that caused it.

If a knob makes it slower, revert it and say so out loud. That is a useful result, it will happen to somebody here tonight, and the guess-ahead one is the usual culprit.

Fill in the card.

Run the same task we ran hosted at 19:55, against your own endpoint, timed the same way. Then run it once more and read what it actually wrote, because speed with broken output is not a result.

bars are illustrative · you fill in the numbers

Beating the paid one is not the win condition.

Getting close enough that you would rather keep your code on your own machine is. Say your number out loud when we go round, and say which knob got you there. Somebody with the same laptop is about to save forty minutes because you did.

Last thing, and it is the one that separates a real result from a fast-looking one: does the code it just wrote actually run? If it does not, go back up a shrink level and take the speed loss. A fast model that is confidently wrong is worse than the slow one you started with.

Five ways tonight lies to you.

Every one of these produces a number that looks like a win. Each card says what goes wrong and, more usefully, how you catch it in the room.

The pattern behind all five: a speed number on its own is not evidence. A reading is only real when you also know the answer was still correct and you changed one thing to get there.

What is in your bag at 22:00.

Not notes. Four things that keep working tomorrow on the machine you carried in.

Where to read next.

Commands from tonight, the scorecard, and whatever the room finds go into the Telegram after demos. That is also where you post your number, and it is worth doing, because somebody with your exact laptop is going to search it. t.me/+EBFzKXmJAVk5ZGU0

Who paid for the hour.

Both of these are in every ticket tonight, and both of them are doing actual work in the format rather than sitting on a logo wall.

If you want a night like this pointed at your product, the format is the same every time: your thing has to be the thing people build with for an hour, not a slide before the build. Come find the facilitator.