The Baseline Was Three Months Stale
I ran an overnight benchmark to see if a viral inference engine could beat my local DeepSeek setup. It couldn't. The bigger finding was that the number I was trying to beat was wrong.
A couple of weeks ago a project called llama-modchip started going around. It's a fork of llama.cpp with custom kernels and its own heavily compressed model files, and the demo video showed DeepSeek V4 Flash, a model that normally needs well over 100GB of memory, running at a usable speed on a laptop. I run that same model on a Dell PowerEdge server at home, and my notes said it generated about 1 token per second. Slow enough that I only use it for overnight batch jobs. If modchip could do 8 tok/s on a laptop, I wanted to know what it would do on a server.
So I had Claude Code run the test unattended on Friday night while I slept.
Writing the run, not doing it
Most of the work was in the prompt, not the benchmark. The server is a first-generation EPYC with 8-channel DDR4 and about 224GB of RAM in the VM. It powers on through Home Assistant and has a 5am automation that shuts it off to save power. Other jobs depend on it overnight. So the instructions were mostly about what not to touch: check the grid is up before pressing the power button, never disable the 5am shutdown, never modify the existing Ollama models, only one large model resident in memory at a time, new directories only, delete nothing. The repo also publishes "abliterated" model variants with the safety tuning stripped out, so that was an explicit never-download.
Then a timed plan with hard boxes. Test A ran modchip against the 144.5GB Q4 file I already had, a like-for-like comparison. Test B ran modchip on its own 57GB compressed file, the setup from the video. After that, the current Ollama setup on the same fixed prompt, temperature 0, 256 tokens. Every step had a time limit, and the report was due by 4:30am regardless of where things stood.
It finished at 12:55am and restored everything it had stopped.
What came back
Modchip lost, clearly. On my existing Q4 file it took almost nine minutes to load and then produced garbage on every variant: with and without KV cache quantization, through both API endpoints, with no errors in the log. Ollama runs the same file fine, so the file isn't the problem. My best guess is that modchip's DeepSeek path only works with its own model files.
On its own compressed file it worked. The output was coherent, it loaded in 40 seconds, and it used 58GB of RAM instead of 145GB. But it generated at 1.04 tok/s. The agent varied the thread count (12, 24, 40) and speed went up with every step, so on this CPU the custom kernels are compute-bound, not memory-bound. The README's laptop numbers came from a much newer chip. My first-generation EPYC lacks the AVX-512 support the fastest kernel builds target, which is the likely reason it landed nowhere near the README.
Then the Ollama baseline came in at 3.8 to 4.4 tok/s.
The number I was trying to beat
My notes said 1.0. The real number was roughly four times that. Somewhere between August and now, a flash-attention change and a small share of the model landing on the GPU had quietly quadrupled my generation speed, and I never re-measured. I'd been planning around a number that stopped being true months ago.
If the test had only run modchip, I would have seen 1.04 tok/s, compared it to my notes, and concluded it was a wash with a big memory win. Switching would have looked reasonable. I would have traded a 4x speed loss for RAM I don't actually need, based on a baseline nobody had checked since summer.
What saved that decision was one line in the plan: run the current setup on the same prompt, the same night, under the same conditions. I put it in mostly out of habit, so the numbers would be comparable. It ended up being the most useful result of the night.
The run also turned up a real problem I hadn't asked about. Ollama's first load attempt timed out at its five-minute limit on a cold start, and only succeeded once the file was cached. That same timeout could be hitting my overnight jobs any time the server cold-boots, and I wouldn't know unless I went looking.
What I'm taking from it
When you evaluate a new tool, the old setup is part of the experiment, not a constant. A benchmark from three months ago is a historical record, not a baseline. Systems drift: dependencies update, configs change, hardware gets shared differently. The thing you're comparing against has usually moved too.
This applies well past local LLMs. Every "should we switch vendors" or "is the new model better" decision I've been part of came with a number for the incumbent that someone measured once and never revisited. Re-measuring costs one more run. Not doing it means the comparison is against a version of your system that no longer exists.
I'm keeping Ollama. The 61GB download can go.