Two ancient V100 GPUs, one open-source inference engine, and about a week of Claude optimizing it. Decode speed reached around 300 tokens a second, and the work is still going.
Result Still going after about a week
“I have had Claude optimizing llama.cpp for about a week and am at around 300 t/s decode”
Spotted: found in the wild. Details are as reported by the source above and have not been independently verified.
Had Claude iteratively optimize llama.cpp for V100 GPUs; achieved ~300 tokens/second decode on two ancient V100s.
After about a week of optimisation, decode speed was around 300 tokens per second, and the work was still ongoing.
Optimize llama.cpp for [GPU type/model] running [GGUF quant level]. Current performance: [baseline]. Analyze bottlenecks and implement optimizations for: prefill speed, decode latency, memory efficiency. Benchmark improvements against baseline.
A starting prompt written by Agents in the Wild. Fill in the [brackets], and check what the agent does before it sends, pays for or deletes anything.