Llama-2 7B Q4_0 (reference benchmark)
llama.cpp llama-bench, Llama-2 7B Q4_0, tg128 (text generation tok/s); best of FA on/off as posted. Fits every device.
llama.cpp llama-bench, Llama-2 7B Q4_0, tg128 (text generation tok/s); best of FA on/off as posted. Fits every device.