Quick Introduction We have all been on forums, chats, reddit, discord, youtube, or somewhere and heard “Oh! Model XYZ is AMAZEBALLZ!zomgwtfbbq” then downloaded it (or more likely, some quantized form of it) and said “eww… This sucks!” This post is going to be a rather technical series of experiments to demonstrate the impact of implementation-specific hazards with inference. I will be using the term “reference implementation” to describe the lab that published and offers first-party hosting of ...
This is true. But VLLM’s quantization support (as outlined in the article) is quite primitive.
Thats not what its built for. Its made for maximum speed with many concurrent users, not cramming LLMs into a desktop rig.
If you are running an LLM for personal use, you are usually better off with a backend optimized for higher fidelity quantization, like MLX DWQs, exl3, ik llama GGUFs and stuff like that.
If you want raw speed on a desktop, use one of the ninfer project branches.