Q4_K_M

4-bit quantization using the K-quant method, medium variant. The most popular general-purpose quantization — good balance of quality, speed, and memory use. The “K” indicates a more sophisticated mixed-precision approach vs. plain Q4_0.

• llama-quant.cpp