Benchmarking Apple M4 24GB (Mac mini) (Apple Silicon, 120 GB/s memory bandwidth, US MSRP $799 / UK £769) across Llama 3.3, DeepSeek-R1, Qwen 2.5, and custom models.
ollama run llama3.1:8b
Sizing an 8.03B Parameter LLM on 1x Apple M4 24GB (Mac mini) (24 GB)
| Quantization Format | Bitrate | Weights | KV Cache | Total VRAM | Throughput | Quality Retention | Compatibility |
|---|---|---|---|---|---|---|---|
|
FP16 / BF16
Full Precision
|
16.00 bpw | 16.06 GB | 1.00 GB | 18.56 GB | ~5.4 tok/s |
100% Lossless
+0.000 (Baseline)
|
100% GPU VRAM |
|
FP8 / Q8_0
Production Grade
GGUF / vLLM
|
8.50 bpw | 8.53 GB | 1.00 GB | 11.03 GB | ~10.1 tok/s |
99.9% Near-Lossless
+0.004 ppl
|
100% GPU VRAM |
|
Q6_K (6-bit)
High Precision
GGUF k-quant
|
6.56 bpw | 6.58 GB | 1.00 GB | 9.08 GB | ~13.1 tok/s |
99.5% Excellent
+0.021 ppl
|
100% GPU VRAM |
|
Q5_K_M (5-bit)
Balanced Plus
GGUF k-quant
|
5.69 bpw | 5.71 GB | 1.00 GB | 8.21 GB | ~15.1 tok/s |
99.0% Very High
+0.056 ppl
|
100% GPU VRAM |
|
Q4_K_M (4-bit)
Recommended
GGUF k-quant
|
4.85 bpw | 4.87 GB | 1.00 GB | 7.37 GB | ~17.7 tok/s |
98.2% Sweet Spot
+0.152 ppl
|
100% GPU VRAM |
|
AWQ / GPTQ 4-bit
High Throughput
vLLM / TensorRT
|
4.50 bpw | 4.52 GB | 1.00 GB | 7.02 GB | ~19.1 tok/s |
98.0% Fast Serving
+0.160 ppl
|
100% GPU VRAM |
|
IQ4_XS (i-Matrix)
GGUF i-quant
|
4.25 bpw | 4.27 GB | 1.00 GB | 6.77 GB | ~20.3 tok/s |
97.8% Compact 4-bit
+0.180 ppl
|
100% GPU VRAM |
|
EXL2 (4.0 bpw)
Max Speed
ExLlamaV2
|
4.00 bpw | 4.01 GB | 1.00 GB | 6.51 GB | ~21.5 tok/s |
97.6% Ultra Fast
+0.195 ppl
|
100% GPU VRAM |
|
Q3_K_M (3-bit)
GGUF k-quant
|
3.91 bpw | 3.92 GB | 1.00 GB | 6.42 GB | ~22.0 tok/s |
94.5% Noticeable Loss
+0.480 ppl
|
100% GPU VRAM |
|
IQ3_XXS (3-bit)
GGUF i-quant
|
3.06 bpw | 3.07 GB | 1.00 GB | 5.57 GB | ~28.1 tok/s |
92.0% Degraded
+0.710 ppl
|
100% GPU VRAM |
|
Q2_K (2-bit)
GGUF k-quant
|
2.63 bpw | 2.64 GB | 1.00 GB | 5.14 GB | ~32.7 tok/s |
85.0% Experimental
+1.650 ppl
|
100% GPU VRAM |
|
IQ1_S (1.58-bit)
GGUF Sub-2-bit
|
1.56 bpw | 1.57 GB | 1.00 GB | 4.07 GB | ~55.2 tok/s |
72.0% Heavy Loss
+3.800 ppl
|
100% GPU VRAM |
Side-by-side Q4_K_M memory utilization and token speed across NVIDIA, Apple Silicon, AMD Radeon, and Intel Arc.
Dedicated architecture profiles with pre-computed GQA KV cache tables. Supports unlimited dynamic URLs (e.g., /model/any-model-name-70b).
Meta Llama 3.3 70B Instruct delivers 405B-class reasoning and code synthesis in a dense 70.6-bi...
Meta Llama 3.1 8B is the industry-standard edge LLM for consumer GPUs, capable of running at 10...
Meta Llama 3.1 405B is one of the largest dense open-weights foundation models, engineered for ...
DeepSeek-R1 671B is a Mixture-of-Experts (MoE) reasoning model with 671B total parameters and 3...
DeepSeek-R1 Distill Qwen 32B packs frontier chain-of-thought math and coding capability into a ...
Distilled from DeepSeek-R1 onto the Llama 3.3 70B backbone, offering elite reasoning accuracy f...
Qwen 2.5 72B Instruct is a top-tier multilingual and coding foundation model with 80 transforme...
Qwen 2.5 Coder 32B is the premier open-weights code generation and refactoring model designed t...
Qwen 2.5 14B bridges the gap between 8B and 32B models, running effortlessly on 12GB and 16GB G...
Mistral Small 3 24B is engineered for low-latency local inference on 16GB and 24GB GPUs with on...
Mistral Large 2 123B is a flagship dense model ideal for 96GB+ Mac Studio configurations or dua...
Mixtral 8x7B uses a sparse Mixture-of-Experts architecture where 46.7B parameters reside in VRA...
Mixtral 8x22B combines 141B total weights with a fast 39B active parameter path per token....
Google Gemma 3 27B delivers exceptional performance-per-watt on single 24GB GPUs such as the RT...
Microsoft Phi-4 14B is trained on high-density synthetic textbook data, achieving STEM reasonin...
Cohere Command R+ 104B specializes in enterprise Retrieval-Augmented Generation (RAG), citation...
Our edge calculator models the exact memory allocation used by llama.cpp, Ollama, ExLlamaV2, and vLLM:
Weights_GB = (Params_B × Bits_Per_Weight) / 8
KV_GB = (2 × Layers × KV_Heads × Head_Dim × Context × (KV_Bits/8)) / 1024³
Speed = (Bandwidth_GBs × 0.72) / Active_Weights_GB
In full FP16/BF16 precision, an 8.03B Parameter LLM requires approximately 18.6 GB of VRAM including an 8,192-token KV cache and CUDA runtime buffer. Using 4-bit GGUF quantization (Q4_K_M at 4.85 bits per weight), total VRAM drops to 7.4 GB, while 8-bit quantization (Q8_0) requires 11.0 GB.
GGUF (used by llama.cpp, Ollama, and LM Studio) supports both pure GPU execution and hybrid CPU+GPU RAM offloading, making Q4_K_M the most versatile format. EXL2 (ExLlamaV2) runs strictly inside GPU VRAM and offers fine-grained bitrates with maximum single-user tokens/sec. AWQ (Activation-aware Weight Quantization) is optimized for high-concurrency production inference engines like vLLM and TensorRT-LLM.
Every token in your active context window stores Key and Value tensors across all transformer layers. With Grouped-Query Attention (GQA), KV cache memory equals 2 * Layers * KV_Heads * Head_Dim * Context_Tokens * Bytes_Per_Element. Quantizing the KV cache from FP16 (16-bit) to Q8_0 (8-bit) or Q4_0 (4-bit) cuts context memory overhead by 50% to 75%.
Autoregressive token generation (batch size 1) is memory-bandwidth bound. For each generated token, the GPU reads all active model weights from VRAM into Tensor Cores. Peak speed equals Memory Bandwidth (GB/s) divided by Active Model Size (GB), typically achieving 65% to 78% real-world controller efficiency.