Best local LLM for 8 GB
For RTX 4060 (8 GB), RTX 3080 (10 GB) and Mac, 16 GB. Method
Behind the frontier23 monthsFirst reached by o1, Dec 2024
Models that fit in 8 GB
Top 7#ModelIndexFormatMemoryCheapest APILowest output price per 1M.
By memory
Index, highest firstMemorySmartest fitIndexFormatBehindAPI equivalent
Method
- Models: ranked models with open weights, an official Hugging Face repository and medium or high confidence. One line per set of weights.
- Memory need: the weights at the format's bits per weight (4-bit 4.85, 8-bit 8.5, 16-bit 16), plus 5% and 1 GB for a short context.
- Format: the most bits that fit. The index is for the full model; a 4-bit copy can score a little lower.
- Shared memory (Mac, DGX Spark, Ryzen AI Max): about two thirds goes to the model up to 36 GB, three quarters above.
- Behind the frontier: months from the first release with the same index or more, among models with medium or high confidence.
- Same by API: the lowest blended price (3 input : 1 output) for that index or more, per 1M tokens, among the same models.