Local AI does not have one memory requirement. The model weights, the conversation context, the runtime buffers, and the operating system all compete for space. That is why a model that appears to fit on a graphics card can still fail when you paste in a long document or open a second session.
The useful starting point is to separate the three terms buyers often mix together: VRAM is memory attached to a discrete GPU, RAM is system memory attached to the CPU, and a KV cache is a runtime store for attention data from the current context. NVIDIA’s CUDA programming guide describes CPU and GPU memory as distinct physical spaces, while Hugging Face’s cache explanation describes how key and value tensors are retained during autoregressive generation.
How much VRAM does a local AI model need?
Start with the weights. A rough lower-bound calculation is:
weight memory in bytes = parameter count × bytes per parameter
That is a planning calculation, not a promise about a particular download. Hypothetically, a 7-billion-parameter model stored at roughly 4 bits per weight would have about 3.5 GB of raw weight data before scales, metadata, temporary buffers, and the cache are added. On the same idealized basis, 8B is about 4 GB, 14B about 7 GB, 32B about 16 GB, and 70B about 35 GB. These figures use decimal parameter labels and the idealized 0.5 bytes per weight, so they are arithmetic examples for comparing scales, not measured model-file sizes or purchasing recommendations.
Quantization is what makes those smaller numbers possible. The llama.cpp project supports 1.5-bit through 8-bit integer quantization for reduced memory use. The actual model file is larger than the simple multiplication because quantizers store extra information needed to reconstruct groups of weights. The runtime also needs room for activations and scratch buffers. Leave headroom instead of choosing a model whose idealized weight total exactly equals the card’s advertised VRAM.
The measured model file is only one allocation. The context cache and runtime buffers also need space, and architecture, quantization type, context length, batch size, GPU offload settings, and inference engine all change the result. The correct check is to load the intended model with the intended settings and inspect the runtime’s weight, KV, and compute-buffer allocations. A parameter label by itself cannot establish a hardware requirement.
Why context length changes the answer
During generation, a transformer repeatedly needs the attention keys and values created by earlier tokens. A KV cache keeps those tensors so the runtime does not recompute them for every new token. Hugging Face describes the default dynamic cache as growing as generation adds more tokens, and notes that cache memory grows with sequence length even though caching reduces the repeated computation.
That makes “supports 128K context” a software and model limit, not a statement that a laptop can run 128K tokens comfortably. The cache has to be allocated somewhere, and its size depends on the model’s layer count, key and value head shape, cache data type, context length, and number of active sequences. The same model can fit at 4,096 tokens and fail at 32,768 tokens without changing its weights.
Concurrency multiplies the pressure. One 16,000-token conversation and four 4,000-token conversations can consume a similar total token budget if the server sizes a shared cache that way. llama.cpp exposes context, cache type, and parallel-slot controls because they are memory decisions as much as quality or latency decisions. Start with the context window your application actually needs, then increase it while watching the runtime’s reported allocations.
Long context also has a second cost: it can reduce throughput even when the allocation succeeds. More tokens must be attended to, and a large fixed cache can reserve memory that short requests never use. Hugging Face documents this tradeoff for static caches, which can improve compilation and latency consistency while consuming more memory than a dynamic cache for short sequences.
RAM versus VRAM for local AI
On a desktop with a discrete GPU, keeping the active model and its hot working data in VRAM avoids depending on transfers between memory domains. System RAM still matters for loading the model, memory-mapped files, CPU-side layers, the operating system, and applications running beside the inference server. If the runtime places model layers or cache data in host memory, it may continue to work, but the transfer path has different latency and bandwidth from local VRAM.
NVIDIA’s guide makes the important distinction directly: CPU and GPU have their own attached DRAM and ordinary allocations live in separate memory spaces. Peer access, mapped memory, and CUDA managed or unified-memory features can change how data is reached or moved, but the guide still says performance is best when data stays close to the processor using it. “It runs” and “it runs at a useful speed” are different outcomes.
Integrated graphics and some laptop systems change the accounting. They may use a shared or unified memory pool, so a machine advertised with 32 GB of RAM may let the GPU and model draw from that same pool. That can make a larger model fit than a discrete 8 GB card would, but the model is competing with the operating system and applications, and memory bandwidth is shared. Do not compare a unified-memory laptop’s total RAM directly with a discrete GPU’s VRAM as if the numbers described the same resource.
The practical buying rule is simple:
- Choose VRAM first when you want predictable speed from a discrete GPU.
- Choose system RAM generously when you expect CPU inference, model offload, large model files, or a unified-memory machine.
- Choose context and concurrency deliberately. A shorter context with one active user can be more usable than a headline context limit that exhausts the cache.
- Measure the model file and runtime allocation together. Do not size a machine from parameter count alone.
A repeatable sizing check
Before buying hardware, write down the model file size, target context length, number of simultaneous sequences, quantization format, and whether the runtime offloads the KV cache. Load the model with the intended settings and record the weight, KV, and compute-buffer allocations. Then leave room for the desktop, browser, drivers, and any retrieval or document-processing process that shares the machine.
Neon Control’s local AI hardware-fit tool is useful for a first estimate because it separates checkpoint parameters, bit depth, context reserve, and runtime overhead. It is not a benchmark and does not predict tokens per second, thermals, driver compatibility, or backend support. For the wider hardware picture, the site’s memory-bandwidth explainer covers why a model can be compute-capable yet starved for data, while the NPU versus GPU versus CPU guide explains which processor is doing the work.
The durable lesson is that local AI capacity is a budget, not a single specification. Weights set the entry cost. KV cache turns context and concurrency into live memory use. RAM provides operating room and can extend what a system loads, while VRAM determines how much of a discrete-GPU workload stays on the fast path. Size all three before you buy.
Photo: ElooKoN, CC BY-SA 4.0. Resized.
.jpg){kind=link}