Running an AI model locally versus through a cloud service changes where the weights and computation live, which in turn changes privacy, latency, hardware limits, cost structure, and model choice.

The first hard constraint is memory. A rough raw-weight calculation is:

• 8 billion parameters × 16 bits ≈ 16 GB just for weights;
• the same 8B model at 4-bit ≈ 4 GB raw weights;
• 70B at 16-bit ≈ 140 GB;
• 70B at 4-bit ≈ 35 GB raw weights.

Real inference needs additional memory for runtime buffers and the KV cache, which grows with context length and can be substantial. Quantization therefore makes models fit that otherwise would not, but it may trade some quality or throughput depending on method/hardware.

Local strengths

• prompts and model inference can stay on your machine;
• no network round trip once the model is installed;
• fixed hardware can make repeated use cheap;
• you control model/version and can work offline.

Cloud strengths

• access to much larger accelerators and models;
• easier scaling and updates;
• no need to own/manage high-end hardware;
• service can provide tools, retrieval, safety systems, and optimized serving that are not contained in the raw model alone.

“Local is private” also needs a boundary: local inference reduces exposure to a model provider, but the surrounding OS, apps, browser, telemetry, and downloaded model provenance still matter.