Quick summary: Generative-AI workloads depend mainly on GPU and VRAM; there are three common configuration tiers — entry (single mid-range GPU), production (data-centre GPUs) and cluster (multiple servers) — choose based on real needs, not flashy specs.

When you decide to deploy Local AI, the first technical question is always: what kind of server do we need? Unlike typical enterprise applications that lean on the CPU, generative-AI workloads depend mainly on the GPU and its graphics memory (VRAM). Grasping a few fundamentals helps you avoid both waste and shortfall.

The most important factor is VRAM — the memory on the GPU. This is the constraint that decides how large a model you can run. A rough rule: a model needs memory roughly equal to its parameter count times bytes per parameter, plus room for context. Thanks to quantisation (reducing the precision of weights), a 7–8 billion parameter model runs comfortably on a 16–24GB GPU, while a 70 billion parameter model usually needs multiple GPUs or a high-capacity one.

Configurations fall into three common tiers. An entry configuration uses a single mid-range GPU (for example a 24GB card), suitable for a department-scale internal knowledge assistant serving tens to hundreds of users. A production configuration uses a dedicated server with one or more data-centre GPUs (L40S, A100, H100), serving the whole enterprise and many concurrent tasks. A cluster configuration links several GPU servers for large organisations needing high availability and very large models.

Beyond the GPU, do not overlook the surrounding components: system RAM should be ample for loading and pre-processing data; fast NVMe storage speeds up loading models and the vector database; internal network bandwidth matters when linking multiple GPUs; and cooling and power supply must be adequate, because AI GPUs draw significant power.

Another strategic decision is buy versus rent. Many companies start on private-cloud GPU servers to experiment, then move to on-premise hardware once volume stabilises — the moment when ownership cost becomes far more attractive than renting.

The overarching principle: choose the configuration for the real problem, not for flashy specs. Start just big enough for today's need, design for growth, and prioritise the balance of performance, cost and operability. DABO works with customers from the reference-configuration stage so the hardware investment is right from day one.

Frequently asked questions

How much VRAM is enough? A 7-8 billion parameter model runs comfortably on a 16-24GB GPU thanks to quantisation; a 70 billion parameter model needs multiple GPUs or a high-capacity one.

What scale suits the entry configuration? A single mid-range GPU (24GB) suits a department-scale internal knowledge assistant with tens to hundreds of users.

What else matters besides the GPU? Ample system RAM, fast NVMe storage, internal network bandwidth, and adequate cooling and power supply.