Sizing GPU Infrastructure and Cloud Compute for Enterprise AI
A practical engineering guide to calculating compute requirements, memory bandwidth, and total cost of ownership for self-hosted LLMs, fine-tuning workloads, and inference microservices.
- Size GPU VRAM around total parameter weights plus dynamic key-value attention buffers
- Differentiate memory bandwidth bound inference from compute bound training workloads
- Evaluate hyperscaler elasticity against reserved colocation pricing over eighteen months
- Adopt continuous batching runtimes such as vLLM or Triton for high token throughput
- Implement scale-to-zero microservices and spot instance pools for off-peak workloads
Understanding the memory wall and KV cache sizing
When engineering teams estimate infrastructure requirements for large language models, they often focus solely on model parameter counts. For example, a 70-billion parameter model in 16-bit precision requires roughly 140 gigabytes of memory just to hold its weights in GPU memory. Assuming that an inference service only requires 140 gigabytes of VRAM leads directly to out-of-memory crashes under real production workloads.
The hidden memory consumer is the Key-Value (KV) attention cache. As input context length and concurrent user concurrency increase, the attention vectors generated for every token in flight must stay in high-bandwidth memory. For long-context document analysis or agentic loops processing 32,000 tokens across sixteen concurrent streams, the KV cache alone can exceed 60 gigabytes of VRAM. Sizing formulas must include model weights, KV cache allocations, runtime execution buffers, and quantization overheads (such as FP8 or AWQ 4-bit) before provisioning hardware clusters.
Inference versus fine-tuning compute requirements
Inference and model training stress hardware architectures in completely different ways. Generative inference is fundamentally memory bandwidth bound during token generation: the accelerator must sweep through gigabytes of weights to produce a single new token. Memory bandwidth, measured in terabytes per second, dictates the generation speed per client.
In contrast, parameter-efficient fine-tuning (PEFT, LoRA) and full pre-training workloads are compute bound and require extensive gradient backpropagation state. Training workloads consume between three to four times more memory per parameter than inference, requiring optimizer states (such as AdamW 32-bit parameters) and activation checkpoints. Understanding this distinction prevents organizations from provisioning expensive multi-node training clusters when their true daily requirement is low-latency, bandwidth-optimized inference.
Cloud instances versus dedicated bare metal hosting
Enterprise technology leaders face an ongoing choice between hyperscaler cloud instances (AWS, Google Cloud, Microsoft Azure) and specialized GPU bare-metal colocation providers. Hyperscalers offer tight integration with managed identity systems, virtual private clouds, and object storage, but their per-hour GPU list prices carry substantial premiums.
For experimental workloads, bursty prototypes, and seasonal applications, hyperscalers provide invaluable elasticity. However, when an enterprise operates dedicated AI inference workloads with steady 24x7 utilization exceeding 70 percent, hyperscaler instance costs become punitive. In steady-state operational profiles, leasing dedicated bare-metal GPU nodes or committing to multi-year reservations reduces hardware expenditure by 40 to 60 percent, provided the internal operations team possesses container orchestration expertise.
Containerized model serving frameworks and runtime tuning
Serving large models through generic Python web frameworks like Flask or FastAPI results in poor hardware saturation and serial request queuing. High-performance model deployment relies on purpose-built inference runtimes such as vLLM, TensorRT-LLM, or NVIDIA Triton Inference Server.
These engines introduce PagedAttention, which allocates non-contiguous physical memory for the KV cache similar to virtual memory paging in operating systems. This technique virtually eliminates memory fragmentation and unlocks continuous batching, where incoming requests are dynamically inserted into active GPU iteration passes without waiting for prior generation sequences to finish. Proper engine selection triples throughput per dollar compared to naive container wrappers.
Cost governance, autoscaling, and spot instances
Without rigorous cost governance, GPU infrastructure clusters quickly become significant balance sheet liabilities. Idle GPU nodes draw substantial electrical power and invoice hundreds of dollars per day while serving minimal user traffic. Cost control begins with declarative autoscaling based on queue depth rather than traditional CPU utilization metrics.
Configure inference pods to scale down to warm standby nodes or scale to zero during non-business hours if workloads are internal-facing. For asynchronous background tasks such as batch vector embedding, offline transcription, and daily summary generation, utilize spot or preemptible instances. By engineering resilient task retry queues, batch workloads can run on surplus compute at discounts of up to 70 percent without jeopardizing customer-facing SLAs.
Further reading
Primary references behind the technical guidance in this article.
Put the thinking into practice.
Explore our cloud and AI infrastructure practice, GPU sizing benchmarks, and dedicated private LLM architectures.
Explore cloud AI infrastructure
