Limited VRAM does not mean experiments must be trivial. It means the training setup must be designed deliberately. Every choice affects memory, speed, numerical stability and the size of model that can be tested.

Chrome GPU styled as a fashion accessory
FIG. 01 — The GPU is serving looks and approximately twelve gigabytes of VRAM.

Mixed Precision

Mixed-precision training stores and computes many values using lower-precision formats while keeping selected operations in higher precision. This can reduce memory use and increase throughput, particularly on modern GPUs.

with torch.autocast(device_type="cuda", dtype=torch.float16): logits = model(input_ids) loss = criterion(logits, targets) scaler.scale(loss).backward() scaler.step(optimizer) scaler.update()

Gradient Accumulation

When a desired batch does not fit in memory, several smaller micro-batches can contribute to one optimizer update. This approximates a larger batch without storing the full batch at once.

optimizer.zero_grad() for step, batch in enumerate(loader): loss = model_step(batch) / accumulation_steps loss.backward() if (step + 1) % accumulation_steps == 0: optimizer.step() optimizer.zero_grad()

Activation Checkpointing

Checkpointing saves memory by discarding selected intermediate activations during the forward pass and recomputing them during backward propagation. Memory consumption decreases, but the extra computation makes training slower.

TechniqueMain benefitMain cost
Mixed precisionLower memory, higher speedNumerical care
Gradient accumulationLarger effective batchMore steps per update
CheckpointingLower activation memoryExtra compute
Shorter contextLarge memory reductionLess long-range information
On a small GPU, context length can be more expensive than glamour, and glamour is already expensive.

What I Will Measure

  • Peak VRAM use
  • Tokens processed per second
  • Training loss per unit of compute
  • Wall-clock time
  • Validation performance

Practical Order of Operations

I will first establish a stable baseline. Then I will enable mixed precision, measure the difference, add accumulation only when needed, and use checkpointing after verifying that the speed tradeoff is worthwhile.