Cloud GPU Pricing: Where to Train and Inference Without Going Broke

You need to train a 7B parameter model. Or run inference at scale. Or fine-tune on your own data.

Your laptop isn't cutting it. You need GPUs. Real GPUs. The kind that cost more than a car.

But where do you get them? AWS? GCP? Some random website with "vast" in the name? The prices vary by 10x. The reliability varies even more. And if you pick wrong, you'll spend a month debugging why your training run keeps failing instead of actually training.

This episode is your field guide to cloud GPU providers. What they cost. What they're good for. And where the hidden traps are.

The Big Three: AWS, GCP, Azure

AWS (EC2 + SageMaker)

The safe choice. The expensive choice. The "nobody ever got fired for choosing AWS" choice.

Instance Type GPU VRAM On-Demand Spot

AWS Pros:

Reliability is top-tier

SageMaker makes training jobs easy

Spot instances are actually available (unlike GCP)

Integration with the rest of AWS ecosystem

AWS Cons:

Most expensive option

GPU availability can be spotty for A100/H100

Complex pricing (egress, storage, IP addresses all add up)

Google Cloud Platform (GCP)

The ML-native choice. TensorFlow was built here. TPUs live here. But their GPU availability is... inconsistent.

Instance Type GPU VRAM On-Demand Spot

GCP Pros:

Best TPU support if you're in that ecosystem

Good integration with Vertex AI

Preemptible pricing can be very cheap

GCP Cons:

GPU availability is the worst of the big three

Spot/preemptible instances get reclaimed constantly

Support can be... Google-y (documentation > humans)

Azure

The enterprise choice. The "we already pay for Microsoft" choice. Actually quite good for AI workloads.

Instance Type GPU VRAM On-Demand Spot

NC4as T4 v3 T4 16 GB $0.526/hr ~$0.08/hr

NC24ads A100 v4 A100 80 GB $3.60/hr ~$1.08/hr

ND96asr A100 v4 A100 × 8 640 GB $36.29/hr ~$10.89/hr

ND96isr H100 v5 H100 × 8 640 GB ~$90/hr ~$27/hr

Azure Pros:

OpenAI partnership means easy GPT-4 access

Good availability for A100s

Spot pricing is aggressive

Best for Windows/enterprise workflows

Azure Cons:

Portal UX is... not great

Documentation can be fragmented

Less community content than AWS

The Challengers: Lambda Labs, RunPod, Vast.ai

These providers rent GPUs cheaper than the big three. Much cheaper. But with tradeoffs.

Lambda Labs

The "researcher-friendly" alternative. Simple pricing, good support, focused on ML workloads.

GPU VRAM Price/hr Notes

RTX A6000 48 GB $0.80 Great for inference

Lambda Pros:

2-3x cheaper than AWS for same GPUs

Persistent storage (your data stays between sessions)

Jupyter notebooks built-in

Good customer support (actual humans)

Lambda Cons:

Smaller ecosystem than AWS/GCP

Occasional availability issues

Fewer regions (mainly US)

RunPod

The "serverless GPU" platform. Pay by the second. Good for bursty workloads.

GPU VRAM Community Cloud Secure Cloud

RTX 4090 24 GB $0.44/hr $0.69/hr

RTX A6000 48 GB $0.80/hr $1.19/hr

RunPod Pros:

Cheapest option for many GPUs

Serverless endpoints (scale to zero)

Community cloud is very cheap (but less reliable)

Good for inference endpoints

RunPod Cons:

Community cloud can be unreliable (random failures)

Network storage costs add up

Support is community-based for lower tiers

Vast.ai

The "eBay of GPUs." Individuals rent out their personal GPUs. Cheapest option, highest variance.

GPU VRAM Price Range

RTX 3090 24 GB $0.20 - $0.50/hr

RTX 4090 24 GB $0.40 - $0.80/hr

RTX A6000 48 GB $0.60 - $1.20/hr

Vast.ai Pros:

Cheapest GPUs anywhere

Massive selection

Can find deals 5x cheaper than AWS

Vast.ai Cons:

Reliability is a coin flip

No support (you're renting from random people)

Data security concerns (who owns that machine?)

Machines can disappear mid-training

Setup is manual (SSH, configure yourself)

When to use Vast.ai:

Experiments and prototyping

When cost matters more than reliability

When you can checkpoint frequently

Never for production or sensitive data

🔧

Spot vs On-Demand: The 70% Discount

Spot instances (AWS) / Preemptible (GCP) / Spot (Azure) are excess capacity sold at discount. The catch: they can be reclaimed with 2 minutes notice.

Provider Discount Interruption Rate Best For

AWS Spot Up to 90% 5-10% Training with checkpointing

GCP Preemptible Up to 80% 10-20% Fault-tolerant batch jobs

Spot Strategy for Training

python

Pseudo-code for spot-aware training

import signal

import sys

checkpoint_path = "/workspace/checkpoint.pt"
def save_checkpoint():
torch.save({
        'epoch': epoch,
        'model_state_dict': model.state_dict(),
        'optimizer_state_dict': optimizer.state_dict(),
    }, checkpoint_path)
    print(f"Checkpoint saved to {checkpoint_path}")

def handle_sigterm(signum, frame):

"""AWS/GCP send SIGTERM before reclaiming spot instance"""
    print("Spot interruption detected! Saving checkpoint...")
    save_checkpoint()
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_sigterm)

Also checkpoint regularly

for epoch in range(num_epochs):

train_epoch()
    if epoch % 5 == 0:
        save_checkpoint()

Rule of thumb: If your training job can checkpoint and resume, use spot. If it can't, pay for on-demand.

📊

Cost Comparison: Training a 7B Model

Let's say you're fine-tuning Llama 3.1 8B on a custom dataset. You need an A100 80GB.

The lesson: Spot instances and alternative providers can cut your training costs by 70%. For a month-long training run, that's the difference between $3,600 and $1,000.

Hidden Costs

The hourly GPU price is just the start. Watch out for:

Cost AWS GCP Lambda RunPod

Example: Training a model that generates 100GB of checkpoints weekly:

AWS storage: $10/month

Downloading those checkpoints: $9 × 4 = $36/month

Suddenly your "cheap" spot instance has $46 in extra costs

Choosing a Provider: Decision Matrix

What's your priority?

│

├── Reliability is critical (production)

│ ├── Need managed ML platform → AWS SageMaker or GCP Vertex

│ └── Just need reliable GPUs → AWS On-Demand or Azure

│

├── Cost is critical (research, experiments)

│ ├── Can handle interruptions → AWS/GCP Spot

│ ├── Need persistence → Lambda Labs

│ ├── Need serverless scaling → RunPod

│ └── Maximum cheap, can tolerate risk → Vast.ai

│

├── Training large models (70B+)

│ └── Need multi-node → AWS p4d/p5 or Lambda 8xH100

│

└── Inference at scale

├── Steady load → Lambda or RunPod dedicated
    └── Bursty load → RunPod serverless or AWS SageMaker endpoints

🔬

Practical Takeaways

Spot instances save 70% — if your workload can handle interruptions, always use spot

Lambda Labs is the sweet spot — 2-3x cheaper than AWS with better reliability than Vast

Vast.ai for experiments only — cheapest prices, but don't run production there

Watch hidden costs — egress and storage can add 20-30% to your bill

Multi-node training = AWS or Lambda — few providers offer reliable 8xGPU instances

Start with on-demand, move to spot — prove your code works, then optimize for cost

🛡️

What's Next?

Episode 69: TPUs vs GPUs — Google bet on TPUs. NVIDIA owns GPUs. Which is better for your workload? We'll dive into systolic arrays, XLA compilation, JAX vs PyTorch, and when Google's custom chips actually win.

← Previous

Ep 67: Apple Silicon for AI

Next →

Ep 69: TPUs vs GPUs

Next: Episode 69 — TPUs vs GPUs

NVIDIA GPUs power virtually every AI model you've heard of. So why did Google build its own chip? Here's the TPU story.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.