Feature Stores: The Data Layer Your Models Are Missing

Your recommendation model needs to know: what did this user buy in the last 7 days? What's their average order value? How many times did they visit the product page?

Your fraud detection model needs to know: how many transactions has this card processed in the last hour? What's the average transaction amount? Is this merchant in a high-risk category?

These aren't prompts. These aren't RAG chunks. These are features — computed values that feed directly into ML models at prediction time. And if you're computing them on the fly for every request, you're doing it wrong.

Feature stores are specialized data systems that compute, store, and serve features to ML models — consistently, at scale, and without duplicating work across teams.

What Is a Feature, Actually?

A feature is a measurable property that a model uses to make predictions. It's the bridge between raw data and model input.

Raw data: user_123 bought item_456 at 2024-03-15 14:30:00 for ₹2,500

Features derived from this:

user_total_purchases_7d = 12
user_avg_order_value = ₹1,800
user_purchase_frequency = 3.2/week
item_category_preference = "electronics" (0.7), "clothing" (0.2)
time_since_last_purchase = 2.3 hours
is_weekend = true

The model doesn't see the raw transaction. It sees these numbers. Feature engineering — creating the right features — is often the difference between a model that works and one that doesn't.

The Problem Feature Stores Solve

Without a feature store, every team computes features independently:

Team A (Recommendations):

• Wrote SQL to compute user_avg_order_value
• Runs it in a Spark job, stores in Redis
• Uses Python to compute during training

Team B (Fraud Detection):

• ALSO needs user_avg_order_value
• Wrote DIFFERENT SQL (slightly different logic!)
• Runs it in a separate pipeline, stores in DynamoDB
• Uses Pandas during training

Team C (Search Ranking):

• Needs the same feature AGAIN
• Wrote yet another version
• It disagrees with Team A and Team B by 0.3%

Three teams, three different values for the same feature. This is called training-serving skew — the features used during training don't exactly match the features used during serving. It's the #1 silent killer of ML performance in production.

A feature store solves this by providing one definition, one computation, one source of truth:

Feature Store:

feature: user_avg_order_value

definition: AVG(amount) WHERE timestamp > NOW() - 7 days

online: Redis (low-latency serving)

offline: Parquet in S3 (batch training)

→ Team A reads from it ✓

→ Team B reads from it ✓

→ Team C reads from it ✓

→ Training and serving use IDENTICAL values ✓

🔧

Online vs Offline Feature Stores

This is the key architectural concept. Models need features in two very different contexts:

Offline Store Online Store

When Training time Inference time

Offline store is where you keep historical feature values for training. When training a model, you need to know "what was user_avg_order_value for user_123 at the exact moment this transaction happened?" — not what it is NOW, but what it WAS THEN. This is called point-in-time correctness and it's surprisingly hard to get right.

Online store is where you keep the LATEST feature values for real-time inference. When a user hits your API, you need their current features in milliseconds.

Training (offline):

"What was user_123's avg order value on March 15th?"

→ Query offline store → ₹1,800 (historical)

Inference (online):

"What is user_123's avg order value RIGHT NOW?"

→ Query online store → ₹2,100 (current)

📊

Feast: The Open-Source Feature Store

Feast (Feature Store) is the most popular open-source option. Let's build a real example.

Define your features:

python

feature_repo/features.py

from feast import Entity, Feature, FeatureView, FileSource

from feast.types import Float32, Int64

from datetime import timedelta

Entity: the thing we're computing features for

user = Entity(
name="user_id",
    description="Customer ID"

)

Data source: where raw features come from

user_features_source = FileSource(
path="s3://features/user_features.parquet",
    timestamp_field="event_timestamp"

)

Feature view: defines which features exist

user_features = FeatureView(
name="user_features",
    entities=[user],
    ttl=timedelta(days=7),  # Features expire after 7 days
    schema=[
        Feature(name="avg_order_value", dtype=Float32),
        Feature(name="total_purchases_7d", dtype=Int64),
        Feature(name="purchase_frequency", dtype=Float32),
        Feature(name="days_since_first_purchase", dtype=Int64),
    ],
    source=user_features_source,

)

Get features for training (offline):

python

from feast import FeatureStore

store = FeatureStore(repo_path="feature_repo/")

Training data with point-in-time correct features

training_df = store.get_historical_features(
entity_df=training_entities,  # DataFrame with user_id + timestamp
    features=[
        "user_features:avg_order_value",
        "user_features:total_purchases_7d",
        "user_features:purchase_frequency",
    ],

).to_df()

Result: each row has features AS THEY WERE at that timestamp

No data leakage. No future information contamination.

Get features for inference (online):

python

Materialize latest features to online store

store.materialize_incremental(end_date=datetime.now())

Real-time feature lookup (< 10ms)

features = store.get_online_features(
features=[
        "user_features:avg_order_value",
        "user_features:total_purchases_7d",
    ],
    entity_rows=[{"user_id": "user_123"}],

).to_dict()

{'user_id': ['user_123'], 'avg_order_value': [2100.0], 'total_purchases_7d': [12]}

The Feast architecture:

┌──────────────────┐
                    │   Feature Repo   │  (feature definitions in code)
                    │   (Python/YAML)  │
                    └────────┬─────────┘
                             │
              ┌──────────────┼──────────────┐
              ▼              ▼              ▼
     ┌────────────┐  ┌────────────┐  ┌────────────┐
     │  Offline   │  │  Online    │  │  Registry   │
     │  Store     │  │  Store     │  │  (metadata) │
     │  (S3/BQ)   │  │  (Redis)   │  │             │
     └────────────┘  └────────────┘  └────────────┘
           │                │
    Training data     Inference data
    (batch, slow)     (real-time, fast)

Feature Engineering at Scale

Computing features is where the real engineering happens. Here are the patterns:

Batch Features (Updated hourly/daily)

sql

-- Run in Spark/BigQuery on a schedule

SELECT

user_id,
    AVG(amount) as avg_order_value,
    COUNT(*) as total_purchases_7d,
    COUNT(*) / 7.0 as purchase_frequency

FROM transactions

WHERE timestamp > CURRENT_TIMESTAMP - INTERVAL 7 DAYS

GROUP BY user_id

Good for: features that don't need to be up-to-the-second fresh. Most features are batch.

Streaming Features (Updated in real-time)

python

Flink/Kafka Streams job

class TransactionProcessor:

def process(self, event):
        user_id = event['user_id']

# Update running aggregates in real-time
        self.state[user_id]['transaction_count_1h'] += 1
        self.state[user_id]['total_amount_1h'] += event['amount']
        self.state[user_id]['avg_amount_1h'] = (
            self.state[user_id]['total_amount_1h'] /
            self.state[user_id]['transaction_count_1h']
        )

# Push to online store immediately
        feature_store.push(user_id, self.state[user_id])

Good for: fraud detection, real-time recommendations, anything where "5 minutes ago" matters.

On-Demand Features (Computed at request time)

python

Computed during inference, not stored

def compute_on_demand_features(request):

return {
        "is_weekend": request.timestamp.weekday() >= 5,
        "hour_of_day": request.timestamp.hour,
        "time_since_last_interaction": (
            datetime.now() - get_last_interaction(request.user_id)
        ).seconds
    }

Good for: features that depend on the exact request context and can be computed instantly.

The Enterprise Landscape

Beyond Feast, here's what companies use:

Solution Type Best For

Feast Open-source Startups, mid-size teams

Honest take: If you're a small team with < 50 features, you might not need a feature store yet. A Redis cache + a Spark job gets you far. Feature stores shine when you have multiple teams, hundreds of features, and the training-serving skew is costing you real model quality.

🔬

Practical Takeaways

Features are the bridge between raw data and model input — good features > bigger models

Training-serving skew is the silent killer — feature stores eliminate it by providing one source of truth

Online vs offline stores serve different needs — offline for training (batch), online for inference (real-time)

Point-in-time correctness prevents data leakage — training features must reflect what was known THEN, not NOW

Start with batch features — streaming features add complexity, only add them when freshness matters

Feast is the open-source standard — start there, graduate to Tecton or managed services when you scale

🛡️

What's Next?

Episode 88: Data Pipelines for AI — Features need data. Data needs pipelines. We'll explore ingestion, cleaning, chunking at scale, Apache Airflow for orchestration, and streaming pipelines that feed your feature stores and training jobs in real-time.

← Previous

Ep 86: CI/CD for AI

Next →

Ep 88: Data Pipelines for AI

Next: Episode 88 — Data Pipelines for AI

Every AI system is only as good as its data. The best model with a garbage data pipeline produces garbage.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.