AI/ML7 min read

MLOps

Operationalize ML models with reproducible training pipelines, experiment tracking, model registries, CI/CD, and production monitoring.

MLOpsML PipelineModel RegistryMLflowMonitoring

skill: MLOps category: AI/ML tags: [MLOps, ML Pipeline, Model Registry, MLflow, Monitoring] readingTime: 7

MLOps applies software engineering rigor to ML - without it, models trained on a laptop can't be reproduced, compared, or monitored in production. Start with experiment tracking (MLflow, W&B) before building full pipelines - it pays off immediately. Add CI/CD and monitoring as the model matures.

Workflow

Track experiments with MLflowWithout tracking, you lose hyperparameters, metrics, and artifacts from every training run. MLflow records all of it automatically.
Version datasets and modelsReproducing a model means having the exact same data, code, and hyperparameters. Dataset versioning is as important as code versioning.
Build a reproducible training pipelineA pipeline script ensures training is deterministic and runnable by anyone - not just the person who wrote it on their own machine.
Register and promote modelsA model registry tracks model versions, their training provenance, and promotion status (Staging → Production) - ensuring only validated models reach production.
Monitor model performance in productionModel performance degrades silently as data distribution shifts - production inputs diverge from training data over weeks or months.

Output Format

MLOps stack: MLflow tracking → DVC data versioning → reproducible training pipeline → model registry → drift monitoring.

mlops/
├── data/               # DVC-tracked datasets
├── src/
│   ├── train.py        # Reproducible training script
│   ├── evaluate.py
│   └── serve.py        # Model serving
├── dvc.yaml            # Pipeline DAG
├── params.yaml         # Hyperparameters
└── .github/workflows/
    └── train-and-evaluate.yml

Examples

Input

Output

Edge Cases & Gotchas

Non-determinism from GPU parallelism - CUDA operations are not fully deterministic even with fixed seeds unless you set torch.backends.cudnn.deterministic = True and benchmark = False. This slows training by ~10%.
Training/serving skew - The model receives data in a different format or scale at serving time than training. Log preprocessing code alongside the model artifact.
Model size vs. serving latency - A 1GB model loaded fresh on each request takes 500ms to load. Serve models from memory with a persistent server (FastAPI + uvicorn, Triton) rather than reloading per request.
Retraining triggers - Retraining on a fixed schedule is wasteful if drift hasn't occurred. Trigger retraining on drift metrics or when production accuracy drops below a threshold.

When to Break These Rules

Use managed ML platforms (SageMaker, Vertex AI, Azure ML) instead of self-hosted MLflow when you don't want to manage infrastructure and your team is small.
Skip full MLOps if you're doing a one-off analysis or prototype - invest in it when models go to production and need to be updated or monitored.

Reference Files

https://mlflow.org/docs/latest/
https://dvc.org/doc