AI/ML7 min read

Scikit-learn & ML

Build and evaluate machine learning models with scikit-learn - pipelines, cross-validation, feature engineering, hyperparameter tuning, and model selection.

Scikit-learnMachine LearningPythonClassificationRegression

skill: Scikit-learn & ML category: AI/ML tags: [Scikit-learn, Machine Learning, Python, Classification, Regression] readingTime: 7

Most ML problems are solved better with a well-tuned gradient boosting model (XGBoost, LightGBM) than with a neural network. Start simple (linear regression, logistic regression, decision tree) and only increase complexity when the simple model is clearly insufficient. Understand your metrics before picking a model.

Workflow

Split data and establish a baselineAlways establish a baseline before building models - a DummyClassifier that predicts the majority class is your floor. Beat it first.
Build preprocessing pipelinesPipelines prevent data leakage - preprocessing is fit on training data only and applied to test data at inference time.
Evaluate with cross-validationA single train/test split gives high-variance accuracy estimates. Cross-validation averages performance across K splits.
Tune hyperparametersDefault hyperparameters are rarely optimal. Grid search or random search finds better configurations systematically.
Interpret and deploy modelsA model that can't be explained can't be trusted in production. Feature importance and SHAP values reveal what drives predictions.

Output Format

ML pipeline: train/test split, preprocessing Pipeline, cross-validated evaluation, RandomizedSearchCV tuning, SHAP explanations, joblib export.

Pipeline([
  ('preprocessor', ColumnTransformer([...])),
  ('model', XGBClassifier()),
])
cross_val_score(pipeline, X, y, cv=5, scoring='roc_auc')
RandomizedSearchCV → best model
joblib.dump(best_model, 'model.pkl')

Examples

Input

Output

Edge Cases & Gotchas

Class imbalance - Accuracy is misleading on imbalanced datasets (99% negative → 99% accuracy by always predicting negative). Use AUC-ROC, F1, or set class_weight='balanced'.
Feature leakage - Features that encode the target (e.g., "was_refunded" to predict churn) make the model look amazing on test data but useless in production.
Overfitting on test set - Running GridSearch on the test set overfits hyperparameters to it. Always use a held-out final test set seen once - use cross-validation for tuning.
Model serialization drift - A model saved with scikit-learn 1.3 may not load with 1.5. Pin the exact sklearn version in your production Docker image.

When to Break These Rules

Use XGBoost or LightGBM instead of scikit-learn's GradientBoostingClassifier for tabular data - they're 10-50x faster with better defaults.
Use PyTorch or TensorFlow for deep learning tasks (images, text, sequences) - scikit-learn's neural net is a toy by comparison.

Reference Files

https://scikit-learn.org/stable/user_guide.html
https://shap.readthedocs.io/