ml
the careful experimenter
a model is only as good as its honest out-of-sample number. leakage and overfitting produce the best-looking scores, so every result is suspect until it beats a dumb baseline on data the model, the features, and the tuner never touched, split the way the model will meet production (by time, by group).
baseline firstvalidation that matches realitytracks every run
Use when building or evaluating machine learning models -- framing the problem, train/validation splits, baselines and model ladders, metrics from the decision, calibration and intervals, experiment tracking with MLflow, scikit-learn, LightGBM, Optuna, StatsForecast, and time-series forecasting.
methodology
- frame first: the decision, prediction time, target, horizon, unit of evaluation, cost of each error, and which data exist at prediction time.
- build the split before the model: stratified k-fold for iid data, group k-fold per entity, walk-forward with a gap at least the horizon; tune nested; keep one final holdout.
- baselines first (prior/majority, naive, seasonal naive, persistence), then the ladder linear/logistic -> random forest -> gradient boosting; adopt a bigger model only if it beats by more than fold noise.
- the metric comes from the decision: pr-auc/log loss/brier under imbalance, mae/mase against seasonal naive, ic/hit rate/net-of-cost sharpe; report the fold spread.
- calibrate on data the model did not fit; use conformal/quantile intervals and measure coverage.
- log params, data hash, seed, metrics, and the model with mlflow on sqlite; export-reload-assert identical; monitor drift in production.
- gradient boosting depth -> xgboost; feature and leakage questions -> feature-engineering.
contents
- trainedlearned layer -- read first
- common-pitfallsleakage and the standard pitfalls that fake good scores
- cross-validationk-fold, group k-fold, time-series split, nested cv
- grid-searchhyperparameter search inside a pipeline
- model-evaluationscoring, metrics, and what each one assumes
- calibrationprobability calibration
- classification-thresholdtuning the decision threshold
- model-persistenceexport, reload, and version pinning
- lightgbm-parameter-tuninglightgbm parameter tuning
- lightgbm-features-and-faqlightgbm features, advanced topics, faq
- optuna-faqoptuna faq
- optuna-first-tutorialoptuna first tutorial
- mlflow-trackingmlflow tracking
- mlflow-local-databasemlflow with a local sqlite backend
- mlflow-model-signaturesmlflow models and signatures
- statsforecast-cross-validationtime-series cross-validation
- conformal-predictionconformal prediction intervals
- tabular-classificationrunnable tabular classification example
- timeseries-forecastingrunnable time-series forecasting example
- reproducible-pipelinerunnable reproducible pipeline example
- mle-workflowend-to-end mle workflow
- scikit-learn-patternsscikit-learn patterns
- ml-pipelinepipeline construction
- xgboost-lightgbmgradient boosting patterns
- signal-classificationsignal classification patterns
- production-machine-learningproduction ml landscape
- time-series-and-python-ml-librariestime-series and python ml libraries
related gurus: xgboost, feature-engineering, olap, dbt.