ml trained

Leakage and overfitting both show up as a better score. The splitter decides what "out of sample" means, so choose it before choosing the model. See the official common pitfalls page and the cross-validation page.

Three worked examples, each writing only to the working directory:

• tabular-classification: model ladder, PR-AUC trap, and calibration.
• timeseries-forecasting: baselines vs statsforecast vs LightGBM, interval coverage, and shuffled-KFold inflation.
• reproducible-pipeline: Optuna inside walk-forward, MLflow sqlite, and export parity.

Honest time-series CV with a purge:

Calibration without refitting the model: wrap it in FrozenEstimator instead of the removed cv="prefit":

The scikit-learn calibration guide and scikit-learn#29893 cover the change.

Conformal intervals in statsforecast:

See the conformal prediction guide.

• Put the baseline in the same table. A model is reported only as a ratio to its baseline. Use MAE/SNaive (MASE-style) for forecasts and the lift over prevalence for PR-AUC. In the practiced cases the baselines gave the needed context: PR-AUC 0.117 for the prior, and SNaive24 MAE 0.64.
• Keep every fitted transform inside the CV loop (Pipeline, ColumnTransformer). Scalers, encoders, imputers, and feature selection fitted on all rows leak information (the common pitfalls page). Feature-level leakage audits belong in feature-engineering.
• Match the splitter to production.
  • Use StratifiedKFold(shuffle=True) for iid rows with a rare class.
  • Use GroupKFold or StratifiedGroupKFold when a customer, patient, or ticker repeats.
  • Use TimeSeriesSplit(gap=h) or a rolling-origin loop for anything temporal.
  • scikit-learn has no purged/embargoed splitter. gap covers the purge. Add an embargo by widening gap past the feature lookback when labels also feed features. Rolling-origin min_train_size/step options are only proposed so far (scikit-learn PR #33424).
• Nest tuning. Optuna and GridSearchCV score on inner folds. The holdout is touched once, after the choice is made (the grid search tutorial).
• Seed the sampler with TPESampler(seed=...) and run it sequentially if you need identical studies. Parallel trials are not reproducible (the Optuna faq).
• Use boosting defaults first. For LightGBM, control complexity with num_leaves (keep it below 2^max_depth) and min_data_in_leaf (the parameter tuning guide). For XGBoost depth, early stopping, and GPU, see xgboost.
• Handle imbalance with the metric and the threshold, not resampling. Report PR-AUC, log loss, and Brier. Pick the operating threshold with TunedThresholdClassifierCV (the classification threshold guide). Class weights move probabilities, so recalibrate after using them. SMOTE inside a time series pulls future neighbours into synthetic rows.
• Choose the tracking backend deliberately. Use MLflow with sqlite:///mlflow.db, and log params, seed, data hash, metrics, and the model with input_example, which infers the signature.
• Prove the exported artifact. Reload it and compare predictions against the in-memory model before calling it shipped.

• On tabular data, gradient-boosted trees are the default winner at little tuning cost. They were best or tied in both measured tables. LightGBM and XGBoost scored within fold noise of each other (PR-AUC 0.453 vs 0.453, SD about 0.005-0.011).
• statsforecast MSTL is a strong, fast-to-write forecaster for multi-seasonal series. It was the best point forecast here (MAE 0.667x of SNaive24), with close-to-nominal conformal coverage.
• Conformal intervals are model-agnostic and only need held-out residuals.
• Optuna with the ask/tell or objective API wraps any validation loop, including walk-forward.

• Tuning rarely pays as much as framing. 30 TPE trials improved walk-forward MAE from 0.4449 to 0.4387 (1.4%). Adding lag features over seasonal naive improved it 29% (0.633 to 0.448 holdout).
• Rolling-origin evaluation with classical stats models is slow. AutoETS and MSTL with 5 conformal windows took 1,027 s for 30 cutoffs on 2,880-hour windows (single process, contending with another job).
• deterministic=True in LightGBM does not cover every source of nondeterminism (LightGBM#6731). Pin the thread count, force_row_wise, and the seed as well.
• Prophet is not benchmarked here. Treat it as one candidate that must beat seasonal naive and MSTL on rolling-origin CV before adoption, never as the default. It was not run here, so this is unverified.
• Neural nets (darts, sktime deep models, TabNet-style models) are worth trying only with many related series, rich exogenous signals, or raw text/image/sequence inputs. None were measured here.

1. Accuracy on imbalance hides everything. The majority-class dummy scored 0.883 accuracy against 0.894 for the best model, an apparent gain of 1.1 points. PR-AUC for the same pair was 0.117 vs 0.453, a 3.9x lift. A class-weighted LightGBM scored below the dummy on accuracy@0.5 (0.840).
2. ROC-AUC flatters imbalanced problems. The best model reached ROC-AUC 0.80 but PR-AUC only 0.45. Read PR-AUC against the prevalence baseline.
3. Class weights destroy calibration. is_unbalance=True gave a mean predicted probability of 0.317 against a true rate of 0.117, and Brier went from 0.1354 to 0.0820 after sigmoid calibration. Calibration does not change ranking: ROC-AUC was identical at 0.7891. Isotonic calibration slightly lowered PR-AUC (0.4425 to 0.4217) because it creates ties.
4. Shuffled KFold on overlapping time targets invents skill. On the forward 24h BTC return, shuffled KFold gave R² 0.347, a 69.6% hit rate, and a gross Sharpe of 9.7. Walk-forward with gap=24 on the same model gave R² -0.221, a 50.3% hit rate, and a Sharpe of -0.97.
5. Unshuffled KFold is also wrong for time. It trains on the future. Here it gave R² -0.11 and a Sharpe of 0.77, spuriously positive.
6. Interval coverage is not what the label says. Nominal 90% intervals covered 75-76% for naive-family conformal intervals, 79.7% for AutoETS, 88.5% for MSTL, and 83.2% for LightGBM quantile intervals, measured over 720 held-out hours. Always measure coverage. Quantile GBMs under-cover because each quantile is overfit separately.
7. MAPE explodes near zero and is asymmetric. Prefer MAE/MASE, or model the log target (as in the volume case) where the error is relative. Also, conformal scores computed in log space become multiplicative after expm1 and can blow up (statsforecast#1119).
8. A leak can come from a column. The bank-marketing V12 column is call duration, known only after the outcome. Ask for every column: is it known at prediction time?
9. Binance spot kline timestamps are µs from 2025-01-01 onward and ms before that. Detect the unit per file (open_time > 1e14 means µs), or 2025+ bars land in 1970.
10. LightGBM lives at lightgbm-org/LightGBM, and issue links under microsoft/ redirect. Categorical columns must stay pandas category dtype at predict time (LightGBM#5244).

Each entry lists the version, the defect, and the workaround. No patches are used.

• MLflow 3.16.1: set_tracking_uri("file:./mlruns") raises MlflowException: filesystem tracking backend ... is in maintenance mode. The official tracking docs still say "mlruns by default", which no longer holds. The fix is sqlite:///mlflow.db (the local database guide), or mlflow migrate-filestore for old data. MLFLOW_ALLOW_FILE_STORE=true opts out but is not recommended. Discussion: mlflow#23525.
• onnxmltools with LightGBM: target_opset=17 raises "higher than ... converter support (15)". Pass target_opset=15 or omit it. Categorical LightGBM splits have a long-open conversion issue (onnxmltools#309). Convert numeric/ordinal-encoded models, or verify parity per model.
• XGBoost 3.4.1 / LightGBM 4.7.0: fit(..., early_stopping_rounds=...) raises TypeError. XGBoost takes it in the constructor. LightGBM uses callbacks=[lgb.early_stopping(n)]. The community xgboost-lightgbm snippets use the removed form.
• XGBoost 3.x: use_label_encoder=False only warns ("Parameters not used"). Remove it from the community signal-classification snippets.
• statsforecast conformal intervals and log transforms: the intervals blow up after inverse transform (statsforecast#1119). Report intervals in the modelled space, or conformalize on the original scale.

The run used 45,211 rows with 11.7% positive. V12 (call duration) was dropped as a leak. Stratified 5-fold, mean scores:

The calibration check used a held-out 25% test set. The model was fitted on 70% of train, and the calibrator on the remaining 30%:

Forecasting log quote volume, 24h ahead, 30 daily cutoffs (720 hours). Stats models were fitted on a trailing 120 days. LightGBM used all history before each cutoff.

Seasonal naive lost to plain naive because volume regimes shift faster than the daily cycle repeats. MSTL beat LightGBM on this single series.

Forward 24h log return, same LightGBM, 23,184 rows:

The base rate for "up" was 0.521. The honest model does not beat always-long, and no costs were applied. The trading-specific path lives in xgboost.

The 30-trial run (sqlite backend) tuned and logged: walk-forward MAE 0.4449 by default and 0.4387 at best; holdout MAE 0.4484, against 0.6325 for SNaive24.

The pipeline with target_opset=15 in the ONNX export and LightGBM n_jobs=4 was run twice with --trials 12:

• Both runs were identical. They produced the same best params (num_leaves=9, min_child_samples=252, lr=0.0167, n_estimators=353), the same data hash (04c5b2ac16f4fbbc), walk-forward MAE 0.43927, and holdout MAE 0.4539. The seeded TPESampler(seed) with sequential trials and LightGBM deterministic=True held here.
• Export parity against in-memory predictions over 1,440 holdout rows:
  • joblib: exact.
  • LightGBM text model.txt: exact.
  • MLflow pyfunc reload: exact.
  • ONNX (float32): max |diff| 1.47e-05. ONNX is not bit-identical, so assert it with a tolerance.
• Wall time was 36-45 s per run. The 30-trial run with default LightGBM threads took 1,184 s (sys 446 s) while another job was running. The cause was thread oversubscription. Pin n_jobs in tuning loops.
• More trials did not win on holdout. 12 trials scored holdout MAE 0.4539, and 30 trials scored 0.4484. Both were about 0.7x of SNaive24. The search budget moved the result less than the baseline gap did.

• Official docs (local text snapshots, pinned to a recent commit of each project):
  • scikit-learn reference and guides
  • LightGBM docs (repo at lightgbm-org/LightGBM)
  • Optuna docs and tutorials
  • MLflow classic ML docs
  • statsforecast README and notebooks (converted to markdown, outputs stripped)
• Community skills (from skills.sh): mle-workflow (affaan-m/ecc, 5.6K installs), scikit-learn (k-dense-ai/scientific-agent-skills, 1.9K), ml-pipeline (jeffallan/claude-skills, 3.5K), xgboost-lightgbm (tondevrel/scientific-agent-skills), and signal-classification (agiprolabs/claude-trading-skills).
• Where a community skill conflicts with the official docs, the official side wins:
  • Early stopping is passed in the constructor or through callbacks, not fit.
  • use_label_encoder is gone.
  • A bare TimeSeriesSplit(n_splits=5) without gap in the community scikit-learn evaluation guide is unsafe for horizon > 1.
  • SMOTE examples need a time-series caveat.
• Awesome lists (trimmed): josephmisiti/awesome-machine-learning (Python section), EthicalML/awesome-production-machine-learning (serving, monitoring, and tracking sections), and cuge1995/awesome-time-series.
• Libraries by job:
  • Models: scikit-learn, LightGBM, XGBoost (xgboost), CatBoost.
  • Classical forecasting: statsforecast.
  • Multi-model and deep forecasting: sktime, darts, neuralforecast (not practiced).
  • Tuning: Optuna.
  • Tracking and registry: MLflow.
  • Export: ONNX (skl2onnx, onnxmltools), native model files, joblib (same versions only).
  • Drift: Evidently, NannyML, and the others listed in awesome-production-ml (not practiced).
  • Upstream data: features (feature-engineering), warehouse extracts (olap), and modelled training tables (dbt).