The split is the experiment — five ways data leaks

A leak is any path by which information from the evaluation set reaches the model during fitting. The result is always the same shape: a validation score that looks excellent and does not survive contact with anything new. Since the number is higher than it should be, nothing about it feels like a bug.

1. Preprocessing fitted before the split

The most common one, and it looks completely reasonable:

# Leaks
X_scaled = StandardScaler().fit_transform(X)
X_tr, X_te, y_tr, y_te = train_test_split(X_scaled, y)

fit computed a mean and standard deviation over all rows, test included. Every training row now carries a trace of the test distribution. The same applies to SimpleImputer, PCA, SelectKBest, target encoding — anything with a fit.

The fix is structural rather than a matter of discipline: put the steps in a pipeline, so fit can only ever see the training fold.

model = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("clf", LogisticRegression()),
])

X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)
model.fit(X_tr, y_tr)          # every fit inside is train-only, by construction

This is also why cross_val_score(pipeline, X, y) is correct while manually scaling then cross-validating is not: the pipeline is refitted inside each fold.

2. Duplicate rows

Scraped and merged datasets are full of near-duplicates. A duplicate that lands on both sides of the split lets the model memorise the answer and be scored on it. Deduplicate before splitting, on content rather than on row id.

3. Group leakage

Ten photographs of the same plant, or forty questions written by the same author, are not ten and forty independent samples. Split by group, not by row:

from sklearn.model_selection import GroupShuffleSplit

splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(splitter.split(X, y, groups=plant_id))

Without this the model can learn “this leaf, this lighting” and score well without having learned the disease at all. This is the leak that most quietly inflates image-classification projects.

4. Temporal leakage

Any target that depends on time — demand, price, churn — must be split by time, because a random split trains on the future to predict the past:

from sklearn.model_selection import TimeSeriesSplit

Also check the features. A column that is only populated after the event you are predicting (“date_closed” for a churn model) is a perfect predictor and entirely useless in production. The tell is an accuracy near 1.0 with one dominant feature — treat that as a leak until proven otherwise.

5. Tuning on the test set

Fifty hyperparameter configurations evaluated on the same test set means that set has been used fifty times. The best score is then partly a selection artefact. Three splits fix it:

train  →  fit parameters
val    →  choose hyperparameters, architectures, thresholds
test    → touched exactly once, at the very end

If test has to be used twice, it is no longer a test set; it is a second validation set, and the honest write-up says so.

The habit

Before reporting any number, I now answer four questions:

1. Was every fit restricted to training data?
2. Could a single real-world entity appear on both sides?
3. Does any feature depend on knowing the target?
4. How many times has the test set been looked at?

And one sanity check that costs nothing: shuffle the labels and refit. A model that still scores well above chance on shuffled labels is not learning the task — it is reading a leak.

What to take forward

• Pipelines make train-only fitting structural instead of a thing to remember.
• Deduplicate and group-split before anything else.
• Time-ordered targets need time-ordered splits, and future-dated features are leaks.
• The test set is spent after one use.

---

— Ishaan SandhwarEvaluationscikit-learnMethodology