PSO feature selection for credit-card fraud
Binary particle swarm optimisation picks 7 features out of 30 on the Kaggle fraud dataset, and the honest result is a trade — better ranking metrics, slightly worse F1, at a quarter of the input.
- features kept
- 7 of 30
- ROC-AUC, baseline → PSO
- 0.974 → 0.977
- F1, baseline → PSO
- 0.906 → 0.888
The actual problem
Fraud detection is the standard imbalanced-classification exercise: 492 fraud cases in a 50,000-row stratified sample, a ratio of 0.98%. Most versions of it stop at “train a model, quote the accuracy”, which on this data is meaningless — predicting not fraud every time scores 99%.
The question I wanted to answer instead was whether a search over feature subsets buys anything, and what it costs.
What it does
Binary PSO for feature selection. Each particle is a 30-bit mask over the features, moved by the standard velocity update and mapped through a sigmoid transfer function into flip probabilities, with a repair mask so an all-zero subset can never survive. Fitness is scored with logistic regression because it is cheap and the search calls it thousands of times; the final model is a random forest, fitted once on the winning subset.
The swarm converged at iteration 18 of a possible 50 and stopped early:
selected: Time, V2, V9, V10, V12, V14, V25 7 of 30The result, stated in both directions.
| Baseline (LR, 30 features) | PSO + RF (7 features) | |
|---|---|---|
| ROC-AUC | 0.9744 | 0.9773 |
| PR-AUC | 0.8813 | 0.8857 |
| Precision | 0.9880 | 0.9326 |
| Recall | 0.8367 | 0.8469 |
| F1 | 0.9061 | 0.8877 |
| MCC | 0.9084 | 0.8877 |
Ranking quality improved and thresholded quality fell. With 77% of the features gone, ROC-AUC and PR-AUC both went up slightly, while precision dropped five points and took F1 and MCC with it. Whether that is a win depends entirely on what the model feeds: an alerting queue that a human works through cares about ranking, and an auto-block rule cares about precision. Reporting only the metric that improved would have been the easy version of this table.
The thresholds are worth noticing too — the baseline needs 0.99999 to reach that precision, while the PSO model sits at 0.70. A model that only separates classes at the extreme end of its output range is fragile to any drift in calibration.
Engineering notes
SMOTE is applied inside the training fold only. Oversampling before the split is the standard way this experiment gets silently ruined — synthetic minority points generated from test rows leak the answer, and the score comes out excellent and meaningless.
The raw CSV is 144 MB, over GitHub’s per-file limit, so it is not committed; the
first run writes a parquet cache which makes subsequent runs 5–10× faster.
Trained models and the reports in outputs/ are committed, so the dashboard
runs and the metrics can be inspected without re-training or downloading
anything.
Every hyperparameter lives in configs/config.yaml — swarm size, iterations,
early-stopping rounds, sample size — rather than being scattered through the
training script. Seven unit tests cover the PSO itself: the transfer function,
the repair mask, early stopping, and that fitness is monotone in the metric it
claims to optimise.
What I would change
The threshold is tuned by maximising F1 on the test set, which is a small leak of exactly the kind I wrote about here. It should be tuned on a validation split and then applied to test once. The comparison is also not like-for-like — logistic regression against random forest changes two things at once, so the clean version runs both models on both feature sets.