SmartChurn

A model that scores well
and a model you can deploy
are not the same thing.

Customer attrition is a direct hit to lifetime value, and Great One Bank wanted to intervene before customers left rather than after. The Churn Shield initiative built a Gradient Boosted Trees classifier to flag who was about to go.

By Phase 2 the model looked strong: 92.36% accuracy, solid precision and recall, and good behaviour on imbalanced data. The team had consensus. The obvious next step was to ship it.

Phase 3 existed to prevent exactly that. Before production rollout, the model had to survive contact with records nobody had seen — and be judged on criteria that had very little to do with the accuracy number everyone was quoting.

Scored against three criteria.
Accuracy was the smallest one.

Production readiness was assessed on a weighted rubric, and the weighting is the most interesting artefact of the whole project:

The validation design was deliberately blind. Each team member received 10 customer records with the true outcome withheld. The population was constructed so that exactly two of every ten had churned — but which two was unknown going in. Guessing conservatively would be caught by the ratio check; guessing aggressively would be caught by it too.

Why boosting, and why
its probabilities lie.

Gradient boosting does not fit one model to the data. It fits a sequence of shallow trees, each one trained on what the ensemble before it got wrong.

The first tree makes a crude split. The second is trained on the residuals of the first — the gap between prediction and truth. The third corrects what the first two still miss, and so on. Each learner is weak on its own; the sum is not.

That mechanism suits this dataset. Churn signals are interaction effects, not main effects — a falling transaction amount matters differently depending on relationship count and months inactive. Trees capture those interactions natively, without anyone having to specify them. Boosting also tolerates mixed types and unscaled features, which is most of this table.

GBT against CatBoost: the real difference

The comparison was not arbitrary. More than half these features are categorical — education level, marital status, income band, card tier — and the two algorithms handle that very differently.

One pipeline, two branches.

The process was built so the training path and the blind-validation path could not contaminate each other — the model is fitted and cross-validated on the bank dataset, then applied, frozen, to the held-out records.

The two branches. Top: the bank churn dataset flows through attribute selection, missing-value replacement and role assignment, then splits for cross-validation and performance measurement. Bottom: the blind record set gets identical preprocessing, then meets the already-trained model at Apply Model. Same transformations, no shared fitting.

The ratio landed exactly.
The confidences did not.

On the blind set the model returned 8 Existing Customers and 2 Attrited Customers — the 80/20 target, hit precisely, with no knowledge of the answer. Phase 3 accuracy came in at 96.51%.

Then the confidence column turned the result into something more useful than a pass mark:

Averaged across the set, confidence in Existing Customer sat at 0.854 against 0.146 for Attrited. That gap is not a bug — it is the imbalanced training distribution showing through into inference. The majority class is simply better learned.

Row-level output on the blind set. Records 1–8 classified Existing at 0.995–0.998. Records 9 and 10 classified Attrited at 0.585 and 0.851. Both churners sit above the median credit limit — but so does record 04, at 16,835, the highest in the set and confidently retained. With ten rows, that is an observation, not a feature attribution.

Model comparison

CatBoost scored higher on every headline metric — nearly five points of accuracy and 92.48% recall. It was not selected, because it reproduced the class split at 70/30 rather than 80/20. Under a rubric that weights ratio alignment at 35% and accuracy at 15%, the better-scoring model loses. That is the rubric working as designed, not a mistake in it.

A 0.998 does not mean
99.8% certain.

The rubric puts half its weight on confidence scores. Boosted trees produce some of the least trustworthy confidence scores in common use.

Gradient boosting minimises log-loss by driving the margin between classes as far apart as it can. The number that falls out of the final sigmoid is a ranking score, not a probability. It reliably tells you record 10 is a better churn candidate than record 9. It does not reliably tell you that record 1 has a 0.2% chance of leaving.

The clustering is the tell. Eight records landing between 0.995 and 0.998 is not eight independent assessments of near-certainty — it is the ensemble saturating. On an imbalanced target the majority class is over-represented in every residual round, so the model learns to be loud about it. Platt scaling or isotonic regression against a held-out fold would map these scores onto real frequencies, and a reliability curve would show how far off they currently are.

Where the decision boundary actually sits

The 0.5 cut-off is a library default, not a business decision. Drag it and watch what the model does with the real attrition confidences from the blind set:

The useful discovery is the empty space. Nothing sits between 0.006 and 0.584. Every threshold across that range — a span covering most of the scale — returns the identical answer. Threshold choice is not where the fragility lives.

The fragility is that record 09 sits at 0.585, one thousandth above the edge of that dead zone. Nudge the cut-off up by 0.002 and a real churner silently stops being flagged, and the 80/20 ratio the bank is budgeting against quietly becomes 90/10.

Where this validation
stops short.

Ten records per analyst is enough to catch an obviously broken model. It is not enough to certify one.