A distribution business came to us wanting a model. Specifically, they wanted something that would predict which sales orders needed a manager’s eye before they shipped — the ones at risk of going out underpriced, against credit limits, or to accounts that were already a problem. It had become a folk skill: one or two experienced people could spot a risky order, but they were not always there, and orders slipped through when they were not. The ask was for a model to learn that instinct and apply it consistently.
The right recommendation was the one that talked them out of the project they came in asking for.
What mattered most
- The thing they wanted flagged was, on inspection, governed by a few crisp business rules — not a fuzzy pattern that needed learning.
- The managers needed to understand and trust why an order was flagged; an opaque score would not have earned that trust.
- There was not enough clean, labelled history of past problems to train a model on responsibly anyway.
- Whatever we built, the manager still had to make the call on a flagged order.
How we approached it
We started from the problem rather than the request, and sat with the experienced staff to find out what actually made an order worth a second look. Most of it came down to three things you could state plainly — margin below a threshold, an account over its credit limit, or a customer with an open dispute. That is not a prediction problem; it is a rule. Building a model to approximate three rules we could simply write down would have added cost, delay and opacity, and given a worse answer than the rules themselves. We said so, plainly, even though it was a smaller engagement than the one they came for.
So we built the three rules into the order system and produced a clear daily report of flagged orders, each annotated with the reason it was flagged, for a manager to review and clear. No model, no score, nothing to retrain. We measured against the one thing that mattered: whether risky orders were being caught before they shipped, against a target the managers set. If the rules ever stop being enough — if the risk genuinely becomes a pattern no rule can capture — a model can be revisited then, on evidence rather than on hope.
Where it stands
Risky orders now get caught regardless of who is in the office, and every flag is something a manager can read and understand in a sentence. It cost less, it shipped sooner, and it is easier to maintain than the model would have been. We would rather be remembered for the project we advised against than for one that looked impressive and earned its keep less honestly.