Approach
Eight steps. The uncomfortable ones come early, because that is where models are lost, and late discovery is expensive.
Nothing here is novel
Every step below is standard practice in an experimental science. None of it is a competitive secret, and we would be glad if every firm adopted it.
It is on this page because in commercial machine learning it is frequently skipped, and because a client cannot tell the difference from the outside. The only defence is to publish the method and invite you to hold us to it.
Frame the question
Most business questions are not yet answerable. 'Reduce churn' becomes 'predict which accounts will lapse within sixty days, better than the current rules engine.' Nothing proceeds until it is measurable.
Assess the data
Volume, labels, leakage, drift. We look for the target variable hiding in a feature, which is the single most common reason a model dazzles in development and fails in production.
Establish the baseline
A seasonal naive forecast. A rules engine. Last year's number. Whatever you use today. The model must beat it by an agreed margin or it does not ship.
Split, then do not look
The test set is separated before any modelling and is not opened until the end. Validation happens on a separate split. This is not pedantry, it is the difference between a result and a rehearsal.
Model, iterate, report honestly
Weekly notes that say what failed. Most experiments fail, and a report that hides that is hiding the only information you are paying for.
Open the test set, once
One evaluation, reported with a confidence interval. If it does not beat the baseline, we say so and you do not deploy. This has happened.
Ship with monitoring
Drift detection, alert thresholds, and a retraining plan. A model without monitoring is an unowned liability.
Watch, and retire when needed
Models decay. We tell you when yours has stopped earning its place, even though the recommendation ends our own retainer.
The engagement is only half ours
- Access to the real data, not a cleaned sample prepared to impress us
- A domain expert for two hours a week, who can explain what a column actually means
- An honest account of how the decision is made today, including the informal parts
- Agreement on the metric before we begin, and the discipline not to change it when it becomes inconvenient
- Willingness to hear that the answer is a SQL query
The fifth one is the hardest, and the most valuable.
What we will not do
We will not train a foundation model for a business problem. We will not report accuracy on a class imbalanced dataset and call it performance. We will not evaluate on the training set, present a metric without an interval, or ship a model with no monitoring.
We will not build a system that makes a consequential decision about a person with no route to human review.
Each of these has cost us an engagement. Each is why the clients we keep, we keep.
Send us your data problem. We will tell you whether it is one.
A working data scientist reads every enquiry. You will get an honest read on whether your data can support the model you have in mind, before anyone quotes you a number.