Three reasons why you always need to explain your model


Tom Breur

25 August 2018

“Building a model is easy, getting it implemented is the hard part” is an adage that all experienced data scientists are familiar with. When I read about this in Dorian Pyle’s classic “Business Modeling and Data Mining” (2003) this rang very true for me. A corollary to this is a practice I have adopted to explain each and every model I deliver. And that holds regardless, or even ‘despite’ the fact that sometimes such an explanation is not even wanted or requested.

There are occasions when you need a model to explain, and sometimes ‘only’ to predict as Galit Shmueli has clarified in an insightful paper (“To Explain or Predict”, 2010). But although a predictive rather than an explanatory model may be all that was requested, my bias is to always, always explain the dynamics and results of a predictive model. My reasons for that are threefold:

  1. To provide a sanity check
  2. To drive innovation
  3. To foster adoption

The last of these three, is what Dorian Pyle emphasized in 2003 when he mentioned that building a model is easy, but getting it adopted is the hard part.


Provide a sanity check

Data are fickle stuff. An important reason to highlight the dynamics of your model, any model regardless of its purpose, is to provide one more sanity check to ensure the data actually make sense. Call it self-preservation. Of course, you already checked the data. You always do. And yet…

One day I was delivering my final boardroom presentation for a segmentation project with a private banking client. At some point I showed the distribution of profitability across their client base. It looked something like your average pareto chart:


With their distribution being considerably more skewed. Something like 7% of customers made up 93% of profits. And the bottom 20% were all very close to zero profitability. My presentation went well, results met with enthusiasm. Until at the very end, someone from IT (don’t you hate those nitpickers!) asked me: “And what about the customers with negative profitability?” To cut a long story short, it turned out that in the EBCDIC conversion from their mainframe somehow the minus signs had been lost… Not a pretty sight. I begged Scotty to beam me up, but he was unresponsive. In the end, after rerunning the results, that omission turned out not to matter one iota: a tiny positive or tiny negative profitability equated to the exact same treatment, and therefore implementation of the segmentation project remained unchanged. But needless to say, I would have much preferred to avoid the embarrassment.


Drive innovation

Another reason to provide an explanation for each and every model you deliver is on the hopes this may trigger some unforeseen innovation in business practices. Usually that occurs on the back of “insight” – some revelation that occurred in the process of building or refining the model. The data exploration phase often sheds light on consumer behavior, for instance. And otherwise the explanation that accompanies your model might.

This is undoubtedly the reason that pays off least often. However, when it does, it can be a whopper. Some 15 years ago I built a model to predict acquisition of first-time mortgages. I noticed an unusual cluster of customers that were characterized by three things: they had high income (> $250K), yet very low mortgage amounts (< $100K), and they were pretty much the only customers to purchase their mortgage through the call center (which at the time was still highly unusual). Since this was their first mortgage with my client’s company, I concluded this must be their second mortgage. You might consider it an ‘oversized’ home equity loan.

My first instinct was: something must have gone wrong in the data extraction process. After double and triple checking, that didn’t turn out to be the case. Still puzzled, I asked my business partner if she had any idea why this phenomenon might be occurring. She had none whatsoever. So together we trotted out to the manager of the call center to find out if they had an explanation. There was only a very small group of highly specialized agents that were authorized to close such deals on the phone (it takes a lot of product knowledge). And that elite group were the only agents that were allowed to negotiate interest rates… And indeed, they confirmed they were getting some cut-throat offers from prospects who held their primary accounts with competitive lenders. And they typically had excellent credit ratings (a primary ground to reconsider the interest rate). This finding eventually led to an entirely new mortgage proposition, aimed at highly affluent prospects for home equity loans.


Foster adoption

The final reason, arguably the most “obvious” one, is to provide an explanation so that your business stakeholders more readily embraces use of a model. This is meant to improve your odds that it actually gets implemented. In case you are wondering: tons of data science work gets delivered at great expense and tremendous effort, only to wind up as shelf ware. Facilitating adoption, “marketing” the fruits of you labor if you may, is tremendously important. Contributing to the data science value chain emphatically does not stop when you deliver your model. After all this effort, would you leave adoption to chance? Of course not!

Every time a (new) model is implemented, somewhere, someone needs to change their ways of working. It is almost as if they “surrender” their authority to the invisible magic contained in a machine learning algorithm. About half a century ago, for example, lenders were beginning to realize that human underwriters made more mistakes evaluating loan requests than credit scoring models did. To be clear: a financially economic balance implies some loans that are granted will default. Because of the asymmetric misclassification costs (profits are often a mere 10% of losses on a default, or even less), you cannot afford to guess wrong very often. Or else you get exposed to huge liabilities. From monitoring loan acceptance accuracy, we learned that credit score models performed better (classified more accurately) than humans. In the end, many underwriters became less and less necessary as models began to take over. At first, this was not intuitive. How can these experienced staff make more errors than a simple model? Knowing and understanding what criteria surfaced in the models greatly facilitated that transition.



Data science models are increasingly becoming a “commodity”: the process of data gathering and deriving a predictive or explanatory model can be standardized. It can even be automated to a large extent, like with this neat H2O product. I don’t expect data scientists to go the way of the dodo like many credit underwriters did, though. One of the (main) reasons for that is because building the model is only one part of the equation. In my mind, framing the problem, and overseeing implementation and model monitoring, will not be automated any time soon.

More importantly, explaining your model has an element of data storytelling to it, and fostering adoption is just as important (if not more so!) as building the model. Whatever your motivation for executing data science projects, it has become my firm belief that providing an explanation along with your model is a “must” – skip it at your own peril.





Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s