As fair lending analysis becomes increasingly technical, industry practitioners have had to familiarize themselves with the terminology of statistical analysis. Statistical significance is one of the most common and foundational concepts to successfully navigating these new waters. Moreover, it is a concept that, when misunderstood, may result in serious error.
In its simplest form, statistical significance can be understood as a yes/no answer to a single hypothesis. While there are an infinite number of possible hypotheses, in this post, we discuss the primary hypothesis to which we apply the concept of statistical significance in fair lending – is there a pricing/underwriting disparity for a protected class of borrowers.
In statistics terminology, the hypothesis is “is the relation between protected class and pricing/underwriting equal to zero?” When we observe a regression result that is statistically significant, the answer to this question is no.
However, practitioners should be cautious about inferring more information from “statistical significance” that what it really implies. Statistical significance tells us the result is different from zero. It does not tell us if our result is exactly right. Consider the following example of this common mistake.
An Example of a Common Mistake
Suppose a financial institution conducts an analysis on the pricing of its loans. The results show that there is a pricing difference between female and male borrowers of 8 basis points. Moreover, this result is statistically significant.
One common misunderstanding of statistical significance in this case is that it implies that we are confident that the difference is 8 basis points.
Actually, statistical significance refers to another hypothesis altogether – the likelihood that we would observe this result if, in fact, there is no relation between the protected class and the price. Put another way, it indicates whether we can reject the hypothesis that there is no relation between protected class and price.
When we conclude that a price disparity is statistically significant, we are simply saying that our estimate is statistically different from zero. Therefore, it is very unlikely that this disparity occurs when there is actually no discrimination.
So, statistical significance in the case mentioned above implies that we are confident that females and males are not charged the same price (the relation between female and price is not zero). But, whether the difference is exactly 8 basis points is another question altogether. It is quite possible that our estimate is statistically different from zero but not statistically different from, say, 2 basis points. Therefore, we should not put too much confidence in the precision of the estimate simply because the estimate is statistically significant.
An Analogy
Consider the following analogy. Suppose an individual is accused of robbing a bank. If we presume the individual is innocent until proven guilty, then the burden of proof is on the accuser. When the accuser presents evidence to demonstrate guilt, the jury must decide whether the evidence is enough to convict the accused of robbing the bank.
In statistical-talk, the “null hypothesis” is that the individual is innocent. Given enough evidence, the jury may decide to reject the null hypothesis and conclude that the individual is not innocent. Note here that the decision is based on the likelihood of being guilty rather than absolute proof.
When it comes to drawing conclusions from statistical results the principle is the same. In fact, through science, much (if not all) of what we know and believe about the world is based on the probability of being right rather than absolute proof.
In statistics, we make such decisions based on the p-value. The p-value tells us the probability that the provided evidence is simply a coincidence and the accused is truly innocent. The general rule of thumb is that a p-value less than 0.05 is enough to conclude statistical significance. Therefore, when there is less than a 5% chance that the provided evidence occurs coincidentally, then it is sufficient evidence to reject the null of innocence and conclude the accused is not innocent.
If the jury decides that the accused is guilty, representing statistical significance, their decision tells us nothing about how much money the individual stole from the bank, nor is it the jury’s purpose. Their purpose, at least in our analogy, is to determine whether the accused is guilty or innocent. It would be a mistake for us to read into how much money the accused stole from their binary decision of innocence/guilt.
Similarly, we cannot conclude that the financial institution discussed previously is discriminating against female borrowers by exactly 8 basis points, because the statistical significance of the relation between female and price only tells us that the result is non-zero.
How to cite this blog post (APA Style):
Premier Insights. (2018, October 25). Understanding Statistical Significance [Blog post]. Retrieved from https://www.premierinsights.com/blog/understanding-statistical-significance.
