Your fair-lending regression results indicate a statistically significant disparity… now what? In our last blog post, we discussed the importance of a common-sense approach to statistical analysis. One common error in statistical analysis is to assume that a result is practically meaningful just because a result is statistically different from zero. This in not always the case. In fact, finding a statistically significant result may or may not be meaningful.
Statistical Versus Economic Significance
Researchers acknowledge this fact by distinguishing between statistical significance and economic significance.
Statistical significance, discussed in a recent blog post, answers the question: Is the disparity significantly different from zero?
Economic significance answers the question: Is the disparity of any real consequence?
It is quite possible that a regression result is statistically significant but of little practical significance.
For example, it is possible to observe a pricing disparity between protected and non-protected classes in a bank’s lending portfolio that is statistically significant (meaning the results indicate the difference is non-zero) but only a few basis points in size. In practice, a few basis points may be immaterial depending on the size and term of the loan. In addition, many times a model does not include possibly impactful factors that could explain such minor differences.
Sample Size
What are some possible explanations for why a result would appear statistically significant but not economically significant? One explanation is a large sample size.
Here’s a quick primer on statistical significance – Statistical significance is determined by the p-value. The most common rule of thumb is that a p-value below 0.05 is statistically significant. The p-value is calculated using the t-statistic, where a higher t is associated with a lower p-value. A t stat is calculated by dividing the regression coefficient by the standard error.
Therefore, a smaller standard error results in a higher t-score and a lower p-value. Larger samples may produce smaller standard errors, which may be more likely to result in statistical significance.
An Example
Suppose a financial institution has a very large consumer lending portfolio, including hundreds of thousands of loans. The average loan amount is $10,000 at 10% interest over 36 months. A fair lending analysis reveals that the institution charges protected class borrowers a higher price of 2 basis points on average. And, because the sample size is so large and, thus, the standard error is so small, this pricing difference is statistically significant.
But, is the pricing difference economically significant?
In other words, the non-protected-class customer borrows $10,000 for 36 months at 10.00% and a protected-class customer borrows $10,000 for 36 months at 10.02. Although technically statistically significant, it is difficult to see how there could be greater pricing parity with such a minute difference observed. And, again, as stated earlier, such a minor difference could easily be attributable to other factors rather than the borrower’s race, gender, or ethnicity.
Let’s consider another application outside of fair lending which may be more intuitive than discrimination. Suppose that a company tests the benefits of a dietary supplement it has developed to promote weight loss.
Assuming a proper research design which allows the effects of the supplement to be isolated, the results indicate that those who took the supplement lost .01 pounds more annually than those who did not. Although the result may be statistically significant, there likely won’t be a host of investors clamoring to invest in the product.
Statistical significance, therefore, may be an indicator of precision rather than a critical finding.
Because of such ambiguities, the field of statistics sometimes seems complicated. There are few absolutes and hard decision rules. A regression result may or may not be meaningful for a variety of reasons.
This blog does not even scratch the surface of the considerations necessary in interpreting statistical results, and particularly regression. However, regression results clearly should be interpreted thoughtfully with a dose of common sense.
How to cite this blog post (APA Style):
Premier Insights. (2018, November 8). Are My Fair Lending Statistical Regression Results Meaningful? [Blog post]. Retrieved from https://www.premierinsights.com/blog/are-my-fair-lending-statistical-regression-results-meaningful.
