When conducting fair lending regression analysis of underwriting, we are examining a sample of loan applications that were either approved or denied. The practice is to regress denial (y=1 if denied, 0=approved) on a target group indicator variable and other attributes upon which the loan decision should have been based.
Since the outcome variable being measured is dichotomous (all are either approved or denied), it is common for the functional form used for the parameter estimation to be what are called “non-linear” models such as Logit or Probit. Although the use of these models may be appropriate, the difficulty becomes interpretation. In reality, the estimated coefficients in these models are of little use except to tell whether a variable has a positive or negative effect.
Suppose the estimated target group coefficient is 0.8 and is statistically significant. This tells us a disparity exists, but the coefficient tells us nothing about the magnitude of potential discrimination. The interpretation, therefore, is not straightforward.
In the last two decades, Logit has emerged as the model of choice for underwriting regression analysis because it offers an easy solution to this problem. Through a simple transformation of the regression coefficient, the impact of the variables in the model can be assessed by what is known as the Odds Ratio.
The Odds Ratio can be generated during estimation by most regression packages (such as Stata or SAS). Although the Odds Ratio provides a simple solution for interpretation, it suffers from a number of flaws – it is difficult to interpret, can be severely distorted in samples where either of the two outcomes is common (i.e., above 90%), and it may not give an accurate indication of the number of customers actually effected and therefore the true impact.
The Odds Ratio compares the odds of the outcome being measured (approval or denial) for the target and control group. The interpretation is simplistic; if the Odds Ratio = 1.00 there is no disparate treatment across groups. If the Odds Ratio is > 1 or < 1 then the event is more/less likely between the groups.
As an example, if the target group were denied 25% of the time and control group 20% of the time the Odds Ratio would be 1.33. The way this would be commonly interpreted with respect to fair lending is that the odds of denial for target group applicants was 1.33 times that of the control group.
Although this provides interpretation of an otherwise meaningless coefficient, what is more meaningful is the difference in the rate of denial for each group which in this case is 5%.
One issue with the Odds Ratio is that many pairs of outcomes give the exact same Odds Ratio. For example, if the target group were denied at a rate of 5% and the control group 4%, the Odds Ratio would be also be 1.33 although the difference in the rate of denial was only 1% compared to 5%.
Additionally, denial rates of 10% and 7.7% (a 2.3% difference) and 50% and 42.9% (a 7.1% difference) would all produce exactly the same Odds Ratio. Note as well that the Odds Ratio of 1.33 does not tell us how many of the target group customers were actually impacted by the disparate treatment which makes it useless for evaluating or estimating restitution.
In addition, odds ratios get very large for extremely common or rare events, even when very few applicants are actually impacted. For example, denial rates of 2.5% vs. 0.5% yields an odds ratio of 5.103 despite only 2 applicants out of 100 being affected. Denial rates of 99.5% vs 97.5% yields the exact same Odds Ratio. However, denial rates of 60% vs. 30% (a 30% disparity) only yield an odds ratio of 3.5. It is clear from this that the Odds Ratio can not only be misleading but has little, if any, economic meaning.
Instead of odds ratios, economists generally favor computing marginal effects. Marginal effects are simpler to interpret and understand, are not affected by extreme values, and give a direct measure of the number of affected applicants. For example, if denial rates for the two groups are 5% vs. 4% then the marginal effect is 1%.
This identifies directly the number of target group applicants impacted by potential disparate treatment. However, denial rates of 25% vs. 20.9% produce the exact same odds ratio even though it is clear that the number of applicants affected is much greater. Thus, marginal effects provide a much better and more informative statistic compared to odds ratios.
Finally, of note to CFPB supervised institutions, the CFPB also recognizes the limitations of the Odds Ratio in underwriting analyses compared to marginal effects. Some of their comments concerning this can be found starting on page 29 of the following bulletin: http://files.consumerfinance.gov/f/201510_cfpb_supervisory-highlights.pdf.
