The Growing Disparate Impact of the BISG Proxy Model on Fair Lending Compliance

The Bayesian Improved Surname Geocoding (BISG) proxy model has become an essential tool used by lenders, fintechs, regulators, and enforcement agencies to analyze consumer lending outcomes and identify potential discrimination over the past decade. By combining an individual‘s surname and geographic location with Census demographic data, the BISG model estimates the probability that the individual belongs to a particular race or ethnicity. These BISG probabilities are then used to test for disparate treatment and disparate impact in lending outcomes.

However, as the racial and ethnic diversity of the U.S. population has increased significantly between 2010 and 2020, evidence suggests that the BISG model itself is exhibiting a growing disparate impact, especially for Black Americans. The accuracy of the model in predicting an individual‘s race or ethnicity has decreased, with higher error rates and more minorities being misclassified. This raises important questions about the fairness and reliability of fair lending analyses based on BISG proxies.

Increasing Racial and Ethnic Diversity in the U.S.

The 2020 U.S. Census data shows that the U.S. adult population has become more diverse compared to 2010:

• The Black population share increased by 0.1 percentage points to 12.1%
• The Hispanic population share increased by 2.6 percentage points to 18.2%
• The Asian & Pacific Islander population share increased by 1.2 percentage points to 6.3%
• The White population share decreased by 5.5 percentage points to 60.1%
• The Multiracial population share increased by 1.3 percentage points to 2.9%

These national-level demographic shifts are the result of two factors according to the Census Bureau: actual demographic changes (births, deaths, and migration) as well as improvements in how the Census questionnaires collect race and ethnicity information.

At the state level, most states (41) saw an increase in the Black population share between 2010 and 2020, led by North Dakota, Nevada, Minnesota, Georgia, and Delaware. However, a handful of states, notably the District of Columbia, South Carolina, and North Carolina, experienced significant decreases in Black representation. For Hispanics, all 50 states and D.C. increased their Hispanic population share, with the largest gains in the Northeast and Mid-Atlantic. The Asian & Pacific Islander population share increased in 49 states, with Hawaii being the only exception.

Looking at migration, Texas, Georgia, and Florida saw the largest net increases in Black population share from migration while California, New York, and Illinois saw the largest decreases. For Hispanics, Florida, Pennsylvania, and Washington had the highest in-migration while California, New York, and New Mexico had the highest out-migration. Texas, Washington, and North Carolina gained the most in Asian & Pacific Islander migration share while California and Hawaii lost the most.

These state-level demographic changes are significant because they mean that the racial and ethnic composition of lenders‘ borrower bases are likely shifting as well, depending on the lenders‘ geographic footprints. A lender concentrated in the Southeast may be seeing a borrower population that is becoming less Black, while a lender focused on the Mid-Atlantic is probably seeing a large influx of Hispanic borrowers.

At the neighborhood level, the 2020 Census shows that Black and Hispanic Americans are living in slightly more diverse neighborhoods compared to 2010:

• The average Black adult lives in a neighborhood (Census block group) that is 42.1% Black, down from 46.2% in 2010.
• 61.4% of Black adults live in neighborhoods that are less than 50% Black, up from 56.5% in 2010.
• The average Hispanic adult lives in a neighborhood that is 43.2% Hispanic, down slightly from 43.8% in 2010.
• 60.0% of Hispanic adults live in neighborhoods that are less than 50% Hispanic, up from 58.8% in 2010.

The increasing neighborhood diversity for Black and Hispanic Americans is important because neighborhood racial/ethnic composition is one of the key factors driving the BISG model‘s predictive power. As neighborhoods become less segregated, it will likely be harder for the BISG model to predict race/ethnicity as accurately, especially for minorities.

Deteriorating Accuracy of the BISG Model

Incorporating the new 2020 Census demographic data, the updated 2020 BISG model exhibits lower inherent accuracy compared to the 2010 BISG model when applied to a representative sample of the current U.S. adult population:

• The overall error rate in predicting an individual‘s race/ethnicity based on the BISG probabilities increased to 17.3% for the 2020 model versus 16.8% for the 2010 model.

• For Black Americans, the 2020 BISG model fails to identify 43.1% of actual Black adults (the "false negative rate"), up from 42.4% for the 2010 model. It also wrongly identifies 29.3% of predicted Black adults who are actually non-Black (the "false positive rate"), up from 28.4%.

• Hispanics saw a slight improvement, with the false negative rate decreasing from 21.8% to 21.1% and the false positive rate decreasing from 21.8% to 21.1%. However, the BISG model still fails to identify over 1 in 5 actual Hispanic adults.

• For Asian & Pacific Islanders, the false negative rate increased from 57.4% to 58.2% while the false positive rate increased from 39.0% to 39.1% between the 2010 and 2020 models.

This deterioration in BISG accuracy is likely due to the broad trend of greater racial/ethnic integration occurring both across states and within neighborhoods. As the U.S. population becomes more diverse and residentially integrated, an individual‘s surname and geographic location become less predictive of their actual race or ethnicity.

The increase in BISG prediction errors for Black Americans is especially concerning from a fair lending perspective. Not only does the BISG model fail to correctly identify over 40% of actual Black adults, it is also more likely to misclassify non-Black individuals (predominantly lower-income Whites) as Black. This means fair lending analyses based on BISG proxies are more likely to exclude a substantial portion of actual Black borrowers while wrongly including many non-Black borrowers in tests for disparate impact.

Impact on Fair Lending Analyses

What do these changes in diversity and BISG accuracy mean for fair lending compliance? To quantify the impacts, I conducted two simulation analyses using a representative sample of the current U.S. adult population.

In the first analysis, I simulated a scenario where a lender engages in disparate treatment by charging Black and Hispanic borrowers a hypothetical $100 fee that is not charged to non-Hispanic White borrowers. I then compared the BISG model‘s estimates of this disparity to the actual perfectly-known $100 disparity.

The results show that using the 2020 BISG model underestimates the true disparate impact by:

• 34.4% for Black borrowers (vs. 34.0% using the 2010 model)
• 23.0% for Hispanic borrowers (vs. 23.0% using the 2010 model)

In other words, the BISG model masks about a third of the actual disparate treatment against Black borrowers and nearly a quarter of the disparate treatment against Hispanic borrowers. Using the outdated 2010 BISG model with 2010 Census data on a current borrower population exacerbates the underestimation, failing to identify 37.0% of the disparity for Blacks and 25.7% for Hispanics.

For the second analysis, I simulated a disparate impact scenario where borrower fees are tied to neighborhood income levels, with borrowers in lower-income neighborhoods paying higher fees and vice versa. Since Black and Hispanic borrowers are more likely to live in lower-income neighborhoods compared to non-Hispanic White borrowers, this income-based pricing has a disparate impact on minority borrowers though it is facially neutral.

Using the 2020 BISG model to estimate this income-based disparate impact produces highly inflated estimates:

• +114% relative overestimation for Black borrowers (vs. +106% using the 2010 model)
• +98% overestimation for Hispanic borrowers (vs. +88% using the 2010 model)
• -84% underestimation for Asian & Pacific Islander borrowers (vs. -87% using the 2010 model)

The large overestimates for the 2020 BISG model are due to the fact that the actual Black and Hispanic individuals most likely to be misclassified by the BISG model as non-Black and non-Hispanic (the "false negatives") tend to have higher-than-average incomes for their racial/ethnic groups. Conversely, the non-Black and non-Hispanic individuals most likely to be misclassified as Black or Hispanic (the "false positives") tend to have lower-than-average incomes.

This means that the BISG-predicted Black and Hispanic groups used for fair lending analysis are likely to have substantially lower average incomes (and therefore higher fees in this simulation) than the actual Black and Hispanic populations. At the same time, the BISG-predicted non-Hispanic White group will have a higher average income and lower fees because many lower-income White borrowers are misclassified as minorities. The net result is a highly exaggerated estimate of the disparate impact.

While this simulation focused on a hypothetical scenario where pricing is solely determined by neighborhood income, a similar effect is likely to occur for pricing determined by other borrower characteristics correlated with race/ethnicity and income, such as credit scores. To the extent that the BISG model misclassifies higher-income, more creditworthy minorities as non-minorities and lower-income, less creditworthy Whites as minorities, fair lending analyses are likely to produce inflated disparate impact estimates.

The Troubling Implications

Fourteen years after its introduction and widespread adoption, the evidence indicates that the BISG proxy model – a primary tool used by lenders, regulators, and enforcement agencies to identify fair lending risk – is becoming increasingly inaccurate as the U.S. population becomes more racially and ethnically diverse and integrated. This growing inaccuracy is undermining the reliability of fair lending compliance assessments, particularly for the minority groups that the BISG model is intended to protect.

The implications are troubling:

• Lenders relying on BISG proxies to identify and reduce fair lending risk may be significantly underestimating potential disparate treatment in their lending practices, leaving them vulnerable to regulatory violations and enforcement actions.

• Fair lending analyses based on BISG proxies appear to be substantially overestimating potential disparate impact against Black and Hispanic Americans, while simultaneously underestimating potential disparate impact against Asian & Pacific Islander Americans. This distorts fair lending priorities and resource allocation.

• Fintechs and technology companies using BISG proxies to evaluate and de-bias their AI/ML underwriting and marketing algorithms may be relying on faulty benchmarks, undermining their efforts to ensure equitable access to credit.

• Enforcement agencies and consumer advocacy groups relying on BISG proxies to identify fair lending violations and consumer harm may be misdirecting their investigations and efforts based on unreliable disparity estimates.

The fact that the BISG proxy model seems to be doing the greatest disservice to Black Americans in particular – by failing to identify over 40% of Black adults, excluding higher-income Blacks, and wrongly including many lower-income non-Blacks – raises uncomfortable questions about why a model with an apparent growing disparate impact continues to be so widely used in the fight against credit discrimination.

As algorithmic bias in AI/ML models rightly attracts greater scrutiny and concern, it is time to reckon with the algorithmic bias that has taken hold in the conventional tools and methods used for fair lending compliance. A critical rethinking of the role and usage of BISG proxies is urgently needed.