Computational method for grouping and reducing representative metrics for identification and mitigation of bias and unfairness in machine learning models

Abstract

Bias and unfairness in machine learning models happen when they make biassed or unfair decisions that perpetuate and amplify the unfair discrimination and exclusion of people. Identifying and addressing bias and unfairness in those models in different application domains is a multifaceted challenge. While numerous unfairness metrics have been pro posed, determining an optimal set of metrics for assessing a model’s unfairness remains an open question in the literature due to the diverse nature of these metrics and the lack of comprehensive approaches to ensure fairness across multiple applications. Consequently, there is a pressing need to narrow down the metric space and identify representative metrics for algorithmic unfairness evaluation. The current literature presents a limited number of studies aimed at reducing the number of fairness metrics used when evaluating a model, with the available techniques facing limitations, including restriction to spe cific application areas, dependence on the user’s understanding of the problem, and high computational cost. Therefore, this study aims to propose a computational method that allows the selection of the most representative metrics for bias and unfairness assessment in post-processing for binary classification machine learning models in different contexts. To achieve this goal, four case studies were used in the fields of criminal judgement, bank loans, demographic census, and advertisement, with unfairness identified against the sen sitive attributes: race, gender, race, and age group. Furthermore, a correlation-based strategy was used as a heuristic for selecting unfairness metrics. The potential problems with the approach were then analysed, and solutions were proposed to mitigate these problems and evaluate its effectiveness. The method starts the procedure using bootstrap sampling in conjunction with the Markov chain Monte Carlo method. Modifications and validation strategies are proposed, such as transitioning to a stratified sampling method to better represent the data biases, incorporating a stopping criterion to reduce the com putational cost, shifting from Pearson to Kendall correlation for more robust estimations, and validating the method by examining different aspects of the selected metrics. A sub stantial reduction in computational cost was noted, with an average decrease of 64.37% in the number of models required and of 20.00% in processing time. Moreover, the pro posed method maintains result consistency by effectively pairing metrics with similar behaviour. The proposed experiment was able to group metrics with similar equations more frequently, making the presence of a similar term in the equation a strong indicator of a direct relationship between two metrics. While no standout metric emerges across all contexts, within specific models or datasets, certain metrics consistently stand out. For the analysed cases, the Predictive Parity met ric was highlighted in the criminal judgement, demographic census, and advertisement scenarios, while the Error Ratio metric was highlighted for the demographic census, and Equalized Odds was in evidence in criminal judgment. Overall, the proposed method suc cessfully selects the representative metric with a considerable gain in computational costs