Pipelines for Social Bias Testing of Large Language Models

Examples of possible integration of social bias tests

Abstract

The maturity level of language models is now at a stage in which many companies rely on them to solve various tasks. However, while research has shown how biased and harmful these models are, systematic ways of integrating social bias tests into development pipelines are still lacking. This short paper suggests how to use these verification techniques in development pipelines. We take inspiration from software testing and suggest addressing social bias evaluation as software testing. We hope to open a discussion on the best methodologies to handle social bias testing in language models.

Federico Bianchi
Federico Bianchi
Postdoctoral Researcher at Stanford University

My research interests include meaning in natural language and programming languages.

Related