AI is more likely than humans to form biases when hiring
Original reporting by MIT Technology Review
Large Language Models (LLMs) are AI systems capable of processing and generating human-like text, but new research reveals they can independently develop and amplify their own biases through experience, particularly when making hiring decisions. While AI is increasingly deployed to screen job applications, previous studies have shown that LLMs often inherit human biases present in their vast training data. Now, a groundbreaking study from Princeton University and the University of Chicago demonstrates that these models can *generate* their own stereotypes, exhibiting a tendency to segregate job applicants by demographic group even more profoundly than humans.
Researchers put leading LLMs, including ChatGPT, Claude, and Gemini, through a simulated hiring game. Models were tasked with filling 20 diverse jobs for a fictional city, choosing candidates from four equally qualified, fictional ethnic groups. Unbeknownst to the LLMs, every candidate was equally likely to succeed in any role. Yet, the models quickly began assigning specific groups to particular jobs based on early, random outcomes. For instance, if a candidate from one group failed a "doctor" role, the model would subsequently steer all members of that group towards "janitor" positions. This propensity for stereotyping was striking: LLMs scored roughly 65% higher on a segregation scale than human participants in a similar psychology study, with advanced reasoning models showing even stronger biases. This rapid formation of stereotypes stems from LLMs' core optimization for generalizing from limited data, a strength in logic puzzles that becomes a critical flaw in social contexts. As AI companies enhance models with memory and agency, these novel biases pose a significant challenge for fair and equitable AI deployment.
The research from Princeton and the University of Chicago reveals a critical vulnerability in large language models: their inherent drive to generalize from limited data makes them susceptible to forming their *own* stereotypes based on experience. In simulated hiring scenarios, LLMs not only developed biases but did so more aggressively than human participants, rapidly segregating fictional ethnic groups into job niches. This tendency, rooted in their optimization for tasks requiring quick pattern recognition, highlights how the very strengths of these models in analytical domains can become significant liabilities when applied to nuanced social decision-making, where the costs of premature generalization are substantial.
Beyond the hiring game This finding extends far beyond resume screening. As LLMs become integrated into high-stakes domains such as loan approvals or parole recommendations, their capacity to generate novel biases from real-world feedback presents a profound challenge. The emergence of these "ever-present" biases, not directly inherited from human training data, makes them particularly insidious and difficult to predict or mitigate. The path forward demands a fundamental shift in AI design, moving beyond simple fairness directives to embed desirable social values directly into models’ core objectives and providing them with rich, individual-specific data. Without such deliberate architectural choices, the increasing deployment of agentic AI systems with enhanced memory risks automating and amplifying discriminatory patterns that no human explicitly taught them.
Frequently asked questions
- How do large language models develop new biases when screening job applications?
- Large language models (LLMs) can develop novel biases from experience by over-generalizing from limited hiring outcomes. When optimized for pattern recognition, LLMs quickly form stereotypes based on early data, rather than exploring diverse options. This 'exploration-exploitation dilemma' means they prioritize sticking with what appears to work, leading them to categorize applicants into specific roles based on demographic group, even if initial results are coincidental, making them prone to biased decision-making in recruitment.
- Are AI hiring systems more prone to stereotyping job candidates than human recruiters?
- Research indicates that AI hiring systems, specifically large language models, can be significantly more prone to stereotyping job candidates than humans. In simulated hiring scenarios, LLMs showed a much higher tendency to segregate candidates by demographic group into specific job niches compared to human participants in a similar study. This heightened inclination results from the models' optimization to rapidly generalize from limited data, which can unfortunately lead to quick and persistent stereotyping.
- What methods can effectively reduce bias in AI models used for hiring and talent acquisition?
- To reduce bias in AI hiring models, it's crucial to design goals that actively incorporate desirable social values, such as offering bonuses for diverse hiring. Simply instructing models to be 'fair' is often insufficient. Additionally, providing models with relevant personal information about candidates, like age or education, can help reduce demographic-based stereotyping. Conversely, irrelevant personal details do not mitigate bias. The key is to structure incentives and data input to encourage socially responsible decision-making.