Regulome
Search regulations…⌘K
For ProvidersFree Checker

Algorithmic Discrimination

When an automated system treats people differently based on protected characteristics — such as race, gender, age, or disability — resulting in unfair or unlawful outcomes.

Also known as: AI discrimination, automated discrimination, discriminatory algorithm

Detailed Definition

Algorithmic discrimination occurs when an AI or automated decision-making system produces outcomes that systematically disadvantage individuals based on protected characteristics, even if those characteristics are not explicitly used as inputs.

The term covers a wide range of AI-powered decisions: who gets a job interview, who receives a loan, who is flagged for additional scrutiny at an airport, who qualifies for housing, and who is recommended for parole. In each case, the harm is the same — the AI encodes bias that produces unfair outcomes at scale.

How It Happens

Algorithmic discrimination can emerge from:

  • Biased training data — historical data that reflects past discrimination gets encoded into the model. A hiring algorithm trained on résumés from a historically male workforce learns to prefer patterns associated with male candidates.
  • Proxy variables — features that correlate with protected characteristics (e.g., zip code as a proxy for race, or name origin as a proxy for ethnicity). Even when race is excluded as an input, models can infer it from correlated features.
  • Feedback loops — systems trained on their own outputs can amplify initial biases over time. A predictive policing tool that directs more officers to minority neighborhoods generates more arrests there, which is then fed back as evidence those neighborhoods are "high crime."
  • Measurement bias — labels or outcomes in training data that reflect human bias. If a training dataset labels someone "successful" based on a supervisor's rating, and those ratings have historically been lower for women, the model learns to predict success through a discriminatory lens.
  • Sample size disparities — models trained on predominantly majority-group data perform poorly on minority groups because they have less data to learn from. Facial recognition systems, for example, have been shown to have significantly higher error rates for darker-skinned faces.

Legal Significance

Colorado AI Act

Under the Colorado AI Act (SB 24-205, effective February 2026), deployers of high-risk AI systems must use "reasonable care" to protect consumers from known risks of algorithmic discrimination. The law applies to AI used in consequential decisions across employment, credit, education, housing, healthcare, and insurance. Deployers are required to:

  • Conduct and document impact assessments
  • Implement a risk management policy
  • Provide consumers with notice and an explanation when an AI system makes a consequential decision about them
  • Allow consumers to appeal consequential decisions and request a human review

Violations expose deployers to enforcement by the Colorado Attorney General and potential civil penalties.

NYC Local Law 144

Under NYC Local Law 144, employers using Automated Employment Decision Tools (AEDTs) for NYC hiring must commission annual independent bias audits, publish audit results publicly, and give candidates advance notice of AEDT use. The law targets a specific mechanism for detecting algorithmic discrimination: statistical comparison of selection rates across race/ethnicity and sex categories. An impact ratio below 0.80 triggers an adverse impact flag.

EU AI Act

The EU AI Act treats the prevention of algorithmic discrimination as a fundamental rights obligation. High-risk AI systems — including those used in employment, education, credit, law enforcement, and migration management — must be designed and tested to minimize discriminatory outcomes. Providers must implement data governance practices to examine training data for bias, and deployers must conduct fundamental rights impact assessments before deploying high-risk systems.

Real-World Examples of Algorithmic Discrimination

Hiring: Amazon's experimental AI recruiting tool was discontinued in 2018 after the company found it systematically downranked women's résumés. The model had been trained on ten years of résumés submitted to Amazon, which reflected the historical male dominance of the tech industry.

Facial recognition: Multiple studies — including MIT Media Lab research — have documented that commercial facial recognition systems misidentify darker-skinned women at error rates up to 34 percentage points higher than lighter-skinned men. Several cities have banned government use of facial recognition for this reason.

Healthcare: A widely-used healthcare algorithm was found to assign lower risk scores to Black patients than equally sick white patients, because it used healthcare spending as a proxy for health needs — and Black patients historically had less spent on their care due to systemic barriers to access.

Criminal justice: Predictive risk assessment tools used in bail and sentencing decisions have been shown to assign higher risk scores to Black defendants than white defendants with similar criminal histories, raising serious due process and equal protection concerns.

Preventing Algorithmic Discrimination

Organizations deploying AI in high-stakes decisions should:

  1. Conduct bias audits before deployment — test the system against historical or synthetic data to identify adverse impact before it affects real people
  2. Monitor outcomes continuously — track selection rates by demographic group over time and flag anomalies
  3. Examine training data for representativeness — ensure training data is demographically diverse and that historical biases in labels are identified and corrected
  4. Avoid proxy variables — audit input features for correlation with protected characteristics
  5. Maintain human review pathways — ensure individuals affected by consequential AI decisions can request a human review and an explanation

Regulatory Enforcement Trends

Algorithmic discrimination is increasingly attracting regulatory attention beyond its traditional domain in employment law. The CFPB has issued guidance applying the Equal Credit Opportunity Act to AI-driven credit decisions. The FTC has published guidance on the use of AI in housing and lending. The EEOC has indicated it will treat discriminatory AI as a Title VII violation when used in employment decisions.

State attorneys general — particularly in states with broad AI laws like Colorado — are developing the investigative capabilities to examine algorithmic discrimination claims. As AI adoption spreads, algorithmic discrimination enforcement is expected to accelerate across sectors.

Related Resources

  • AI Bias Audit — How to test AI systems for discriminatory outcomes
  • Impact Assessment — The broader risk assessment framework required by the EU AI Act and Colorado AI Act
  • Adverse Impact — The statistical concept used to quantify discriminatory AI outcomes
  • Colorado AI Act — State law with explicit algorithmic discrimination protections
  • EU AI Act — European regulation requiring fundamental rights protection in high-risk AI
  • NYC Local Law 144 — Mandatory bias audits for hiring AI in New York City
  • Bias Audit Providers — Find firms that conduct independent AI bias audits