How to Commission a Bias Audit: A Step-by-Step Guide for Employers

If you use AI tools in your hiring process and have employees or applicants in New York City, you need a bias audit. Here's the complete guide.

Step 1: Identify What Needs Auditing

Make a list of every AI tool used in your hiring or promotion process:

• Resume screening software
• Video interview analysis tools (Hirevue, Spark Hire with AI features, etc.)
• Candidate scoring or ranking tools
• Skills assessment platforms with AI scoring
• ATS systems that use AI to filter applications

For NYC LL 144, any of these that make or assist in making decisions about NYC-based roles must be audited.

Step 2: Find a Qualified Auditor

The law requires a third-party bias auditor — an independent entity that conducts a bias audit and is not your software vendor.

What to look for in an auditor:

• Experience auditing employment AI specifically
• Knowledge of NYC LL 144 requirements
• Ability to produce the required audit summary format
• Statistical methodology experience (disparate impact analysis)

Browse bias auditors in our directory →

Step 3: Provide the Auditor with Required Data

Your auditor will need:

• Historical decision data from your AEDT (12 months recommended)
• Demographic data on candidates (or methodology for collecting it)
• Documentation of what the AEDT evaluates and outputs
• Your current use policies for the tool

Data collection challenge: Many employers don't have voluntary demographic data from candidates. Auditors have developed methodologies to handle this, including using geographic or surname-based demographic estimation (Bayesian Improved Surname Geocoding / BISG) when direct data isn't available.

Step 4: Review the Audit Results

The audit will produce:

• Selection rates by sex, race/ethnicity, and intersectional categories
• Impact ratios comparing selection rates across groups
• Whether any group falls below the 80% threshold (4/5ths rule)
• Summary of methodology used

Review results carefully. If disparate impact is found, you'll need to decide whether to continue using the tool, modify it, or stop using it.

Step 5: Post the Required Summary

Once you have the audit, you must post a summary that includes:

• Distribution date of the bias audit
• Summary of results including selection rates and impact ratios by category
• Data used in the audit

Post it:

• On your careers page (publicly visible, not behind a login)
• In any job postings where the AEDT is used

The summary must be posted before you use the AEDT.

Step 6: Add Candidate Notices

Add a notice to all job postings or application materials stating:

• That an automated employment decision tool is used
• What the AEDT evaluates (e.g., video interview analysis, resume screening)
• That candidates may request an alternative selection process (if offered)

Step 7: Set Your Annual Audit Calendar

Bias audits must be repeated at least annually and whenever there is a material change to the AEDT.

Set a calendar reminder 60 days before your audit anniversary to:

1. Contact your auditor to begin the new audit
2. Collect updated decision data
3. Update your public posting with new results before the anniversary date

Resources

• NYC LL 144 full compliance guide
• Find bias auditors in our directory
• Check compliance checker — does LL 144 apply to you?