Who Decides? Creating Systems People Trust

AI tools are already making decisions that affect who gets hired, who gets promoted, whose complaint gets flagged, and whose voice gets heard in your organization. Without a governance structure built around equity, those decisions default to whatever biases exist in the data and design choices behind the tool. This lesson gives you the frameworks, vocabulary, and practical steps to build AI governance that actively supports DEI, not just avoids the worst outcomes.

7 Things You Need to Know Before We Start

1. AI governance means the policies, roles, and processes that control how AI tools are selected, used, monitored, and retired inside your organization, it's not just an IT concern.
2. Bias in AI doesn't require bad intentions. It emerges when training data reflects historical inequities, when design teams lack diversity, or when tools are deployed without testing on underrepresented groups.
3. The tools most likely to create equity risks in non-technical workplaces are resume screeners, performance scoring systems, sentiment analyzis tools, chatbots handling HR queries, and meeting transcription/summarization tools.
4. Governance without DEI integration is incomplete. Standard AI governance focuses on accuracy and legal risk. DEI-integrated governance adds demographic fairness, representation, and lived-experience accountability.
5. The EU AI Act (2024) classifies hiring and workforce management AI as 'high-risk', meaning organizations using these tools face legal compliance requirements, not just ethical ones.
6. You don't need a data science team to govern AI equitably. The most important roles are HR leaders, DEI officers, legal counsel, and frontline managers who can spot problems in real workflows.
7. Governance is a living process, not a one-time policy document. Tools change, data drifts, and workforce demographics shift, your governance structures need regular review cycles built in.

What AI Governance Actually Means for DEI Work

AI governance is the organizational system that decides which AI tools your company uses, how they're deployed, who's accountable when something goes wrong, and how performance is measured over time. Think of it like a building code for technology, not every employee needs to understand the engineering, but everyone needs to know the exits are clearly marked and the structure has been inspected. For DEI, governance is the mechanism that ensures AI tools don't quietly undermine the inclusion work your organization is investing in elsewhere.

The gap between standard AI governance and DEI-integrated AI governance is significant. Standard governance typically asks: Is this tool accurate? Is it secure? Does it comply with data privacy law? DEI-integrated governance adds a harder set of questions: Is it accurate equally across racial, gender, age, and disability groups? Were people from marginalized communities consulted during procurement? Does the tool reproduce outcomes we've already identified as inequitable? Answering these questions requires intentional structure, not goodwill alone.

• Procurement review: Evaluate AI tools for bias risk before purchase, not after deployment.
• Stakeholder mapping: Identify which employee groups are most affected by each tool and include their representatives in decisions.
• Fairness metrics: Define what 'working equitably' looks like for each tool, before you turn it on.
• Incident reporting: Create a clear channel for employees to flag AI outputs that feel biased or exclusionary.
• Audit schedule: Build in formal reviews every 6-12 months, or whenever the tool is updated by the vendor.
• Accountability assignment: Name a specific person or role responsible for each AI system's equity performance.
• Exit criteria: Define in advance the conditions under which a tool will be suspended or removed.

Click to enlarge

AI Governance Roles: Who Does What

The Three Layers of AI Bias Risk in the Workplace

Bias in workplace AI doesn't arrive in one form. It operates at three distinct layers, and each layer requires a different governance response. Understanding which layer a problem lives in tells you who needs to act and how fast. Organizations that treat all bias as a single problem tend to apply the wrong fix, or fix one layer while the others keep producing inequitable outputs.

The first layer is data bias, the AI was trained on historical data that reflects past discrimination. A hiring tool trained on 10 years of your company's hire decisions will learn to replicate whoever you hired before, including the patterns that excluded women from technical roles or promoted only certain demographics into leadership. The second layer is design bias, the people who built the tool didn't include diverse perspectives, so protected characteristics get encoded in proxy variables. The third layer is deployment bias, the tool works adequately in general but performs worse for specific groups in your specific context, something that only becomes visible after you're already using it.

1. Data bias. Fix requires: auditing what data the tool was trained on; requesting vendor demographic performance reports; using tools trained on more representative datasets.
2. Design bias. Fix requires: evaluating vendor team diversity during procurement; prioritizing tools with third-party fairness audits; avoiding tools that use names, zip codes, or graduation years as inputs (common proxies for race and class).
3. Deployment bias. Fix requires: piloting tools with diverse employee groups before full rollout; tracking outcomes by demographic segment after deployment; building a feedback loop so affected employees can report problems directly.

Click to enlarge

Prompt Example: Asking an AI Tool to Help Audit for Bias Risk

Building Your AI Inventory: Know What You're Governing

You cannot govern what you haven't mapped. Most organizations have more AI tools in active use than their leadership realizes, because departments purchase or activate AI features independently, often without central oversight. A marketing team enables Salesforce's AI scoring. HR activates the AI matching feature in their ATS. A manager starts using Copilot to summarize performance reviews. Each of these is an AI system making consequential decisions, and each carries equity risk. An AI inventory is the starting document for any governance effort.

Creating an AI inventory doesn't require a technology audit team. It requires a structured survey sent to department heads and team leads asking three questions: What AI tools or AI-powered features are your team currently using? What decisions do those tools influence or automate? Who approved their use? Most organizations discover 30-50% more AI tool usage than their central records show. That gap is your governance blind spot, and it's where the highest-risk deployments often hide, because they were adopted informally without any review process.

Click to enlarge

Part 1 Practice Task: Build a Starter AI Inventory for Your Organization

Part 1 Cheat Sheet: AI Governance Foundations for DEI

• AI governance = policies + roles + processes for how AI is selected, used, monitored, and retired.
• DEI-integrated governance adds demographic fairness, representation, and lived-experience accountability on top of standard accuracy and legal compliance.
• The 6 core governance roles: AI Governance Lead, DEI Integration Reviewer, Legal/Compliance Counsel, Frontline Manager Champion, Employee Feedback Representative, Vendor Accountability Contact.
• Three primary bias layers: Data Bias (what the AI learned from), Design Bias (who built it and how), Deployment Bias (how it performs in your specific context).
• Fourth bias layer to watch: Feedback Loop Bias, when biased human override decisions get fed back into the AI as 'correct' data.
• Proxy variables are the hidden bias risk: zip code (race), graduation year (age), employment gaps (disability/caregiving), extracurriculars (class).
• EU AI Act (2024) classifies hiring and workforce management AI as high-risk, legal compliance is now required, not optional.
• Start governance with an AI inventory: map every tool, every decision it influences, and who approved it.
• Shadow AI (unapproved tools and auto-enabled features) is often where the highest equity risks are hiding.
• Exit criteria matter: define before deployment the conditions under which a tool will be suspended or removed.
• Governance is a cycle, not a document, build in 6-12 month review periods and demographic performance tracking from day one.

Key Takeaways from Part 1

• AI tools in your organization are already making consequential decisions. Governance is how you ensure those decisions don't quietly undermine your DEI commitments.
• Bias operates at multiple layers, data, design, deployment, and feedback loops, and each layer requires a different response from a different person in your organization.
• A DEI-integrated AI governance structure requires named roles with specific accountabilities, not just a general policy commitment.
• Your AI inventory is the foundation of everything. You cannot govern tools you haven't mapped, and most organizations have more AI in active use than they realize.
• Legal exposure is real and growing. High-risk AI tools in employment contexts now fall under regulatory frameworks, and 'we didn't know the tool was biased' is not a defensible position.

Part 1 established why AI governance matters for DEI work. Now the harder question: what does it actually look like in practice? Governance without structure is just good intentions. This section gives you the frameworks, decision tools, and accountability mechanisms that turn policy into action, the kind your legal team, HR leadership, and frontline managers can all work from.

7 Things Every DEI Governance Leader Must Know

1. AI tools used in hiring, promotions, or performance reviews are subject to employment discrimination law in most jurisdictions, the same laws that govern human decisions.
2. "Neutral" algorithms are not automatically fair. A resume screener trained on historical hires can encode past biases and reproduce them at scale.
3. Vendor claims about fairness are not audits. Asking your software provider 'is this unbiased?' is not the same as independent testing.
4. Disparate impact, when a neutral-seeming process disproportionately excludes a protected group, can occur even without discriminatory intent.
5. AI governance for DEI requires three distinct roles: someone who owns the policy, someone who tests the tools, and someone who hears complaints.
6. Documentation is your legal and ethical shield. If you cannot show what data trained a model and who approved its use, you have a governance gap.
7. Employees and candidates have a right to know when AI is materially influencing decisions about them, and in some places, that right is now the law.

Defining the Scope: Where AI Touches DEI in Your Organization

Before you can govern AI for DEI, you need to map where AI is actually operating. Most organizations underestimate this. AI isn't just the resume screener your talent team bought last year. It's the sentiment analyzis tool your HR platform uses to flag "flight risks." It's the performance rating algorithm embedded in your HRIS. It's the chatbot answering employee questions about benefits. It's ChatGPT being used informally by a hiring manager to draft job descriptions. Each of these touchpoints carries DEI implications, and each needs a governance response proportional to its risk level.

A useful starting framework is the AI Impact Tier system. Tier 1 tools directly influence employment decisions: hiring screens, promotion scores, performance ratings, compensation recommendations. These carry the highest legal and ethical risk and need the most rigorous oversight. Tier 2 tools shape the environment without making direct decisions: employee engagement platforms, communication analyzis, learning recommendations. Tier 3 tools support administrative tasks with low DEI sensitivity: scheduling assistants, document formatting, internal search. Your governance intensity should match the tier, not treat every AI tool identically.

• Tier 1 (High Risk): Applicant tracking systems with AI scoring, automated video interview analyzers, AI-driven performance management, pay equity algorithms
• Tier 2 (Moderate Risk): Employee engagement sentiment tools, AI-generated learning paths, internal mobility recommendation engines, AI chatbots handling HR queries
• Tier 3 (Lower Risk): Meeting transcription and summarization, AI writing assistants for internal documents, scheduling automation, benefits FAQ chatbots
• Unmanaged informal use: Employees using personal ChatGPT or Claude accounts for work tasks, often invisible to governance systems but still carrying risk

Click to enlarge

Building the Accountability Structure: Who Owns What

The most common failure in AI governance for DEI isn't bad intentions, it's unclear ownership. When everyone is responsible, no one is. An AI hiring tool causing disparate impact doesn't get caught because HR thinks IT owns it, IT thinks the vendor owns it, and the vendor says it's working as designed. Effective governance requires naming specific people to specific roles before a problem occurs, not after. This isn't about creating bureaucracy. It's about having a phone number to call when something goes wrong.

Three core roles anchor a functional DEI AI governance structure. The AI Policy Owner, typically a senior HR, DEI, or Legal leader, sets the standards and approves tool deployment. The DEI Technology Reviewer, often a DEI director or external consultant, evaluates tools for equity risk before and after launch. The Employee Advocate or Ombudsperson, an independent role, receives and investigates complaints from employees who believe AI influenced a decision unfairly. Smaller organizations can consolidate these roles, but all three functions must exist. Skipping the advocate function is the most common and most consequential omission.

1. Name an AI Policy Owner with authority to pause or reject tool deployments, this role needs real power, not just a title.
2. Create a DEI Technology Review checklist that every new AI tool must pass before going live in any people-related process.
3. Establish a clear complaint pathway: employees must know how to report concerns about AI-influenced decisions, and someone must be required to respond.
4. Set review cycles, not just at deployment, but at 90 days, 6 months, and annually, plus any time the tool updates or the vendor changes its model.
5. Define escalation criteria: what outcomes trigger an immediate review? (e.g., a 15% or greater disparity in outcomes by demographic group)
6. Document every decision: who approved the tool, what data it was tested on, what the test results showed, and who signed off.
7. Include frontline managers in governance, they are often the first to notice when AI outputs feel wrong, but rarely have a formal channel to report it.

Click to enlarge

Using AI Tools to Strengthen DEI Governance (Not Just Monitor It)

Vendor Accountability: What to Demand Before You Sign

Most organizations treat AI vendor selection as a procurement decision. It's also a DEI decision, and one with legal consequences. When a vendor's hiring algorithm produces discriminatory outcomes, the EEOC's position is clear: employer liability does not transfer to the vendor. You own the outcome of every AI-assisted decision made under your roof, regardless of who built the tool. This means due diligence before contract signing is not a nice-to-have. It's risk management.

Vendors often resist detailed fairness inquiries, citing proprietary technology. That resistance is itself informative. A vendor with nothing to hide will provide training data descriptions, demographic performance breakdowns, and audit methodology. They will name the specific fairness metrics their tool is evaluated against, equal opportunity rate, demographic parity, calibration across groups. If a vendor cannot or will not answer your DEI due diligence questions in writing, that is your answer. Move to a vendor who can. The market is competitive enough that you have options.

Click to enlarge

Part 2 Cheat Sheet

• Map before you govern: run an AI inventory across all departments before writing policy, you will find tools you didn't know about.
• Use the three-tier system: Tier 1 (employment decisions) = highest risk and most oversight; Tier 2 (environment shaping) = moderate; Tier 3 (admin support) = lowest.
• Name three specific roles: Policy Owner, DEI Technology Reviewer, and Employee Advocate, all three must exist in some form.
• Disparate impact is your legal exposure: it doesn't require intent, just a neutral-seeming process that disproportionately harms a protected group.
• Vendor responsibility doesn't transfer to vendors: you own the outcomes of AI-assisted decisions made in your organization.
• Demand specifics from vendors: training data description, third-party audit results, demographic performance breakdowns, and a complaint process.
• 'Bias-free' is a red flag, not a reassurance, any vendor using that phrase cannot be taken at face value.
• NYC Local Law 144 is the most specific US employment AI law in force, annual independent audits, public results, candidate notification required.
• AI tools can strengthen DEI governance too: use ChatGPT or Claude to audit job descriptions, review policy language, and draft transparency communications.
• Documentation is your protection: record what data trained each tool, who approved it, what testing showed, and who signed off.

Key Takeaways from Part 2

• Governance without clear ownership fails, name specific people to specific roles before a problem occurs.
• Your AI footprint is larger than you think. Informal use of tools like ChatGPT by individual managers is part of your governance challenge.
• Vendor due diligence is a DEI responsibility, not just a procurement checklist. The questions you ask before signing determine what accountability you can enforce after.
• Proportionality matters: govern Tier 1 tools with maximum rigor, Tier 3 with lighter touch, treating everything identically wastes resources and misses real risks.
• AI tools are also governance allies: use them actively to audit language, review policies, and draft transparency communications, not just as risks to manage.

Equitable AI governance isn't a compliance checkbox. It's an ongoing operational discipline, one that requires clear ownership, documented processes, and regular audits. The organizations getting this right treat AI governance the same way they treat financial controls: structured, assigned, and reviewed on a schedule. Here's what you need to know to make that real in your workplace.

1. AI governance needs a named owner, not just a committee that meets twice a year.
2. Bias in AI outputs often reflects bias in the data used to train or prompt the system.
3. Your AI use policy must explicitly address protected characteristics: race, gender, age, disability, religion.
4. Vendor accountability matters, ask suppliers how their models were tested for fairness before you buy.
5. Employees need a clear channel to report AI outputs that feel discriminatory or exclusionary.
6. High-stakes decisions (hiring, performance reviews, promotions) require human sign-off, not just AI output.
7. Governance documents are living artifacts, review them at minimum annually, or whenever a new AI tool is adopted.

Assigning Accountability: Who Owns AI Equity?

Most organizations have someone who owns data privacy, someone who owns workplace safety, and someone who owns financial compliance. Very few have someone who explicitly owns AI equity. That gap is where harm enters. When no one is accountable, bias complaints go unanswered, vendors go unvetted, and AI tools quietly shape decisions in ways no one has reviewed. Assigning AI equity ownership doesn't require hiring a new person, it requires formally adding it to an existing role, like Chief People Officer, DEI Director, or Chief Compliance Officer.

Ownership means more than a title. It means this person has the authority to pause or remove an AI tool if equity concerns arise, the budget to conduct periodic audits, and a direct line to leadership when systemic issues are identified. Pair the owner with a small cross-functional review group. HR, Legal, IT, and at least one frontline employee representative. That group should meet quarterly, review flagged outputs, and sign off on any new AI tool before it touches people-related decisions.

• Name one primary AI equity owner with explicit authority to escalate or halt tool use.
• Document the owner's responsibilities in a written charter, not just a job description update.
• Include at least one non-manager employee voice in the review group.
• Set a quarterly meeting cadence with a standing agenda: flagged outputs, new tools, policy updates.
• Require the owner to report to senior leadership at least twice per year.
• Store all governance decisions and rationale in a shared, accessible location, not someone's inbox.

Auditing AI Outputs for Bias: A Practical Approach

An AI audit doesn't require a data scientist. It requires intentional, structured testing by people who know your workplace. The goal is to catch patterns, not just one-off odd outputs. Run the same prompt with different demographic signals and compare outputs. Ask ChatGPT or Claude to write a performance review for 'Michael' and then for 'Lakisha' in the same role with identical performance details. If the tone, language, or content differs, you have a bias signal worth investigating.

Document every audit. Record the prompt, the output, the date, the tool version, and who reviewed it. Over time, patterns become visible, and documentation protects your organization if a bias complaint arises. Share audit summaries with your governance group quarterly. If a tool consistently produces biased outputs despite adjusted prompts and settings, escalate to your AI equity owner for a formal tool review. The bar for people-related decisions should be higher than for content generation tasks.

1. Select 3-5 high-stakes use cases where AI touches people decisions: hiring screens, performance summaries, promotion recommendations.
2. Design paired test prompts, identical scenarios, different names or demographic signals.
3. Run tests in ChatGPT, Claude, or whichever tool your team uses.
4. Record outputs verbatim in a shared document or spreadsheet.
5. Compare outputs for tone, length, word choice, and recommendations.
6. Flag any differences and note whether they could disadvantage a protected group.
7. Submit findings to your governance group with a recommended action: accept, adjust prompt, or escalate.

Embedding Equity Into AI Procurement and Policy

Every AI tool your organization buys is a governance decision. Before signing a contract or approving an enterprise tool, your procurement process should include equity due diligence. Ask vendors directly: How was this model tested for demographic bias? What protected characteristics were evaluated? Is there a published fairness report or third-party audit? Vendors who can't answer these questions clearly should not be trusted with people-related use cases. Build these questions into your standard vendor evaluation scorecard.

Internal AI policy needs the same rigor. A policy that simply says 'use AI responsibly' is not a governance document, it's a liability. Effective AI equity policy specifies which use cases are approved, which require human review, which are prohibited entirely, and what the escalation path looks like. It names who can authorize new tools, sets a review cadence, and includes explicit language about protected characteristics. Keep it to 1-2 pages. Long policies don't get read. Short, specific ones do.

Governance Quick-Reference Cheat Sheet

• Name one AI equity owner with escalation authority, don't leave it to a committee.
• Require human sign-off on any AI-assisted decision involving hiring, promotion, pay, or discipline.
• Run paired bias tests quarterly on your highest-stakes AI use cases.
• Document every audit: prompt, output, date, tool, reviewer, action taken.
• Ask vendors three questions before buying: How was bias tested? What demographics were evaluated? Is there a third-party audit?
• Write your AI policy in plain language, one to two pages maximum.
• Explicitly name protected characteristics in your policy: race, gender, age, disability, religion, national origin.
• Review your AI policy annually and whenever a new tool is adopted.
• Create a clear, named channel for employees to report biased AI outputs.
• Store all governance documents centrally, not in personal email or a single person's drive.

Key Takeaways

• Equitable AI governance requires named ownership, not just good intentions.
• Bias audits are accessible to non-technical professionals using free AI tools and structured test prompts.
• High-stakes people decisions, hiring, performance, promotion, must include mandatory human review.
• AI procurement is a governance decision: ask vendors about fairness testing before you buy.
• An AI equity policy must explicitly address protected characteristics to provide legal and ethical protection.
• Governance is a cycle, not a one-time event, schedule regular reviews and treat the policy as a living document.
• Documentation of audits and decisions is your organization's evidence of due diligence.