Stay Ahead of Compliance and Trust

Regulatory and Ethical AI in Finance

Part 1: When AI Gets It Wrong, and Who Pays the Price

This case captures the central tension of AI in finance. The tools are genuinely powerful. They process credit applications in seconds, flag fraud before it clears, and surface investment opportunities that human analyzts would miss. But they also inherit the biases baked into historical data, operate in ways that even their creators struggle to explain, and make decisions at a scale and speed that makes human oversight extremely difficult. Goldman's compliance team wasn't negligent. Their legal team wasn't asleep. The problem was that no one had asked the model to show its work, and no regulatory framework at the time required them to.

That regulatory gap is closing fast. The EU AI Act, which began phasing in from 2024, classifies credit scoring and financial risk assessment as high-risk AI applications, subject to mandatory transparency requirements and human oversight rules. The U.S. Consumer Financial Protection Bureau issued guidance in 2023 stating that lenders cannot use "black box" AI models as the sole basis for adverse credit decisions, they must be able to explain the reasons to applicants. If you work in financial services, insurance, lending, investment advisory, or any adjacent field, these rules affect what tools you can use, how you can use them, and what you have to document when things go wrong.

The Principle Behind the Story

The Goldman case illustrates a principle that runs through every AI ethics conversation in finance: AI models don't discriminate on purpose, but they can discriminate at scale. A human loan officer who harbored bias affected dozens of applicants a year. An AI model running the same bias affects millions of applications a month, silently, consistently, and with a veneer of mathematical objectivity that makes the bias harder to challenge. This is what regulators mean when they talk about "algorithmic accountability", the idea that the organizations deploying AI are responsible for its outcomes, not just its intentions. The model being the decision-maker is not a legal defense. It's a liability.

For non-technical professionals, this principle has a very practical implication. You don't need to understand how a neural network weights its inputs. But you do need to understand what your AI tool is being asked to decide, what data it's drawing on, and whether a human with relevant expertise is reviewing outcomes before they affect real people or real money. These are management and governance questions, not engineering questions. The compliance officer, the HR director approving AI-assisted hiring in financial services, the marketing manager using AI to segment loan product audiences, all of these people are inside the regulatory perimeter now.

How Bias Travels from Data to Decision

Think about how a credit-scoring AI gets built. A team feeds it millions of past loan records: who applied, who was approved, who repaid, who defaulted. The model finds patterns. If historically certain zip codes had higher default rates, zip codes that, due to decades of redlining, correlate with race, the model learns to penalize applicants from those areas. It never sees race directly. It doesn't need to. Zip code, purchasing patterns, the type of phone used to apply, these become proxies. The model isn't racist in any intentional sense. It's a very accurate mirror of a racist history, and accuracy is exactly the problem.

This is called proxy discrimination, and it's one of the most common ways AI produces biased outcomes in finance. The same dynamic plays out in insurance pricing, fraud detection, and investment product recommendations. An insurance AI trained on claims data might charge higher premiums to people who buy their policies online rather than through a broker, not because online buyers are riskier, but because the historical data reflects that lower-income customers disproportionately used online channels, and lower-income customers had more claims. The pattern is real. The causal story is wrong. And the AI doesn't know the difference.

When you use an AI tool to assist with financial decisions, even something as simple as asking ChatGPT or Microsoft Copilot to help draft a client risk profile, or using a vendor's AI scoring tool for SME loan applications, you are touching this territory. The question you need to ask isn't just "Is this output accurate?" It's "What assumptions are baked into this output, and could those assumptions disadvantage a protected class of people?" That question doesn't require a data science degree. It requires professional judgment, which is exactly why regulators are insisting that humans stay in the loop.

Click to enlarge

A Different Industry, the Same Problem: Insurance in the UK

Admiral Insurance, one of the UK's largest car insurers, made headlines in 2016 when it announced a product called FirstRate, an AI tool that would analyze Facebook posts to assess the personality traits of young drivers and adjust their premiums accordingly. Careful, organized writers would get lower rates. Impulsive, overconfident social media behavior would cost more. The UK's Information Commissioner's Office stepped in before the product launched, ruling that Facebook's terms of service didn't allow personal data to be used this way without explicit consent. Admiral pulled the product. But the underlying idea, using AI to find unconventional data signals for financial risk assessment, didn't go away. It went underground, into less visible product features and vendor tools that aren't publicly announced.

The Admiral case shows that the ethical and regulatory risks of AI in finance aren't limited to lending. Insurance pricing, fraud scoring, investment suitability assessments, any time an AI model is making or heavily influencing a financial decision about a person, the same questions apply: What data is it using? Has that data been consented to? Could it produce outcomes that are unfair to protected groups? And critically: does the person affected have any way to understand or challenge the decision? For professionals in these fields, the practical takeaway isn't to avoid AI tools. It's to ask these questions of every vendor you work with, every tool your firm deploys, and every workflow where AI output influences a consequential decision.

The Compliance Officer's Dilemma: A Story from Asset Management

A compliance officer at a mid-sized asset management firm in Chicago, let's call her Sarah, faced a practical version of this problem in late 2023. Her firm's portfolio managers had started using ChatGPT Plus to help draft client investment commentary: the quarterly letters that go out to high-net-worth clients explaining portfolio performance and outlook. The quality improved noticeably. The letters were clearer, better structured, and took a fraction of the time to produce. Then Sarah's phone rang. A client's attorney was asking whether the investment commentary their client received constituted regulated investment advice, and if so, whether it had been produced by a system that met the SEC's recordkeeping requirements for electronic communications. Sarah didn't have a clean answer.

The SEC's recordkeeping rules require that all business-related electronic communications at registered investment advisers be archived and available for examination. ChatGPT conversations, unless routed through a compliant enterprise system, typically don't meet that standard. Sarah's firm hadn't broken any rules yet, no enforcement action was taken, but the conversation forced a policy decision. They moved to Microsoft Copilot for Microsoft 365, which integrates with their existing compliance archiving system, and established a formal policy: AI can draft, a human portfolio manager must review and approve, and the final approved text is what gets archived. The AI's draft is a working document, not a business record. This kind of policy, specific, documented, auditable, is exactly what regulators are asking for.

Putting It Into Practice: What Governance Actually Looks Like

The stories above. Goldman, Admiral, Sarah's asset management firm, aren't cautionary tales about avoiding AI. They're blueprints for what to build around AI use. In each case, the problem wasn't the tool. It was the absence of a governance structure: documented policies about what the tool can be used for, who reviews its outputs, how decisions get recorded, and what happens when something goes wrong. Building that structure doesn't require a technology team. It requires the people who actually use the tools, compliance officers, team managers, department heads, to ask a set of deliberate questions before a new AI workflow goes live.

Start with the decision. Ask: what is this AI tool actually deciding, or heavily influencing? If the answer involves a financial outcome for a client or customer, a rate, a limit, a recommendation, a denial, you're in regulated territory. Document that. Then ask: who is reviewing the AI's output before it becomes final? That reviewer needs to have the expertise to catch errors and the authority to override the AI. "The AI said so" is not an adequate compliance record. The reviewer's name, their assessment, and the date of their review should be in the file.

Finally, ask: what happens when we get it wrong? AI systems make errors. Models drift over time as the world changes but the training data doesn't. A fraud detection model trained in 2020 may perform poorly on 2024 transaction patterns. Your governance structure needs a feedback loop: a regular review of AI-assisted decisions to check for patterns of error, a process for clients or customers to flag incorrect outcomes, and a clear escalation path when the AI produces something that doesn't pass the smell test. None of this is exotic compliance infrastructure. It's the same discipline you'd apply to any high-stakes business process, with the added urgency that AI operates at a scale and speed where errors compound quickly.

Key Lessons from Part 1

1. AI bias in finance is usually not intentional, it's inherited from historical data that reflected past discrimination. The model doesn't need to 'see' race or gender to discriminate by proxy.
2. Regulatory frameworks are now treating credit scoring, insurance pricing, and investment recommendations as high-risk AI applications, with mandatory transparency and human oversight requirements.
3. The CFPB requires lenders to provide plain-language explanations for AI-assisted adverse decisions. 'The model said no' is not a legally adequate reason.
4. The EU AI Act, phasing in through 2027, creates a compliance obligation for any firm operating in European markets, regardless of where the AI tool was built.
5. Archiving and recordkeeping requirements apply to AI-assisted client communications at regulated firms. Not all AI tools meet these requirements by default.
6. Governance isn't a technology problem, it's a management problem. The key questions (what is it deciding, who reviews it, what gets documented) are answered by business professionals, not engineers.
7. AI errors compound at scale. A human making a biased decision affects a handful of people. An AI model running the same bias affects millions, silently and consistently.

When Algorithms Make Lending Decisions: The Bias Problem in Plain Sight

In 2019, Apple launched the Apple Card with Goldman Sachs as its banking partner. Within days, Twitter lit up with complaints. Tech entrepreneur David Heinemeier Hansson posted that his wife received a credit limit twenty times lower than his, despite having a higher credit score. Apple co-founder Steve Wozniak reported the same pattern with his wife. Goldman Sachs insisted the algorithm treated everyone equally and used no gender data. But "no gender data" is not the same as "no gender bias." The algorithm had been trained on historical credit data, data generated in a world where women historically had less access to credit, lower incomes on average, and different borrowing patterns. The bias was baked into the training data before a single line of code was written.

New York's Department of Financial Services launched an investigation. Goldman Sachs faced intense public scrutiny. The episode crystallized something that compliance officers, risk managers, and product leaders in financial services now have to grapple with directly: AI systems can discriminate without anyone intending discrimination. The Equal Credit Opportunity Act (ECOA) prohibits lending decisions based on sex, race, national origin, and several other protected characteristics. But ECOA was written for human decision-makers. When an algorithm makes ten million credit decisions a day, auditing for compliance becomes a fundamentally different challenge, and "the algorithm decided" is not a legal defense.

The principle the Apple Card episode forces into focus is this: AI in finance inherits the inequities of the past unless someone actively works to prevent it. The data that trains these models reflects decades of human decisions, some discriminatory, some structurally unfair, some just shaped by unequal access to opportunity. If you feed that data into a model uncritically, the model learns those patterns and reproduces them at scale. For finance professionals using AI tools, whether for credit analyzis, hiring, fraud detection, or customer segmentation, understanding this inheritance is not optional. It is a core competency.

The Explainability Problem: Why "The Model Said So" Isn't Enough

Imagine you're a loan officer at a regional bank. A small business owner applies for a $250,000 expansion loan. Your AI-assisted underwriting tool flags the application as high risk and recommends denial. The business owner asks why. You open the tool's interface and see a risk score of 67 out of 100, but no breakdown of which factors drove that score. You cannot explain the decision to the applicant. More importantly, under the Fair Credit Reporting Act and Regulation B, you are legally required to provide adverse action notices that give specific, intelligible reasons for credit denial. "The algorithm gave a low score" does not satisfy that requirement.

This is the explainability problem, and it sits at the intersection of ethics, law, and practical business operations. Many of the most powerful AI models used in finance, particularly deep learning models, operate as what researchers call "black boxes." They produce accurate predictions, but the internal reasoning is opaque even to their creators. Regulators in the EU have taken the hardest line on this: the AI Act classifies credit scoring as a high-risk AI application, requiring providers to ensure transparency, human oversight, and the ability to explain outputs. The UK's Financial Conduct Authority has issued similar guidance. In the U.S., the CFPB has signaled that explainability is a live compliance issue, not a future concern.

For finance professionals, the practical takeaway is not to avoid AI tools but to demand explainability as a procurement criterion. When your organization evaluates an AI underwriting tool, fraud detection system, or investment screening platform, explainability should be a non-negotiable feature, not an upgrade tier. Ask vendors specifically: Can this system produce human-readable explanations for individual decisions? Can those explanations hold up to regulatory review? If a vendor cannot answer those questions clearly, that is a significant red flag. The tools that will survive regulatory scrutiny are the ones built with explainability as a design principle, not an afterthought.

Healthcare Finance Shows What Biased Training Data Costs in Practice

The bias problem is not unique to consumer lending, it runs through any sector where AI makes high-stakes recommendations based on historical data. In 2019, researchers published a landmark study in Science exposing a widely used healthcare algorithm that systematically underestimated the medical needs of Black patients. The algorithm used healthcare spending as a proxy for health needs. Because systemic inequities meant Black patients had historically spent less on healthcare, not because they were healthier, but because they had less access, the algorithm concluded they needed less care. Optum, the company behind the algorithm, acknowledged the findings and committed to changes. The algorithm had been used by major insurers and hospital systems affecting tens of millions of patients.

The financial services parallel is direct. When AI tools use proxies, zip code, educational history, social connections, purchase patterns, they can encode racial or socioeconomic bias without ever asking for race or income directly. This is sometimes called "proxy discrimination." A credit model that penalizes applicants who shop at certain stores, live in certain neighborhoods, or have certain employment histories may be producing racially disparate outcomes through entirely race-neutral-looking inputs. For finance professionals, this means you cannot evaluate an AI tool's fairness simply by checking whether protected characteristics appear in the input data. You have to look at outcomes, and that requires testing, not just trust.

Comparing Regulatory Approaches: What Finance Teams Need to Know

Click to enlarge

A Human Resources Director Learns That Hiring AI Has a Compliance Footprint Too

Consider the position of a Head of Talent Acquisition at a global investment bank. Her team processes over 40,000 job applications a year. In 2022, her firm piloted an AI resume screening tool to handle first-round filtering. Six months in, the firm's legal team flagged a problem: the tool had been trained on ten years of successful hires at the firm. The firm's historical hiring had skewed heavily toward candidates from a small set of elite universities, a pattern that, when mapped against demographic data, correlated strongly with race and socioeconomic background. The tool had learned to replicate that pattern. Amazon famously scrapped a similar tool in 2018 after discovering it systematically downranked resumes that included the word "women's", as in "women's chess club." The lesson repeated itself, at scale, across the industry.

The compliance implication for finance firms is significant. In the U.S., AI hiring tools fall under Title VII of the Civil Rights Act. The EEOC issued guidance in 2023 specifically addressing algorithmic discrimination in hiring, stating that employers remain liable for discriminatory outcomes even when those outcomes are produced by a vendor's tool. New York City's Local Law 144, which took effect in 2023, requires employers using AI hiring tools to conduct annual bias audits and publish the results. For finance professionals in HR, talent, or people operations roles, this is a live compliance obligation, not a hypothetical future risk. Auditing the AI tools your firm uses for hiring is now part of the job.

Turning Ethical Principles into Practical AI Policy

Ethics frameworks for AI, fairness, accountability, transparency, explainability, can sound abstract until you connect them to specific decisions your team makes every week. Take transparency. In practice, transparency means your team knows which AI tools are being used, what data they process, and what decisions they influence. It means customers know when an AI has assessed their creditworthiness. It means your board or audit committee can get a clear answer to "what AI are we using and how are we governing it?" Most firms that have faced regulatory action over AI were not operating in bad faith, they simply had no governance structure that made these questions answerable. Transparency starts with documentation, not philosophy.

Accountability is equally concrete. It means someone in your organization owns the answer to "what happens when this AI tool gets it wrong?" Not the vendor. Not "the algorithm." A named person or team who reviews exceptions, handles complaints, and has the authority to override the system. The EU AI Act calls this a "human oversight" requirement for high-risk applications. But even where it is not legally mandated, having a clear escalation path for AI-driven decisions is basic operational hygiene. For finance teams, this might mean designating a model risk officer, ensuring loan officers can flag and override AI recommendations, or establishing a formal review process for AI-generated investment signals before they reach clients.

Fairness in practice means building testing into your workflow, not just at deployment, but continuously. Models drift. The world changes. A model trained on pre-pandemic lending data may behave differently in a post-pandemic economy. A fraud detection model trained on one geography may perform poorly in another. Ongoing monitoring is not a technical nicety, it is what separates firms that catch problems early from firms that discover them in a regulatory examination or a news story. Finance professionals do not need to run the tests themselves, but they do need to ask their vendors and data teams: how often is this model's fairness performance reviewed, and who sees the results?

Key Principles from Part 2

1. AI systems inherit historical bias from their training data, even when protected characteristics are not explicit inputs. The Apple Card case shows that "no gender data" does not guarantee gender-neutral outcomes.
2. Explainability is a legal requirement in financial services, not just an ethical preference. Adverse action notices under Regulation B require specific, intelligible reasons, "the model scored you low" does not comply.
3. Proxy discrimination is real and common. AI tools that use zip code, purchase history, or educational pedigree as inputs can produce racially disparate outcomes through entirely race-neutral-looking variables.
4. Regulatory frameworks vary significantly by jurisdiction. Finance teams operating internationally must map their AI tools against the EU AI Act, CFPB guidance, FCA principles, and MAS FEAT simultaneously.
5. Hiring AI carries its own compliance footprint. Employers, not vendors, bear liability for discriminatory outcomes from AI hiring tools under U.S. law, and New York City now requires annual bias audits.
6. Transparency means documentation and disclosure, not just intention. A firm that cannot answer 'what AI are we using and how are we governing it?' is already exposed to regulatory risk.
7. Accountability requires a named human owner for AI-driven decisions, someone who can review exceptions, handle complaints, and override the system when needed.
8. Continuous monitoring is not optional. Models drift over time, and a tool that was fair at deployment may produce biased or inaccurate outputs months later without anyone noticing.

When the Algorithm Gets It Wrong: Accountability in AI-Driven Finance

In 2023, the Dutch financial regulator AFM investigated a consumer lending platform that had quietly embedded an AI credit-scoring model into its approval pipeline. The model had been trained on historical loan data, data that reflected decades of lending bias. The result: applicants from certain postal codes were denied credit at nearly twice the rate of demographically similar applicants elsewhere. Nobody had programmed discrimination into the system. The bias emerged from patterns in old decisions, and the AI faithfully reproduced them at scale. By the time the regulator intervened, tens of thousands of applicants had been affected.

What made this case particularly instructive was the company's initial defense: the model was a third-party tool, and no human had made the discriminatory decisions. The regulator disagreed. Under EU financial services law, outsourcing a decision to an algorithm does not outsource the legal and ethical responsibility for that decision. The firm, not the vendor, was accountable. This is the core tension that every finance professional using AI tools must understand: AI can scale your capabilities, but it also scales your accountability.

The principle extracted from this case is straightforward and non-negotiable. When AI touches a consequential financial decision, a credit denial, a risk flag, a compliance report, a human professional remains responsible for the outcome. That is not a limitation of AI. It is a feature of a functioning financial system. Your job is not to be suspicious of AI tools, but to use them with the same professional judgment you would apply to any other input, whether that is a junior analyzt's spreadsheet or a consultant's recommendation.

The Explainability Problem, and Why It Matters for Your Monday Morning

Goldman Sachs faced a pointed public moment in 2019 when the Apple Card, issued through Goldman, was accused of offering significantly lower credit limits to women than to men, including in cases where spouses shared finances. Goldman's response was that the algorithm made the decisions based on creditworthiness factors, not gender. The problem was that nobody outside the firm could verify that claim. The model was a black box. Regulators and customers could see the outputs but not the reasoning. The New York Department of Financial Services opened a formal investigation.

This is the explainability problem. A model that cannot show its work is a liability in regulated industries, not because the model is necessarily wrong, but because you cannot defend what you cannot explain. When you use AI tools like ChatGPT or Claude to draft compliance summaries, flag risks in contracts, or analyze portfolio exposure, you face a smaller version of this same challenge. If a client or regulator asks why you flagged a particular risk, 'the AI said so' is not an answer. You need to be able to articulate the reasoning yourself.

The practical solution is to treat AI output as a first draft of your thinking, not a final answer. When Claude summarizes a regulatory document and highlights three compliance gaps, your job is to read those gaps, verify them against the source, and be able to explain them in your own words. This takes fifteen minutes instead of three hours. That is the genuine productivity gain. But skipping the verification step, treating the AI summary as the document itself, is where professionals create real professional and legal exposure.

A Healthcare Parallel: What Finance Can Learn from Medicine's AI Mistakes

In 2019, a widely-used hospital algorithm in the United States was found to systematically underestimate the health needs of Black patients. The algorithm used healthcare spending as a proxy for health complexity, a reasonable-sounding shortcut that failed because, historically, less money had been spent on Black patients, not because they were healthier. The researchers who uncovered this estimated the bias affected millions of patients annually. The healthcare sector's response, mandatory bias audits, diverse training data requirements, explainability standards, mirrors exactly what financial regulators are now requiring for AI in credit and investment.

The parallel is direct and instructive for finance professionals. Any AI model trained on historical financial data inherits the biases embedded in that history. If your firm's past lending decisions, investment recommendations, or risk assessments reflected conscious or unconscious bias, an AI trained on that data will reproduce those patterns, efficiently and at scale. You do not need to be a data scientist to ask the right questions: Who was this model trained on? Has it been tested for demographic fairness? Can the vendor show you an independent audit?

The Fund Manager Who Built Trust by Showing the Work

A boutique asset management firm in Singapore began using AI tools in 2022 to generate preliminary ESG risk summaries for prospective portfolio companies. Rather than presenting these summaries to clients as finished analyzis, the lead portfolio manager developed a practice of sharing both the AI-generated draft and her own annotated version, showing where she agreed, where she corrected the AI, and where the AI had flagged something she would have missed. Clients reported higher confidence in the firm's process, not lower. The visible human judgment made the AI output more credible, not less.

This is the accountability model that regulators and ethicists are pointing toward: AI as a transparent input into a human professional's reasoning, not a replacement for it. The portfolio manager's approach also created a practical audit trail, if a client or regulator ever questioned a recommendation, the documentation showed exactly how the AI output had been evaluated and what human judgment had been applied. That documentation is increasingly not optional. Under frameworks like the EU AI Act and the SEC's 2023 guidance on AI in investment advice, firms are expected to maintain records of how AI tools influenced consequential decisions.

Putting It Into Practice: Your Ethical AI Checklist

The professionals who use AI most effectively in finance are not the ones who trust it most blindly, they are the ones who have developed a consistent, fast verification habit. Before any AI-assisted output leaves your desk, three questions should be automatic: Did I check the AI's claims against the source? Could I explain this reasoning to a regulator or client without referencing the AI? Does this output affect anyone's access to financial products or services? If the answer to any of these requires more work, do that work. The AI has already saved you time, use some of it on verification.

Data privacy is the second practical pillar. Financial data is among the most sensitive personal data that exists. When you paste client account details, loan applications, or investment portfolios into a public AI tool like the free version of ChatGPT, that data may be used to train future models, depending on the platform's current privacy settings. ChatGPT Plus and Claude Pro both offer options to turn off training on your inputs. Microsoft Copilot for Microsoft 365, deployed through an enterprise license, keeps your data within your organization's environment by default. Know which tool you are using and what its data handling terms actually say.

Finally, treat AI regulation as a living document, not a one-time briefing. The EU AI Act, SEC guidance, and CFPB positions on algorithmic decision-making are all evolving in 2024 and 2025. The professionals who stay ahead of this are reading the primary sources, not waiting for their compliance department to issue a memo eighteen months later. Set a quarterly reminder to check for updates from your relevant regulator's website. Thirty minutes a quarter keeps you current. That is a reasonable investment for tools that are now touching your most consequential professional outputs.

1. Accountability does not transfer to algorithms. When AI influences a financial decision, the human professional and their firm remain legally and ethically responsible for the outcome.
2. Explainability is a professional requirement, not a technical nicety. If you cannot explain the reasoning behind an AI-assisted output in your own words, you are not ready to act on it.
3. Historical data carries historical bias. Any AI model trained on past financial decisions may reproduce the inequities embedded in those decisions, at greater scale and speed.
4. Transparency builds trust. Showing clients and colleagues how AI was used, and where human judgment was applied, increases confidence in your work rather than undermining it.
5. Data privacy is your responsibility, not the tool's. Know which AI platforms you are using, read their data handling terms, and never paste sensitive client data into tools without enterprise-level privacy protections.
6. Regulation is accelerating. The EU AI Act, SEC guidance, and CFPB algorithmic fairness standards are all active and evolving. Staying current is a professional obligation, not optional continuing education.

Click to enlarge