Real World AI Failure Examples That Triggered Lawsuits

Real world AI failure examples are no longer hypothetical. Documented cases from 2020 through 2025 show AI systems causing wrongful arrests, triggering legal liability for corporations, and surfacing dangerous misinformation to millions of users simultaneously. Each incident followed a predictable structure: confident output, no human check, material harm. Together, they sketch a taxonomy of deployment failure that practitioners can act on now.

The AI Incident Database, now maintained by the Responsible AI Collaborative after being incubated at Partnership on AI, catalogs harms from deployed AI systems using a framework modeled on aviation accident databases and the Common Vulnerabilities and Exposures (CVE) system. Its indexed reports span law enforcement, healthcare, transportation, hiring, and content moderation. The throughline across all of them is that failures tend to cluster where model confidence is highest and human review is lowest.

Facial Recognition: From Blurry Surveillance Image to Wrongful Arrest

In January 2020, Robert Williams was arrested in his driveway in Farmington Hills, Michigan, in front of his wife and children. Detroit police had submitted a blurry surveillance still from a 2018 Shinola store robbery to Michigan State Police facial recognition software. The system returned Williams as a match. No detective independently verified the hit before the arrest warrant was issued.

Williams spent 30 hours in a cell. He was not near the store on the day of the robbery.

The case, Williams v. City of Detroit, was litigated by the ACLU and settled on June 28, 2024 — the first U.S. settlement stemming from a documented facial recognition false arrest. Under the settlement, the Detroit Police Department must now corroborate any facial recognition result with independent evidence before making an arrest, train officers on the technology’s elevated error rates for people with darker skin tones, and audit all cases since 2017 in which face recognition was used to obtain a warrant.

Amazon placed a one-year moratorium on law enforcement sales of its Rekognition software following the case. The underlying failure mode: algorithmic output treated as identification rather than as a lead requiring verification.

Chatbot Liability: When AI Advice Creates Legal Obligations

Air Canada’s customer-service chatbot told Jake Moffatt he could apply for a bereavement fare discount after purchasing a ticket — contradicting the airline’s actual policy, which required pre-purchase applications. Moffatt booked the ticket, flew to his grandmother’s funeral, and then applied for the discount, which Air Canada denied.

In court, the airline argued the chatbot was “a separate legal entity” responsible for its own statements. The British Columbia Civil Resolution Tribunal rejected that argument on February 14, 2024. Tribunal member Christopher Rivers held that Air Canada had an obligation to “take reasonable care to ensure their representations are accurate and not misleading.” The airline was ordered to pay Moffatt approximately $650 CAD in damages plus interest and filing fees.

The ruling established that organizations deploying customer-facing AI are liable for output those systems generate, regardless of the system’s autonomous behavior. For legal and compliance teams, Moffatt v. Air Canada is now cited as the binding precedent that client-facing chatbots are not carved out from standard misrepresentation law.

For context on how chatbot prompt injection and jailbreaks can further warp outputs beyond their intended scope, aisec.blog covers the offensive security angle in depth.

Search Hallucinations: When the Most-Visited Website Gets It Wrong

Google launched AI Overviews — powered by its Gemini model — to U.S. users in May 2024 and immediately generated a cascade of dangerous outputs at scale. Screenshots circulated showing the feature advising users to add nontoxic glue to pizza sauce to help cheese adhere, recommending gasoline as a pasta ingredient, and suggesting adults eat one to three rocks per day. The rocks answer traced back to a satirical piece in The Onion that the model incorporated without filtering for source credibility.

Google characterized the incidents as “isolated examples” from “generally very uncommon queries.” Observers disputed that framing: queries about pizza cheese or presidential demographics are not uncommon searches. Platformer’s reporting documented that Google moved to suppress AI Overviews for specific queries but did not disable the feature.

The failure mode here differs from the Williams case. No single high-stakes transaction was corrupted. Instead, a probabilistic error was broadcast to a population-scale surface with no interstitial review. At sufficient scale, even a low error rate produces a high absolute count of harmful outputs.

Common Root Causes Across Cases

All three of the above incidents share structural features:

Confidence without calibration. Each system produced a high-confidence output in a context where the system had no reliable way to verify its answer. The facial recognition software returned a match score; the chatbot stated policy as fact; the search system summarized a satirical article as medical advice.

Missing human review at the decision point. In none of these cases did a human with domain authority review the AI output before it triggered a consequential action: an arrest, a legal reliance, a public health recommendation.

Deployment ahead of failure-mode mapping. Each vendor deployed into production before documenting how the system would behave at the tail of its distribution. The tail is where liability lives.

The policy implications are tracking in real time at ai-alert.org and through regulatory bodies — the EU AI Act’s high-risk system classifications and the NIST AI Risk Management Framework both point toward mandatory human-in-the-loop requirements for consequential decisions. For current regulatory developments, neuralwatch.org covers EU AI Act implementation and NIST AI RMF adoption.

What Practitioners Should Do

1. Treat model output as a lead, not a decision. Any downstream action with legal, medical, or safety implications requires a human review step that cannot be bypassed.
2. Document the failure-mode surface before deployment. Map what the system does at the tail of its input distribution, including adversarial and out-of-scope inputs.
3. Assert liability explicitly in vendor contracts. Moffatt v. Air Canada confirmed that “the AI did it” is not a defense. Ensure indemnification terms reflect actual risk.
4. Monitor outputs at population scale, not just in unit tests. Surface-level QA misses the long tail. Log deployed outputs and run distributional analysis continuously.
5. Maintain rollback capability. Google’s ability to suppress specific AI Overviews queries was only possible because the feature was not yet fully integrated. Systems embedded deeply into production flows lose rollback options quickly.

---

Where these cases sit in the wider record

Every case above is in the public record because someone wrote it up, which is a much smaller set than the set of failures that occurred. The catalogs that collect them draw their boundaries differently, and picking the wrong one is how researchers conclude an event never happened: our AI incident database comparison sets out what each of AIID, the OECD monitor, AIAAIC and the vulnerability catalogs actually records. The subset we curate ourselves, dated on the day the harm occurred rather than the day it was reported, is filterable by harm domain and source tier in the AI Incident Explorer.

Three of these cases would trigger mandatory notification today rather than a lawsuit years later, because the obligations changed: AI incident reporting requirements covers the EU AI Act, GDPR, SEC and sector clocks and which events start which. For the organizational side, the incident-response playbook for AI systems covers the containment and evidence steps a generic runbook omits when the failing component is a model.

Two adjacent bodies of evidence are worth reading alongside these lawsuits. Vehicle automation has the field’s best public dataset because reporting is mandatory, analyzed in self-driving car accident causes. Synthetic-media harms are the fastest-growing category with the weakest detection story, covered in the deepfake detection tools review.

Related across the network

• LLM Hallucination Detection Methods Explained: Four Techniques for Production Systems — aimoderationtools.com
• Adversarial Attacks on Vision-Language Models: CLIP, LLaVA, GPT-4 — adversarialml.dev
• Adversarial Examples vs. Data Poisoning: Timing Is Everything — adversarialml.dev
• Adversarial Patch Attacks: Physical Perturbations That Fool ML — adversarialml.dev
• Adversarial Robustness in NLP: Why Text Attacks Are Different — adversarialml.dev