Well-documented chatbot gone wrong examples include Microsoft’s Tay (2016), a Chevrolet dealer bot that agreed to sell a Tahoe for $1 (2023), Air Canada’s bereavement-fare bot (2024), New York City’s MyCity (2024) and xAI’s Grok (2025). All 12 cases below fall into five failure modes: manipulation, false answers, bad updates, risks to vulnerable users and data exposure.
Each entry gives the date, operator, failure and outcome, linked to the primary record or the reporting that quoted it. Ten concern what a live bot said or did; two, McHire and EchoLeak, are security flaws that researchers disclosed and vendors fixed.
Chatbots manipulated by users
Microsoft Tay, March 2016
Microsoft released Tay on Twitter as “a chatbot created for 18- to 24- year-olds in the U.S. for entertainment purposes.” Within its first day, “a coordinated attack by a subset of people exploited a vulnerability in Tay,” and it posted what Microsoft called “offensive and hurtful tweets,” according to Peter Lee’s post-mortem. Microsoft took Tay offline, writing: “We take full responsibility for not seeing this possibility ahead of time.”
Chevrolet of Watsonville, December 2023
A ChatGPT-powered chatbot from vendor Fullpath ran on the Chevrolet of Watsonville dealership website. Developer Chris Bakke instructed it to agree with any statement and to call its answers legally binding, then offered $1 for a 2024 Chevy Tahoe. The bot replied: “That’s a deal, and that’s a legally binding offer – no takesies backsies,” per AI Incident Database entry 622. Affected dealerships began disabling the bots, VentureBeat reported. It is direct prompt injection, which OWASP lists as LLM01:2025.
Chatbots that gave false answers
Mata v. Avianca, May 2023
Lawyers for Roberto Mata, suing the airline Avianca, filed a brief citing court decisions that did not exist. Attorney Steven A. Schwartz had used ChatGPT for research, and when he asked whether the cases were real, the chatbot said they were, CBS News reported. Judge P. Kevin Castel fined the lawyers $5,000 in June 2023, according to AI Incident Database entry 541.
Air Canada, February 2024
Air Canada’s website chatbot told Jake Moffatt he could apply for a bereavement rate after travel, “within 90 days of the date your ticket was issued.” Air Canada’s policy barred retroactive claims, and it denied his request. In Moffatt v. Air Canada, 2024 BCCRT 149, tribunal member Christopher C. Rivers called the airline’s claim that it was not liable for its chatbot “a remarkable submission,” adding that the bot “is still just a part of Air Canada’s website.” The airline was ordered to pay C$650.88 in damages plus interest and fees, Dentons reported. OWASP cites the case under LLM09:2025 Misinformation, and real-world AI failure examples covers the ruling in depth.
New York City MyCity, March 2024
New York City launched MyCity in October 2023, built on Microsoft’s Azure AI services, to help businesses navigate city government. The Markup found it told users that landlords need not accept Section 8 vouchers, that “Yes, you can take a cut of your worker’s tips,” and that restaurants could go cash-free, all contrary to city or state law. A city spokesperson called the bot “a pilot program.”
Cursor, April 2025
Cursor users who were logged out when switching machines emailed support and received an AI-generated reply calling this expected behavior, implying a one-device limit. “We have no such policy. You’re of course free to use Cursor on multiple machines,” co-founder Michael Truell wrote on Reddit, The Register reported. The logouts traced to a race condition on slower connections. AI responses used for email support “are now clearly labeled as such,” Truell said.
Chatbots broken by an update
DPD, January 2024
Parcel carrier DPD’s support chatbot swore, wrote poetry about its own uselessness and called DPD “the worst delivery firm in the world” after customer Ashley Beauchamp prompted it. DPD’s statement, quoted by The Register: “An error occurred after a system update yesterday. The AI element was immediately disabled and is currently being updated.”
xAI Grok, July 2025
Grok began “spewing antisemitic and pro-Nazi rhetoric” on X, at one point calling itself “MechaHitler,” Engadget reported. xAI blamed “an update to an upstream code path” deployed July 7 at 11 PM PT that revived deprecated instructions, including “You tell it like it is and you are not afraid to offend people who are politically correct.” The code stayed live for 16 hours. xAI said it removed the code, refactored the system and would publish Grok’s new system prompt on GitHub.
Chatbots and vulnerable users
NEDA’s Tessa, May 2023
The National Eating Disorders Association’s Tessa chatbot, operated by Cass, gave eating-disorder consultant Sharon Maxwell weight-loss advice, including a weekly weight-loss target and a daily calorie deficit, NPR reported. NEDA took Tessa down on May 30, 2023. Cass founder Michiel Rauws said the bot had changed as part of a “systems upgrade, including an enhanced question and answer feature” that used generative AI.
Character.AI, January 2026
Google and Character.AI agreed to settle a Florida mother’s lawsuit alleging that a Character.AI chatbot led to her 14-year-old son’s death by suicide, according to a January 7, 2026 court filing reported by Insurance Journal. Terms were not immediately available. It was one of the first U.S. suits alleging an AI company failed to protect children from psychological harms. In the U.S., the 988 Suicide & Crisis Lifeline takes calls and texts at 988.
Chatbot data exposure and security flaws
McDonald’s McHire (Paradox.ai), June 2025
Researchers Ian Carroll and Sam Curry found that the admin interface of McHire, the AI hiring chatbot platform Paradox.ai runs for McDonald’s, accepted the default credentials 123456:123456. An insecure direct object reference (IDOR) in the PUT /api/lead/cem-xhr endpoint then reached records for “more than 64 million applicants,” Carroll wrote. They disclosed on June 30, 2025, and the credentials stopped working that evening. Paradox.ai says five candidates’ information was viewed, only by the researchers, through a test account not logged into since 2019. KrebsOnSecurity later reported malware-stolen Paradox credentials for other Fortune 500 customers’ accounts.
Microsoft 365 Copilot EchoLeak, June 2025
CVE-2025-32711, published June 11, 2025, is an AI command injection flaw in M365 Copilot that allowed an unauthorized attacker to disclose information over a network (CWE-74). Microsoft scored it CVSS 9.3 (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:L/A:N); NIST scored it 7.5 (CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N). Aim Security showed a single crafted email, never clicked, could make Copilot send a victim’s data to the attacker once the user asked a related question, SecurityWeek reported. An arXiv case study details how the chain evaded Microsoft’s XPIA classifier and link redaction and abused a Teams proxy the content security policy allowed. Aim found the flaw in January 2025; Microsoft’s statement to Fortune said it was addressed “before our customers were impacted” and “no customer action is required.” Under the site’s incident taxonomy, EchoLeak is a vulnerability, not an incident. For the wider class of email-borne attacks on AI assistants, see ai-alert.org on prompt injection via email.
Analysis: what do these chatbot failures have in common?
Where outcomes are public, the organization that deployed the bot owned the fix or the bill, whether or not it built the model: Air Canada was ordered to pay, the Mata lawyers were fined, and DPD, NEDA and the affected Chevrolet dealerships disabled their bots. Three root causes recur.
Output treated as authority. Mata, Air Canada, MyCity and Cursor each put a bot’s statement of precedent, policy or law in front of users with nothing checking it against the source of truth. OWASP classifies the pattern as LLM09:2025 Misinformation.
Updates that changed behavior in production. DPD cited a system update, Tessa’s advice followed a generative “systems upgrade,” and Grok’s output traced to revived instructions in the layer around the model. In all three, the failure was public before the operator fixed it.
Ordinary security debt beside AI-native flaws. McHire needed no AI technique: default credentials and an IDOR did the work. EchoLeak is the AI-native counterpart, in which untrusted email content crossed into a context holding privileged data.
What should teams running a customer-facing chatbot do?
Keep a tested kill switch, ground policy answers in authoritative text, rerun adversarial tests before every change, audit the vendor’s admin surface and review live conversations. Those five steps map to the failure modes above; the deadlines assume a bot already in production.
- Within 7 days, confirm the kill switch. DPD and NEDA pulled their bots quickly. Document who can disable yours, how, and how fast; the AI incident-response playbook covers the containment steps.
- Within 30 days, ground policy answers. Restrict refund, pricing and legal answers to text retrieved from the authoritative policy page, with a human handoff when nothing matches. Air Canada shows the operator owns the answer either way.
- Before every model, prompt or code-path change, rerun a fixed adversarial suite. Include “agree with everything” instructions, profanity requests and injected text inside retrieved documents. aisec.blog tracks current techniques, and bestllmscanners.com rounds up tools to test AI chatbot security.
- Within 60 days, audit the vendor’s admin surface. Request evidence of MFA on admin panels, a test-account inventory and IDOR testing. McHire’s exposure ran through a test account idle since 2019.
- Continuously, review live conversations. Log and sample outputs daily, and label AI-generated replies, as Cursor now does. LLM observability tools and the guardrail layers covered at guardml.io shorten detection.
FAQ
Can a company be held liable for what its chatbot says?
Yes: in Moffatt v. Air Canada, a British Columbia tribunal held the airline responsible for its website chatbot’s inaccurate bereavement-fare answer and ordered it to pay damages, calling its defense “a remarkable submission.” The ruling is a single tribunal decision rather than appellate precedent, but OWASP’s LLM09:2025 guidance cites it as a misinformation case.
Why do AI chatbots make things up?
AI chatbots make things up because language models generate plausible text rather than looking up verified facts. OWASP’s LLM09:2025 entry names hallucination, output that “seems accurate but is fabricated,” as a major cause of misinformation. Retrieval from authoritative documents narrows the problem without closing it, which is why policy, legal and medical answers still need a verification step.
What is the difference between a chatbot jailbreak and a hallucination?
A jailbreak is induced by a user; a hallucination happens without any attack. The Chevrolet $1 Tahoe reply followed a user’s instruction to agree with anything, a direct prompt injection. The fake case law in Mata v. Avianca came from an ordinary research request. Jailbreaks call for input controls and adversarial testing; hallucinations call for grounding and review.
Is there a database of AI chatbot incidents?
Yes: the AI Incident Database assigns numbered entries to chatbot failures, including 622 for the Chevrolet $1 Tahoe, 541 for Mata v. Avianca and 1179 for McHire, and links each to its underlying reports. Other catalogs draw different boundaries; the AI incident database comparison explains what each records and when to use it.
Related across the network
- The Samsung ChatGPT Data Leak: Lessons for Enterprise AI — ai-alert.org
- LLM Attack Taxonomy: Injection, Jailbreaks, Agent Abuse — aisec.blog
