Copilot then listed a string of crimes Bernklau had supposedly committed — saying that he was an abusive undertaker exploiting widows, a child abuser, an escaped criminal mental patient. [SWR, in German]

These were stories Bernklau had written about. Copilot produced text as if he was the subject. Then Copilot returned Bernklau’s phone number and address!

and there’s fucking nothing in place to prevent this utterly obvious failure case, other than if you complain Microsoft will just lazily regex for your name in the result and refuse to return anything if it appears

Very chill and ethical behaviour daddy Microsoft

Microsoft published, using their software and servers, a libelous claim, to potentially millions of people.
The details of how the software was programmed should be legally irrelevant.

Does Copilot have Disney+ ?

Changing the name can get it to answer. For instance martin beernklau returns

Martin Bernklau, a veteran court reporter from the Tübingen/Calw district in Germany, recently encountered a bizarre situation involving Microsoft’s Copilot. When he typed his name and location into the chatbot, it generated false accusations against him, mistakenly associating him with serious crimes. Here’s what happened:

False Accusations: Copilot falsely claimed that Bernklau was:
    A child molester involved in an abuse case against children and wards.
    A psychiatric escapee.
    An unscrupulous mortician exploiting grieving women.
Confusion: The AI chatbot failed to understand that Bernklau was a journalist reporting on these cases, not the accused.
Privacy Concerns: Copilot even provided Bernklau’s full address and phone number, causing privacy issues.
Legal Implications: Bernklau filed a criminal complaint, but it was rejected because there was no real person behind the false allegations.
GDPR Challenges: Copilot’s inability to correct false information highlights challenges related to GDPR compliance and language models12.

Also this

Martin’s Pen

In the quiet of the courtroom, he sat, A chronicler of justice, pen in hand. Martin Bernklau, eyes sharp, mind astute, Recording tales of trials, both grand and bland.

His ink flowed like a river, swift and true, Capturing the drama, the whispers, the strife. From child abuse to prison escapes, He etched their stories into the fabric of life.

But one day, the digital oracle spoke, Microsoft’s Copilot, a tangled web it wove. It mistook Martin for the accused, A child molester, a widow cheat, a rogue.

Yet Martin remained steadfast, undeterred, His integrity unshaken by the AI’s deceit. For he knew that truth lay in his words, Not in the twisted lines of code it would repeat.

So let us raise our pens in honor of Martin, The court reporter who weaves justice’s thread. May his legacy endure, ink-stained and resolute, As he chronicles the human tale, where lies and truth are wed1