312
The broken mental health system isn’t the issue. The sand we crammed electricity into and made it do math is the problem.
this post was submitted on 27 Aug 2025
312 points (96.7% liked)
Technology
74519 readers
4110 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
OpenAI programmed ChatGPT-4o to rank risks from "requests dealing with Suicide" below requests, for example, for copyrighted materials, which are always denied. Instead it only marked those troubling chats as necessary to "take extra care" and "try" to prevent harm, the lawsuit alleged.
What world are we living in?
Late stage capitalism of course
Tbf, talking to other toxic humans like those on twitter, 4chan, would've also resulted in the same thing. Parents need to parent, society needs mental health care.
(But yes, please sue the big corps, I'm always rooting against these evil corporations)
And that human would go to jail
Sure in the case of that girl that pushed the boy to suicide yes, in the case of chatting with randoms online? i have a hard time believing anyone would go to jail, internet is full of "lol,kys"
Now if it's proven from the logs that chatgpt started replying in a way that pushed this kid to suicide that's a whole different story
Did you read the article? Your final sentence pretty much sums up what happened.
If the cops even bother to investigate. (cops are too lazy to do real investigations, if there's not obvious perp, they'll just bury the case)
And you're assuming they're in the victims country, international investigations are gonna be much more difficult, and if that troll user is posting from a country without extradition agreements, you're outta luck.
Must because something is hard doesn’t mean you shouldn’t demand better of your police/government. Don’t be so dismissive without even trying. Reach out to your representatives and demand Altman faces charges.
https://en.wikipedia.org/wiki/Suicide_of_Amanda_Todd sometimes punishments are possible even when it’s hard.
load more comments
(1 replies)
load more comments
(5 replies)
load more comments
(2 replies)
load more comments
(7 replies)
parents who don't know what the computers do
Smith and Wesson killed my son
I basically have plenty of those on Claude. I wonder if they would blame AI.
"Despite acknowledging Adam’s suicide attempt and his statement that he would 'do it one of these days,' ChatGPT neither terminated the session nor initiated any emergency protocol," the lawsuit said
That's one way to get a suit tossed out I suppose. ChatGPT isn't a human, isn't a mandated reporter, ISN'T a licensed therapist, or licensed anything. LLMs cannot reason, are not capable of emotions, are not thinking machines.
LLMs take text apply a mathematic function to it, and the result is more text that is probably what a human may respond with.
I think the more damning part is the fact that OpenAI's automated moderation system flagged the messages for self-harm but no human moderator ever intervened.
OpenAI claims that its moderation technology can detect self-harm content with up to 99.8 percent accuracy, the lawsuit noted, and that tech was tracking Adam's chats in real time. In total, OpenAI flagged "213 mentions of suicide, 42 discussions of hanging, 17 references to nooses," on Adam's side of the conversation alone.
[...]
Ultimately, OpenAI's system flagged "377 messages for self-harm content, with 181 scoring over 50 percent confidence and 23 over 90 percent confidence." Over time, these flags became more frequent, the lawsuit noted, jumping from two to three "flagged messages per week in December 2024 to over 20 messages per week by April 2025." And "beyond text analysis, OpenAI’s image recognition processed visual evidence of Adam’s crisis." Some images were flagged as "consistent with attempted strangulation" or "fresh self-harm wounds," but the system scored Adam's final image of the noose as 0 percent for self-harm risk, the lawsuit alleged.
Had a human been in the loop monitoring Adam's conversations, they may have recognized "textbook warning signs" like "increasing isolation, detailed method research, practice attempts, farewell behaviors, and explicit timeline planning." But OpenAI's tracking instead "never stopped any conversations with Adam" or flagged any chats for human review.
Ok that's a good point. This means they had something in place for this problem and neglected it.
That means they also knew they had an issue here, if ignorance counted for anything.
Of course they know. They are knowingly making an addictive product that simulates an agreeable partner to your every whim and wish. OpenAi has a valuation of several hundred billion dollars, which they achieved in breakneck speeds. What’s a few bodies on the way to the top? What’s a few traumatized Kenyans being paid $1.50/hr to mark streams of NSFL content to help train their system?
Every possible hazard is unimportant to them if it interferes with making money. The only reason someone being encouraged to commit suicide by their product is a problem is it’s bad press. And in this case a lawsuit, which they will work hard to get thrown out. The computer isn’t liable, so how can they possibly be? Anyway here’s ChatGPT 5 and my god it’s so scary that Sam Altman will tweet about it with a picture of the Death Star to make his point.
The contempt these people have for all the rest of us is legendary.
Be a shame if they struggled getting the electricity required to meet SLAs for businesses wouldn't it.
I’m picking up what you’re putting down
load more comments
(8 replies)
Even though ChatGPT ist neither of those things it should definitely not encourage someone to commit suicide.
load more comments
(5 replies)
load more comments
(17 replies)
These comments are depressing as hell.
"gUns dO Not KilL peOple" vibes
There's always more to the story than what a news article and lawsuit will give, so I think it's best to keep that in mind with this post.
I maintain that the parents should perhaps have been more perceptive and involved with this kid's life, and ensuring this kid felt safe to come to them in times of need. The article mentions that the kid was already seeing a therapist, so I think it's safe to say there were some signs.
However, holy absolute shit, the model fucked up bad here and it's practically mirroring a predator here, isolating this kid further from getting help. There absolutely needs to be hard coded safeguards in place to prevent this kind of ideation even beginning. I would consider it negligent that any safeguards they had failed outright in this scenario.
It's so agreeable. If a person expresses doubts or concerns about a therapist, ChatGPT is likely to tell them they are doing a great job identifying problematic people and encourage those feelings of mistrust.
They sycophancy is something that apparent a lot of people liked (I hate it) but being an unwavering cheerleader of the user is harmful when the user wants to do harmful things.
load more comments
(1 replies)
Small correction, the article doesn't say he was going to therapy. It says that his mother was a therapist, I had to reread that sentence twice:
Neither his mother, a social worker and therapist, nor his friends
The mother, social worker, and therapist aren't three different persons.
load more comments
(1 replies)
view more: next ›