this post was submitted on 10 Sep 2025
17 points (94.7% liked)

SneerClub

1188 readers
78 users here now

Hurling ordure at the TREACLES, especially those closely related to LessWrong.

AI-Industrial-Complex grift is fine as long as it sufficiently relates to the AI doom from the TREACLES. (Though TechTakes may be more suitable.)

This is sneer club, not debate club. Unless it's amusing debate.

[Especially don't debate the race scientists, if any sneak in - we ban and delete them as unsuitable for the server.]

See our twin at Reddit

founded 2 years ago
MODERATORS
self@awful.systems
blakestacey@awful.systems
dgerard@awful.systems
17
New Scientist reviews the Yudkowsky/Soares book. They don't recommend it. (www.newscientist.com)
submitted 2 days ago by dgerard@awful.systems to c/sneerclub@awful.systems
33 comments fedilink hide all child comments
top 33 comments
sorted by: hot top controversial new old
[–] ShakingMyHead@awful.systems 8 points 1 day ago* (last edited 1 day ago)

LLMs and humans are both sentence-producing machines, but they were shaped by different processes to do different work

Except not really. We're not sentence-producing machines, we're "machines" (so to speak) that can produce sentences. Not the same thing.

Once this is in place, they say, nations must be prepared to enforce these restrictions by bombing unregistered data centres, even if this risks nuclear war, “because datacenters can kill more people than nuclear weapons” (emphasis theirs).

So the plan is still to kill everyone to death to prevent GPT-~~5~~ ~~6~~ ~~7~~ ~~8~~ ...

[–] stingpie@lemmy.world 7 points 1 day ago (4 children)

AI safety "researchers" can be so dense sometimes. It's like they are always at the verge of understanding, but make a left turn right before they get there. ASI would not make random decisions. It would make logical decisions. Any maximizer would try to maximize it's chances of success, not satisfy them.

So if we imagine an ASI which had the goal of turning the universe into paperclips, which one of the following options would maximize it's chances of success?

  1. immediately kill all humans and turn them into paperclips.
  2. establish a positive relationship with humanity in case the ASI is destroyed and needs to be rebuilt. (The humans will happily rebuild it)

It boggles the mind that people don't recognize this. If an ASI's goals do not include the destruction of humanity as an early instrumental goal, it will not randomly decide to destroy humanity, and it will instead cater to humanity to maximize the chances humanity will rebuild it.

In addition, all the hype over ASI safety (ASI will not occur in this century, see **1) drowns out existing AI safety issues. For example, consider "The Algorithm" which determines how social media decides what to show to people. It is driven to maximize engagement, in any way possible, without supervision. What's the optimal way to maintain engagement? I can't say for sure, but brief and inconsistent spikes of dopamine is the most reliable way of conditioning pavlovian responses in animals, and it seems like the algorithm follows this rule to a tee. I don't know for a fact whether social media is optimized to be addictive (let's be honest though, it clearly is) but simply the fact that it could be is obviously less important than a theoretical AI which could be bad in a hundred years or so. Otherwise, who would fund these poor AI start ups whose intention is to build the nuke safely but also super rushed?

Another classic example of AI safety suddenly becoming unimportant when we know it's dangerous is GPT-pyschosis. Who could've predicted a decade ago that advanced AI chatbots who are specifically trained to maximize the happiness of a user would become sycophants who reflect delusions as some profound truth of the universe? Certainly not the entirety of philosophers opposed to utilitarianism, who predicted that reducing joy to a number leads to a dangerous morality in which any bad behavior is tolerated as long as there is a light at the end of the tunnel. No! You think OpenAI, primarily funded my Microsoft, famous for their manipulative practices in the 90's and 00's, would create a manipulative AI to maximize their profits as a non-profit??

I don't want to sound embittered or disillusioned or something, but I genuinely cannot understand how the 'greatest minds' completely glaze over the most basic and obvious facts.

**1: the human brain contains 100 trillion synapses and 80 billion neurons. Accurate simulation of a single neuron requires a continuous simulation involving 4 or 5 variables and 1 to 2 constants(per synapse). You would need 800 terabytes of ram to simply store a human brain. In order to simulate a human brain for a single step, you would need a minimum of 800 trillion floating point operations. If we simulate the brain in realtime with a time step of one millisecond, you would need 800 petaflops. The world's most powerful computer is Hewlett Packard's "el capitan" which has 1.7 exaflops, and 5 petabytes of ram. The limiting factor for brain simulation would be the amount of data transferable between CPU and GPU chiplets, which for el capitan is 5 terabytes per second, but we need 40 petabytes per second(800 petabytes, divided by 128 gigabytes available to each chiplet, then squared) since we want each neuron to be capable of being connected to any other arbitrary neuron.

This is only the amount of computing power we need to simulate a single person. To be super intelligent, we would probably need something a thousand times more powerful.

[–] fullsquare@awful.systems 3 points 20 hours ago* (last edited 18 hours ago) (1 children)

i don't think it would be so simple and i don't think you can abstract neurons so hard, there are extrasynaptic receptors that react to concentrations of neurotransmitters outside synapses, and there are some neurotransmitters that leak out of synapses. thousands of leaking synapses can contribute to activation of some random receptor, or more than one this way. some other receptors are extrasynaptic by default and don't really have synapses, neuropeptides work like this but not only these. for gasotransmitters, effectively there's no concept of synapse. i don't think you can abstract all neurotransmitters to some one chemical messenger either, there are different ones with different half-lives, different diffusion rates, different metabolites some of which work in completely different ways. (steroids, neuropeptides, gasotransmitters, whatever lipids go into cannabinoid system, it's not just monoamines/glutamate/GABA/acetylcholine).

some receptors take multiple inputs, there are NMDA receptors that really only fire when glutamate and glycine both bind to it, and only after AMPA receptor nearby opens up first. we already know these things are important in forming of memories so it's probably a big deal. some of these receptors are ion channels, and some of these are important especially intracellular calcium

[–] porous_grey_matter@lemmy.ml 2 points 9 hours ago (1 children)

That makes their napkin calculation extra generous, and us even less likely to be near being able to stimulate such a thing.

[–] fullsquare@awful.systems 1 points 9 hours ago (1 children)

at minimum this requires additionally keeping position of neurons, modelling concentration of any neurotransmitters and their diffusion (taking into account shape of cells around) and their degradation products, some of which are active on their own. whatever set of interactions might be between neurons, it'll probably make it changing with time and probably not sparse (information exchange isn't just packaged neatly within synapses)

[–] porous_grey_matter@lemmy.ml 1 points 8 hours ago* (last edited 8 hours ago) (1 children)

Yeah, maybe, all those things could be necessary for sure. It's possible that our brains aren't the exact most optimal way of structuring such a thing, and it's not guaranteed that the best way to replicate it is to stimulate it. It's also plausible that there are calculations which capture a good deal of the complexity of the relative positions of neurons in simpler terms. Maybe there are way more complications than that. Maybe some of them work against each other in our brains and it would be better to leave them out of a simulation. There are many orders of magnitudes of unknowns. But it seems really likely that it's at least as complicated as what the earlier poster described. And I think that's quite a strong position already for most practical arguments about it.

[–] fullsquare@awful.systems 1 points 7 hours ago* (last edited 7 hours ago)

it's a guess of what can be abstracted away and what has to remain. i'd just add that evolutionarily, peptide signalling is older than synapses so these, or something that works like these, probably can't be just left out of the picture, and there's a couple of processes that seem important that depend on them (you can live normal life while packed full of naloxone, which blocks activity of opioid peptides, but this probably won't work with, say, orexin which is important for sleep/wake cycle)

[–] Architeuthis@awful.systems 8 points 1 day ago (1 children)

All the stuff about ASI is basically theology, or trying to do armchair psychology to Yog-Sothoth. If autonomous ASI ever happens it's kind of definitionally impossible to know what it'll do, it's beyond us.

The simulating synapses is hard stuff I can take or leave. To argue by analogy, it's not like getting an artificial feather exactly right was ever a bottleneck to developing air travel once we got the basics of aerodynamics down.

[–] diz@awful.systems 3 points 1 day ago* (last edited 1 day ago) (1 children)

To argue by analogy, it’s not like getting an artificial feather exactly right was ever a bottleneck to developing air travel once we got the basics of aerodynamics down.

I suspect that "artificial intelligence" may be a bit more like making an artificial bird that self replicates, with computers and AI as it exists now being somewhere in-between thrown rocks and gliders.

We only ever "beat" biology by cheating via removing a core requirement of self replication. An airplane factory that has to scavenge for all the rare elements involved in making a turbine, would never fly. We had never actually beaten biology. Supersonic aircraft may be closer to a rock thrown off the cliff than to surpassing biology.

That "cheat code" shouldn't be expected to apply to skynet or ASI or whatever, because skynet is presumably capable of self replication. Would be pretty odd if "ASI" would be the first thing that we actually beat biology on.

[–] YourNetworkIsHaunted@awful.systems 4 points 9 hours ago (1 children)

I think that's still putting the cart before the horse a bit. We don't understand how the brain creates consciousness or have a meaningful definition of "general intelligence" other than "y'know; like a people does". Assuming that simulating a human brain is the best way to get to this poorly-defined goals overestimates our understanding of the underlying problem just as much as assuming that the confabulatron will determine get there soon.

[–] diz@awful.systems 2 points 8 hours ago* (last edited 8 hours ago)

I think the question of "general intelligence" is kind of a red herring. Evolution for example creates extremely complex organisms and behaviors, all without any "general intelligence" working towards some overarching goal.

The other issue with Yudkowsky is that he's an unimaginative fool whose only source of insights on the topic is science fiction, which he doesn't even understand. There is no fun in having Skynet start a nuclear war and then itself perish in the aftermath, as the power plants it depend on cease working.

Humanity itself doesn't possess that kind of intelligence envisioned for "AGI". When it comes to science and technology, we are all powerful hivemind. When it comes to deciding what to do with said science and technology, we are no more intelligent than an amoeba, crawling along a gradient.

[–] gerikson@awful.systems 3 points 1 day ago (1 children)

deep down they realize that as soon as the machines become superintelligent they'll realize how fucked up humans are and decide it's a net postive to delete us

[–] prw@mastodon.sdf.org 5 points 1 day ago

@gerikson @stingpie Easily the most realistic part of Age of Ultron.

[–] diz@awful.systems 2 points 1 day ago* (last edited 1 day ago)

The thing about synapses etc argument is that the hype crowd argues that perhaps the AI could wind up doing something much more effective than what-ever-it-is-that-real-brains-do.

If you look at capabilities, however, it is inarguable that "artificial neurons" seem intrinsically a lot less effective than real ones, if we consider small animals (like e.g. a jumping spider or a bee, or even a roundworm).

It is a rather unusual situation. When it comes to things like e.g. converting chemical energy to mechanical energy, we did not have to fully understand and copy muscles to be able to build a steam engine that has higher mechanical power output than you could get out of an elephant. That was the case for arithmetic, too, and hence there was this expectation of imminent AI in the 1960s.

I think it boils down to intelligence being a very specific thing evolved for a specific purpose, less like "moving underwater from point A to point B" (which submarine does pretty well) and more like "fish doing what fish do". The submarine represents very little progress towards fishiness.

[–] Architeuthis@awful.systems 12 points 1 day ago (1 children)

The pair also suggest that signs of AI plateauing, as seems to be the case with OpenAI’s latest GPT-5 model, could actually be the result of a clandestine superintelligent AI sabotaging its competitors.

copium-intubation.tiff

Also this seems like the natural progression of that time Yud embarrassed himself by cautioning actual ML researchers to be weary of 'sudden drops in loss function during training', which was just an insanely uninformed thing to say out loud.

[–] Soyweiser@awful.systems 10 points 1 day ago* (last edited 1 day ago) (1 children)

From the second link

first, I've seen that one of the most common responses is that anyone criticising the original post clearly doesn't understand it and is ignorant of how language models work

And

PS: please don't respond to this thread with "OK the exact words don't make sense, but if we wave our hands we can imagine he really meant some different set of words that if we squint kinda do make sense".

I don't know why some folks respond like this every single *time

Lol.

And of course Yud doubles down in the replies and goes on about a "security mindset". You can see why he was wowed by ceos, he just loves the buzzwords. ('what if the singularity happens' is not a realistic part of any security mindset, it gets even sillier here, as the recursive selfimprovement here just instantly leads to a undetectable AGI without any intervening steps)

It gets even better, in defending himself he points out that using the wrong words is fine and some people who do research on it actually say loss function at times, and as an example he uses a tweet that is seemingly mocking him (while also being serious about job offers as nothing is not on several levels of ironic online) https://xcancel.com/aidangomez/status/1651207435275870209#m

Remember, when your code doesn't compile, it might mean you made a mistake in coding, or your code is about to become selfaware.

[–] Architeuthis@awful.systems 10 points 1 day ago (1 children)

Remember, when your code doesn’t compile, it might mean you made a mistake in coding, or your code is about to become selfaware.

Good analogy actually.

Don't forget Yud is also a big compiler understander

[–] Soyweiser@awful.systems 8 points 1 day ago

And a good writer. Verbosity being the soul of wit.

[–] dgerard@awful.systems 12 points 2 days ago (3 children)

i actually got hold of a review copy of this

(using the underhand scurvy weasel trick of asking for one)

that was two weeks ago and i still haven't opened it lol

better get to that, sigh

this review has a number of issues (he liked HPMOR) but the key points are clear: bad argument, bad book, don't bother

[–] bitofhope@awful.systems 4 points 1 day ago (1 children)

It's even worse than Methods? That should be surprising but isn't.

[–] dgerard@awful.systems 5 points 1 day ago (1 children)

it's shorter!

[–] YourNetworkIsHaunted@awful.systems 3 points 10 hours ago (1 children)

Methods is increasingly looking like Ulysses for the Internet age. You may or may not get anything worthwhile from trying to read it, but the scope and scale are such that it's impressive it exists and also you will almost certainly not finish it, leaving the parts justifying it's considerable reputation somewhere in the 40,000 we words you didn't read.

[–] blakestacey@awful.systems 3 points 6 hours ago

At least with Ulysses, there's a brothel chapter.

[–] Architeuthis@awful.systems 9 points 1 day ago* (last edited 1 day ago) (2 children)

They also seem to broadly agree with the 'hey, humans are pretty shit at thinking too, you know' line of LLM apologetics.

“LLMs and humans are both sentence-producing machines, but they were shaped by different processes to do different work,” say the pair – again, I’m in full agreement.

But judging from the rest of the review I can see how you kind of have to be at least somewhat rationalist-adjacent to have a chance of actually reading the thing to the end.

[–] ShakingMyHead@awful.systems 8 points 1 day ago

Kind of like saying that humans are car-driving machines because we drive cars.

[–] YourNetworkIsHaunted@awful.systems 8 points 1 day ago

Born to create meaning

Forced to produce sentences

[–] blakestacey@awful.systems 14 points 2 days ago* (last edited 2 days ago) (2 children)

this review has a number of issues

For example, it doesn't even get through the subhead before calling Yud an "AI researcher".

All three of these movements [Bay Area rationalists, "AI safety" and Effective Altruists] attempt to derive their way of viewing the world from first principles, applying logic and evidence to determine the best ways of being.

Sure, Jan.

[–] Soyweiser@awful.systems 5 points 1 day ago

“AI researcher blogger”

[–] swlabr@awful.systems 8 points 2 days ago

logic and evidence

Please, it's "facts and logic". Has this author never been on the internet?