To argue by analogy, it’s not like getting an artificial feather exactly right was ever a bottleneck to developing air travel once we got the basics of aerodynamics down.

I suspect that "artificial intelligence" may be a bit more like making an artificial bird that self replicates, with computers and AI as it exists now being somewhere in-between thrown rocks and gliders.

We only ever "beat" biology by cheating via removing a core requirement of self replication. An airplane factory that has to scavenge for all the rare elements involved in making a turbine, would never fly. We had never actually beaten biology. Supersonic aircraft may be closer to a rock thrown off the cliff than to surpassing biology.

That "cheat code" shouldn't be expected to apply to skynet or ASI or whatever, because skynet is presumably capable of self replication. Would be pretty odd if "ASI" would be the first thing that we actually beat biology on.

[–] YourNetworkIsHaunted@awful.systems 4 points 9 hours ago (1 children)

I think that's still putting the cart before the horse a bit. We don't understand how the brain creates consciousness or have a meaningful definition of "general intelligence" other than "y'know; like a people does". Assuming that simulating a human brain is the best way to get to this poorly-defined goals overestimates our understanding of the underlying problem just as much as assuming that the confabulatron will determine get there soon.

[–] diz@awful.systems 2 points 9 hours ago* (last edited 8 hours ago)

I think the question of "general intelligence" is kind of a red herring. Evolution for example creates extremely complex organisms and behaviors, all without any "general intelligence" working towards some overarching goal.

The other issue with Yudkowsky is that he's an unimaginative fool whose only source of insights on the topic is science fiction, which he doesn't even understand. There is no fun in having Skynet start a nuclear war and then itself perish in the aftermath, as the power plants it depend on cease working.

Humanity itself doesn't possess that kind of intelligence envisioned for "AGI". When it comes to science and technology, we are all powerful hivemind. When it comes to deciding what to do with said science and technology, we are no more intelligent than an amoeba, crawling along a gradient.