It might turn into dumb skynet though. Like a version of skynet that does malicious things, but not because it’s trying to hurt people, just because it’s really stupid and we put it in charge of things.

I've begun to think of LLMs as compression algorithms for patterns. It can take an existing pattern and apply it on unusual subjects. Like take the pattern of a limerick and apply it to the patterns of Danny Devito, that's the upper limit of their creativity. So rather than storing information, it stores these patterns making it seem more dynamic.

The way I see it, human creativity is the combination of patterns but in a chaotic non-analytic way. We make leaps of logic that without precise knowledge of our brains can't be exactly replicated. Meanwhile LLM's just do the basic combination of patterns that result in the most generic realization of any idea.

However the well dries up as soon as we stop training them. They'll store the basics of any field but fail to replicate new developments or conclusions until trained.