And you're absolutely right about that. That's not the same thing as LLMs being incapable of constituting anything written in a novel way, but that they will readily with very little prodding regurgitate complete works verbatim is definitely a problem. That's not a remix. That's publishing the same track and slapping your name on it. Doing it two bars at a time doesn't make it better.

It's so easy to get ChatGPT, for example, to regurgitate its training data that you could do it by accident (at least until someone published it last year). But, the critics cry, you're using ChatGPT in an unintended way. And indeed, exploiting ChatGPT to reveal its training data is a lot like lobotomizing a patient or torture victim to get them to reveal where they learned something, but that really betrays that these models don't actually think at all. They don't actually contribute anything of their own; they simply have such a large volume of data to reorganize that it's (by design) impossible to divine which source is being plagiarised at any given token.

Add to that the fact that every regulatory body confronted with the question of LLM creativity has so far decided that humans, and only humans, are capable of creativity, at least so far as our ordered societies will recognize. By legal definition, ChatGPT cannot transform (term of art) a work. Only a human can do that.

It doesn't really matter how an LLM does what it does. You don't need to open the black box to know that it's a plagiarism machine, because plagiarism doesn't depend on methods (or sophisticated mental gymnastics); it depends on content. It doesn't matter whether you intended the work to be transformative: if you repeated the work verbatim, you plagiarized it. It's already been demonstrated that an LLM, by definition, will repeat its training data a non-zero portion of the time. In small chunks that's indistinguishable, arguably, from the way a real mind might handle language, but in large chunks it's always plagiarism, because an LLM does not think and cannot "remix". A DJ can make a mashup; an AI, at least as of today, cannot. The question isn't whether the LLM spits out training data; the question is the extent to which we're willing to accept some amount of plagiarism in exchange for the utility of the tool.