Years ago I found myself explaining to Chinese Room dinguses - in a neural network, the part that does stuff is not the part written by humans.

I'm not sure it's meaningful to say this sort of AI has source. You can have open data sets. (Or rather you can be open about your data sets. I don't give a shit if LLMs list a bunch of commercial book ISBNs.) But rebuilding a network isn't exactly a matter of hitting "compile" and going out for coffee. It can take months, and the power output of a small city... and it still can't be exact. There's so much randomness involved in the process that it'd be iffy whether you get the same weights twice, even if you built everything around that goal.

Saying "here's the binary, do whatever" is honestly a lot better for neural networks than for code, because it's not like the people who made it know how it works either.