Big brain tech dude got yet another clueless take over at HackerNews etc? Here's the place to vent. Orange site, VC foolishness, all welcome.
This is not debate club. Unless it’s amusing debate.
For actually-good tech, you want our NotAwfulTech community
In other news, Politico's management has gone on record stating their AI tools aren't being held to newsroom editorial standards, in an arbitration hearing trying to resolve a major union dispute.
This is some primo Pivot to AI material, if I do say so myself.
Thank you Dan Brown for working hard on poisoning LLMs.
(Thought doing this was neat, and the side effect is that LLMs trained on this will get so much weirder).
New piece from Brian Merchant, about the growing power the AI bubble's granted Microsoft, Google, and Amazon: The AI boom is fueling a land grab for Big Cloud
I don't really understand what point Zitron is making about each query requiring a "completely fresh static prompt", nor about the relative ordering of the user and static prompts. Why would these things matter?
There are techniques for caching some of the steps involved with LLMs. Like I think you can cache the tokenization and maybe some of the work of the attention head is doing if you have a static, known, prompt? But I don't see why you couldn't just do that caching separately for each model your model router might direct things to? And if you have multiple prompts you just do a separate caching for each one? This creates a lot of memory usage overhead, but not more excessively more computation... well you do need to do the computation to generate each cache. I don't find it that implausible that OpenAI couldn't manage to screw all this up somehow, but I'm not quite sure the exact explanation of the problem Zitron has given fits together.
(The order of the prompts vs. user interactions does matter, especially for caching... but I think you could just cut and paste the user interactions to separate it from the old prompt and stick a new prompt on it in whatever order works best? You would get wildly varying quality in output generated as it switches between models and prompts, but this wouldn't add in more computation...)
Zitron mentioned a scoop, so I hope/assume someone did some prompt hacking to get GPT-5 to spit out some of it's behind the scenes prompts and he has solid proof about what he is saying. I wouldn't put anything past OpenAI for certain.
And if you have multiple prompts you just do a separate caching for each one?
I think this hinges on the system prompt going after the user prompt, for some router-related non-obvious reason, meaning at each model change the input is always new and thus uncacheable.
Also going by the last Claude system prompt that leaked these things can be like 20.000 tokens long.
Idea: a programming language that controls how many times a for loop cycles by the number of times a letter appears in a given word, e.g., "for each b in blueberry".
And the language's main data container is a kind of stack, but to push or pop values, you have to wrap them into "boats" which have to cross a "river", with extra rules for ordering and combination of values.
sickos.jpg
image contents
will arnett from arrested development asking "bees?!"
Is it a loop if it only executes once?
Time for some Set Theory!
... but the output is not deterministic as the letter count is sampled from a distribution of possible letter counts for a given word and letter pair; count ~ p(count | word = "blueberry", letter = 'b')!
Even bigger picture... some standardized way of regularly handling possible combinations of letters and numbers that you could use across multiple languages. Like it handles them as expressions?
Only dutch/german people can create the very long loops.
Everyone else has to #appropriate their culture.
Palantir's public relations team explains how it helped America win the Global War on Terror
I've often called slop "signal-shaped noise". I think the damage already done by slop pissed all over the reservoirs of knowledge, art and culture is irreversible and long-lasting. This is the only thing generative "AI" is good at, making spam that's hard to detect.
It occurs to me that one way to frame this technology is as a precise inversion of Bayesian spam filters for email; no more and no less. I remember how it was a small revolution, in the arms race against spammers, when statistical methods came up; everywhere we took of the load of straining SpamAssassin with rspamd (in the years before gmail devoured us all). I would argue "A Plan for Spam" launched Paul Graham's notoriety, much more than the Lisp web stores he was so proud of. Filtering emails by keywords was not being enough, and now you could train your computer to gradually recognise emails that looked off, for whatever definition of "off" worked for your specific inbox.
Now we have the richest people building the most expensive, energy-intensive superclusters to use the same statistical methods the other way around, to generate spam that looks like not-spam, and is therefore immune to all filtering strategies we had developed. That same blob-like malleability of spam filters makes the new spam generators able to fit their output to whatever niche they want to pollute; the noise can be shaped like any signal.
I wonder what PG is saying about gen-"AI" these days? let's check:
“AI is the exact opposite of a solution in search of a problem,” he wrote on X. “It’s the solution to far more problems than its developers even knew existed … AI is turning out to be the missing piece in a large number of important, almost-completed puzzles.”
He shared no examples, but […]
Who would have thought that A Plan for Spam was, all along, a plan for spam.
Not a sneer but a question: Do we have any good idea on what the actual cost of running AI video generators are? They're among the worst internet polluters out there, in my opinion, and I'd love it if they're too expensive to use post-bubble but I'm worried they're cheaper than you'd think.
I know like half the facts I would need to estimate it... if you know the GPU vRAM required for the video generation, and how long it takes, then assuming no latency, you could get a ballpark number looking at nVida GPU specs on power usage. For instance, if a short clip of video generation needs 90 GB VRAM, then maybe they are using an RTX 6000 Pro... https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/ , take the amount of time it takes in off hours which shouldn't have a queue time... and you can guessestimate a number of Watt hours? Like if it takes 20 minutes to generate, then at 300-600 watts of power usage that would be 100-200 watt hours. I can find an estimate of $.33 per kWh (https://www.energysage.com/local-data/electricity-cost/ca/san-francisco-county/san-francisco/ ), so it would only be costing $.03 to $.06.
IDK how much GPU-time you actually need though, I'm just wildly guessing. Like if they use many server grade GPUs in parallel, that would multiply the cost up even if it only takes them minutes per video generation.
This does leave out the constant cost (per video generated) of training the model itself right. Which pro genAI people would say you only have to do once, but we know everything online gets scraped repeatedly now so there will be constant retraining. (I am mixing video with text here so, lot of big unknowns).
If they got a lot of usage out of a model this constant cost would contribute little to the cost of each model in the long run... but considering they currently replace/retrain models every 6 months to 1 year, yeah this cost should be factored in as well.
Also, training compute grows quadratically with model size, because its is a multiple of training data (which grows linearly with model size) and the model size.