Not sure what you mean by OS-level

By OS-level, I mean Linux terminated the process because the process tried to do an instruction it wasn't allowed to do, specifically, trying to access a memory location it wasn't allowed to. That leads to a segmentation fault, on Linux.

It’s pretty much a DoS at this point.

Calling it a DoS is missing a lot of nuance because you're blaming Google's script, not Firefox. Having 20000 variables in a single frame is something a javascript program is allowed to do; it's a well-formed program that doesn't violate any rules of Javascript, so the fault is not in google's script, but in Firefox's JS interpreter. That doesn't mean that Google's script is good quality, but it's still valid.

Finally, what makes this particularly bad? Any "undefined behavior" can be exploited as security holes [ § "Undefined Behavior and Security Don't Mix Well"].

Now, it is possible to get a segmentation fault without having gone through a undefined behavior: by allocating memory, and then asking linux to make that memory inaccessible.

It's also possible to have undefined behavior and not get a segmentation fault: by chance you go past the end of a memory allocation into another valid allocation. This would be very bad because then your program definitely has a security hole.

So, if we decide that "google's JS is so bad that we shouldn't run it", the script should stop running because Firefox's JS interpreter stops running it. Firefox should pop up a window saying "the Javascript on this page was cancelled because Google sucks". I think this is incorrect, because the script is valid, but it's not bad because at least there's no exploitable security hole.

Or, you could have Firefox force itself to exit without triggered undefined behavior. That would actually would be a DoS, but at least you can't steal encryption keys or whatever off the system.