Which is doubly worse because those higher-level libraries are black boxes, and you can’t always fix things that arise inside of them easily.

If by "higher level" you mean something like Java libraries, I'd say the opposite is true - at least if you don't have the source for a Java class it is trivial to decompile and have something immediately readable. Can't say the same for something like a dll originally written in C++.