The problem is essentially how do you define ownership? Is there a right to not make something the copyright holder owns publicly available?

I think in the cases of abandonware or more recently the moves by media companies to delist certain media for tax benefits, there's a good argument to be made over forfeiting the copyright, so it's now public domain and fair game. But I also think for something like the Star Wars Holiday Special, where the creator/copyright holder (not sure about that status post-Disney acquisition) genuinely hates it and does not want it available to the public, the owner should be allowed to restrict access to it.