Maybe! The MacOS emoji picker actually does this: You can choose which categories to include or omit, and set favorites... And not all of them are enabled by default. No reason phone keyboards couldn't do the same thing. MacOS calls most of what we'd consider "emojis" to be one category though, lol... So that wouldn't actually solve the problem. But it's possible.

Installing them like an app wouldn't really be a thing though-- Emojis are part of Unicode, which means they're essentially text characters. You wouldn't want to omit those from the system entirely, because if they appear in text, you still want to be able to render them. Kind of like... You might not need (or want) a convenient way to write an "é," but it'd be annoying if somebody wrote "the appetizers were good, but the entrée was just okay" and you saw "entr�e" because you didn't have the right app installed.

Personally, I'd rather have access to everything and just use search to find the one I want, but it might be nice to have the option to omit categories that you aren't interested in.