I did Linux From Scratch recently and they have a brilliant solution. Here's the full text but it's a long read so I'll briefly explain it. https://www.linuxfromscratch.org/hints/downloads/files/more_control_and_pkg_man.txt

Basically you make a new user with the name of the package you want to install. Login to that user then compile and install the package.

Now when you search for files owned by the user with the same name as the package you will find every file that package installed.

You can document that somewhere or just use the find command when you are ready to remove all files related to the package.

I didn't actually do this for my own LFS build so I have no further experience on the matter. I think it will eventually lead to dependency hell when two packages want to install the same file.

I guess flatpaks are better about keeping libraries separate but I'm not sure if they leave random files all over your hard drive the way apt remove/apt purge does. (Getting really annoyed about all the crud left in my home dir)