Thank you!
After posting this yesterday when lying in bed trying to sleep (i posted directly before going to bed) I thought of a similar reason:
Since (as far as i understood it) a modem is just a device that converts between 2 different types of internet signals, the ISP also needs one on their end to connect me to their data center. So it would be way easier for them to spy on their end of the cable (or signal, or whatever type of modem is used) than on my end, since there isn't really much happening in between.
😖