I feel like a big thing that changes how one would percieve Naofumi as a groomer is that he doesn't see any of the children as potential sexual objects. In fact, Raphtalia is quite upset at that point, because she desperately wants him to lewd the tanuki loli and he's not having it.

One peak oldschool example of grooming in anime is definitely GTO. That guy's a creep. The only reason it seems remotely acceptable is he's the protagonist and it's framed like he's a good guy and when he's being a creep it's funny. In addition to the power imbalance of being their teacher, he also uses huge amounts of violence and gang connections to get people to fall in line. The levels of wrong are layered. It's impressive in retrospect.