You're right. Just look at this year, the two deserving teams got in Michigan and Washington. The next 3 teams are essentially wild card. The problem is that the playoff system was supposed to make sure those outlier teams like FSU got in when there wasn't enough room when it was just 2 teams. Under the old system, there wouldn't be a controversy at all, but now there's a controversy to rival 2 SEC West teams snubbing everyone else.

The next controversy I was expecting was 2 G5 teams getting in over an unranked P5 champion. But with the Pac-12 almost certainly dissolving, that seems wildly unlikely to ever happen.