The problem is they wrote the ending to the series in season 1 so they could film it with the kids. Then the show's popularity resulted in extra seasons with more character growth that they had to erase for the ending they had already written.

Although I don't think there was ever a chance I wouldn't have been annoyed by the ending. I was annoyed through all of season 2 when Ted and Robin were dating because we were told episode 1 that she was Aunt Robin.