There is a car-free island off of Toronto with pristine roads. Take a look at around the 2-minute mark of this Not Just Bikes video.

This is also why university campuses, which have tons of pedestrian pavement, aren't full of potholes every spring. Same with outdoor malls, amusement parks, and on and on. The freeze thaw doesn't help, but it isn't the main problem.