Plus you lose all of the cross-cultural understanding that's currently built into the time. The concept of what the number on the clock is and how that relates to the actual time of day has dozens if not hundreds of tiny bits of additional understanding baked into it depending on the situation.

In order to communicate these ideas, people would start referring to their local offset instead of the UTC and then we're just back at time zones again.