Growing up, there was a four-way stop near my house that one of my friends absolutely hated. It was a pretty busy intersection, and he hated that drivers didn't seem to follow the rules that the person to the right goes first or whatever.

One time when I was driving, he was shocked like "what are you doing!? Its not your turn, you're gonna cause an accident!" when I went. I was like "what are you talking about?" I had driven through that intersection hundreds of times and never really thought about it. When I payed attention to the way the intersection flowed, I figured out the unwritten understanding that I and everyone else approached it with. It was basically just "stop and wait for a car or two to go before proceeding". There was no guaranteed order that I could come up with, it was just that everyone in the area seemed to understand.

Written rules are great if everyone is following the written rules. If you follow the written rules at that intersection you'll be fine, but you're likely to annoy someone for a moment. Nobody is going to be confused if you wait, just impatient.

I agree with you. More important than following rules is to pay attention and adapt as appropriate. If you're the only one following the written rules, there's a chance that you're the one acting unpredictably.