Even assuming an observer somehow knows the serial number of a spent bill, there is no central database tracking cash transactions. There is no history attached to cash, and no further trail for the observer to follow. While it is unlikely that an observer knows which specific bill you owned, it is downright impossible for the observer to know how you came into possession of said bill, or how it will be spent in the future.

I'd push back on this. Many people get cash from a bank ATM. Businesses often deposit the cash they receive back into a bank. Truthfully I think there are very few hops between the cash withdrawal and deposit, and banks can easily check serial numbers and associate them with you. So an observer like the government/financial authority can probably piece together in most cases how you came into possession of a bill based on ATM withdrawal and where you spent it, based on deposit.

I think this same method is why Bitcoin's privacy is lacking. Satoshi said in the Bitcoin paper that privacy can be maintained by ensuring our addresses remain pseudonymous. In reality, that's just too difficult to do and too much information is leaked because addresses can be tracked and traced and labelled especially when going to and from exchanges when people want to pay for things that do not accept cryptocurrency. So, although I don't think Bitcoin's privacy is better than cash, mostly to the point that serial numbers are not recorded on non-bank cash transfers, I think it's wrong to say cash is "downright impossible" to track.