I don't think there's a clear cutoff from a taste perspective, it'll just be staler the longer you wait. As with most things, it's just about juggling the convenience vs quality tradeoff. But the win-win solution, especially in the later days, might be that condensed milk everyone's talking about.

Adding or subtracting a paper filter can be interesting.

Airtightness is worth thinking about to keep the brew tasting fresh, especially for concentrate storage.

Upgrade your ice using filtered water, maybe big cubes vs ice tray.

And of course, try a hot brewed ice coffee with the same beans to compare.