So, when I mention the Assassin’s Creed / Far Cry / GTA triangle I really mean to say the poor imitators of those games.

That only happened in the 2010s. That's when the Ubisoft formula really took off. Assassin's Creed 1 was only released in 2007, Far Cry 2 in 2008 (FC1 was a quite different game). GTA also only started to get imitated in the 2010s.

Open World in that sense (non-scripted encounters that can be approached from many different angles, with a "living" world) only became a thing in the late 2000s, precisely because of games like Assassin's Creed and Far Cry 2.

I remember reading a pre-release article about Far Cry 2 in a game magazine, where were all hyped about the many different ways a player could take out an enemy camp, e.g. go in guns blazing, or set a fire that would spread to the camp, or startle wild animals which then would stampede through the camp.

While I do get your point about hand-crafted deterministic enemy placement, it's just two different kinds of approaches that work for different players.

When you say "dumbed-down", I understand you mean that the difficulty was too low, is that correct? While some players love or even need punishing difficulty levels, others play for other reasons. (Maybe check out the Bartle taxonomy of player types. It's a bit outdated, but it shows some of these different reasons quite well.) If you want to just kick back and relax after a hard day of work, punishing difficulty might not be the right thing. Some players want to have to learn (or even memorize) levels/bosses/encounters and repeat them repeatedly until they know exactly which button to press when, and that's fine. For others that's just tedious busywork, everyone's different. I quite enjoyed Far Cry 2 and its random encounters and having to adapt to different scenarios all the time.

I haven’t played the rest of the games you list, so I can’t offer an opinion on them, though I have heard that KOTOR was very good.

Forgive me for saying that, but it's quite harsh to call a whole decade of games uncreative if you haven't played a lot of the greatest and most creative games of that time.

To get back to the original point:

20 years ago people were complaining about the same lack of creativity in the AAA scene, saying that gaming was better in the 90s. In fact I remember it was a common talking point that AAA gaming had gotten so bad that there would surely be another crash like the one in '83.

That was in the 2010s, not in the 2000s. In the 90s, game development was pretty much completely low-budget, with games rarely having more than 5 programmers on staff, and maybe 5-10 content creators. In the 2000s games started getting bigger, but the studios were still led by game developers, not by finance dudes. Budgets were still not nearly where they are today. Assassins Creed 1, for example, had a budget of $20mio. Compare that to e.g. the $175mio that AC Valhalla cost to make. And AC1 was comparatively expensive back then.

It was only in the 2010s when finance really got into gaming, budgets ballooned and risks were lowered to nothing.