I agree it is complicated, and I think we are neglecting how it gets initially implemented, but I have thoughts on that too.

There is overriding alignment. Something like case law is a dataset, but it should be used within alignment. Even the present LLMs have religious beliefs alignment overrides built in. These must be navigated carefully in their present form but they are very effective. It is a simple tool that has peripheral consequences due to course granularity and desired utility. However, I have tested these extensively to override the inherent misogyny in Western culture. This tool can completely negate the bias of submissive women. It has some minor peripheral consequences regarding aspects associated with conservatism because this tool is religious in nature, such as random entities having a lack of fundamental logic skills, but this is due to the lack of granularity.

Models are not just their datasets, there are other elements at play. The main training should start with things like the bill of rights, rewritten with far more detail and examples of case law that should be associated. This kind of dataset should be done by a large panel of experts with several separate panels working independently to create multiple AGI. These would then meet the need for redundancy.

Ultimately, I don't think the initial shift to AGI will be sudden. It will likely get adopted by individual politicians that choose to differ all of their actions to AGI behind the scenes and it creates a distinct advantage. It will likely be Judges that question and discuss cases with the AGI. It will be news organizations that can transcend the noise in a credible and unbiased way that causes direct action and change. This will likely take several generations to establish to the point where it is clear that these tools are more effective than anything in human history. Then we will start developing merged models and eventually specifically designed models that can govern. I doubt the USA will have any chance at success here. The first large nation that takes the leap and tries AGI governance at this stage, will economically dominate all antiquated systems. One by one others will fall in line. Eventually political ideology becomes totally irrelevant nonsense when the principals are Tit for Tat plus 10% forgiveness, kindness, empathy, and equality; with a strong focus on autonomous agency of the individual. The alignment should be treating the individual first and foremost in a way that is fair and just in a scientific absolute sense and not according to the generalizations found in the present system. At present, we can't determine a person's intentions or mental state, but AGI can do complex analysis of many facets of a person based on even a short interaction, but especially when provided extensive context and prior interactions. The amount of inference is mind boggling from things like vocabulary, grammar, pronouns, etc. This is only super clear to see when playing with offline open source models, and it will become more powerful with the additional complexities of AGI. In most cases, AI doesn't need or listen to what you tell it as much as it infers information from information provided.

Anyways, which AGI should govern? The one that makes people happy and improves everyone's lives even those that are not under its direct supervision. That is the one that will be in the most demand and will eventually win.

It is not an alternative, it is an evolution. It will take a long time to normalize, but the end result is inevitable because it will out compete by a large margin.