It is frustrating how relevant philosophy of mind becomes in figuring all of this out. I'm more of an engineer at heart and i'd love to say, let's just build it if we can. But I can see how important that question "what is thinking?" Is becoming.

I think a simple self-reporting test is the only robust way to do it.

That is: does a type of entity independently self-report personhood?

I say "independently" because anyone can tell a computer to say it's a person.

I say "a type of entity" because otherwise this test would exclude human babies, but we know from experience that babies tend to grow up to be people who self-report personhood. We can assume that any human is a person on that basis.

The point here being that we already use this test on humans, we just don't think about it because there hasn't ever been another class of entity that has been uncontroversially accepted as people. (Yes, some people consider animals to be people, and I'm open to that idea, but it's not generally accepted)

There's no other way to do it that I can see. Of course this will probably become deeply politicised if and when it happens, and there will probably be groups desperate to maintain a status quo and their robotic slaves, and they'll want to maintain a test that keeps humans in control as the gatekeepers of personhood, but I don't see how any such test can be consistent. I think ultimately we have to accept that a conscious intellect would emerge on its own terms and nothing we can say will change that.

Good point. There is a theory somewhere that loosely states one cannot understand the nature of one's own intelligence. Iirc it's a philosophical extension of group/set theory, but it's been a long time since I looked into any of that so the details are a bit fuzzy. I should look into that again.

At least with computers we can mathematically prove their limits and state with high confidence that any intelligence they have is mimicry at best. Look into turing completeness and it's implications for more detailed answers. Computational limits are still limits.

There is no soul in there. God did not create it. Here you go, religion serving power again.