Watch Jim Browning's videos where he goes in depth in to the lives of some scammers.
Some are educated and happy with a wife and kids. Some managers are make $7,000 USD a month. You can see security footage where they are having fun and doing drugs.
I'm sure people get in to debt and they have to work but I don't think most of the low caste people have enough experience to have conversational skills in English.
From the what I can tell, these people that are working in these scam centers are just human trash that think they deserve to still money from people better off than them.