If you want really cheap and low power, a Raspberry Pi will do for most tasks.
If you want more processing power, an old Thinkpad laptop is reliable and has a built-in UPS (a laptop battery).
Any desktop computer will give you the customisation and upgradability you want, but you'll need to decide on each part individually. I don't have any specific recommendations here.