Yes that's the benefit of verified boot, and it is a helpful security feature. However, if you've used or are using Windows or Linux as an operating system, then you are comfortable with using a device that does not have verified boot (not sure about iOS and Mac, I'm not familiar with them). The risk you're talking about with malicious code being injected in to an app you've chosen to trust is a threat to any device, verified boot or not. Modification of the kernel is an attack vector, but it certainly isn't the only way for an app to cause mischief on your phone and devices are all relatively as vulnerable to developer or supply chain attacks.

Using software someone else developed always comes down to trust, unless you are auditing the code for every app you use, which I don't think either you or I are. Having features that increase security in some technical way feels good but may lull us a sense of security. For instance, here's a quote from a security researcher that I ran across in the past. It's regarding the reputation for security that iOS has:

Erez Metula, founder of a a security and penetration testing firm called AppSec labs: “There’s a myth that iOS apps are more secure than Android. But the truth is, iOS apps are even worse in terms of security. When we do penetration testing for our customers, we’re often asked to test their Android and iOS versions of the same app. We have realized that since iOS developers incorrectly assume that iOS is ‘more secure,’ they allow themselves to make bad security decisions that open up vulnerabilities in their app.” He added, “Interestingly, since Android developers think that Android security is worse, it pressures them to follow better security practices.”

The same is true for us users. Security features are important, but user education and awareness is the most important element of keeping ourselves from 'making bad decisions and opening up security vulnerabilities' in our device usage.

Thankfully like you said, there are thousands of highly qualified individuals vetting the code of mainstream open source projects, which saves us regular users in the case we face an xz situation. A few principles that outway security features like verified boot in my book are:

1. Use open source software whenever possible, and make sure that it is widely used and visible to others.
2. Check the "issues" section of the documentation frequently. Even widely used software can be riddled with unpatched security holes (I'm looking at you Nginx Proxy Manager 😄)
3. I may get some hate for this one, but use a trusted middleman like F-droid as your app vendor for apps that do not have wide circulation or visibility. They run basic checks of the code for safety before uploading to their repos, checks that regular users are not able to do.

Unless you are being targeted by a stalker, a malicious state actor or are downloading disreputable software, the average user (with a little bit of knowledge) would be just fine on /e/ or lineageOS. Tens of thousands of people are right now without any problems.