The reason F-Droid builds from source is to ensure that they can enforce their inclusion criteria. If you go outside F-Droid you lose that guarantee. For example, self-published apks in github or google play may contain anti-features or proprietary code that are forbidden by the F-Droid standards.

From another point of view, what you call a single point of failure is a third party that represents the interests of the user community, independent from individual developers. This is the same model used in GNU/Linux distributions, and Drew DeVault explains here the role that software distributions play in the free software community.

Of course, this represents a trade-off, in that you are placing trust in the software distribution instead of or in addition to the upstream developer. The question is, how can you solve the problem without foregoing F-Droid's inclusion standards? The answer is reproducible builds, where F-Droid builds from source and compares to the developer's apk, and publishes the developer's apk with their signature if the build reproduces successfully.

Until Reproducible builds are the norm in the Android free software world, I accept the trade-off because I value having software freedom in my computing, and I know I can't trust upstream developers to care about that as much as F-Droid or I do.