I've used both Splunk and Datadog in my current job, but I wasn't particularly impressed with either. In both cases, the costs escalated quickly. Now, we're limited to a 15-day retention period, which, in my opinion, significantly diminishes the system's usefulness.

In another company, where I had greater decision-making authority, I took a different approach. I directed all journald logs to a central repository using systemd-journal-remote and provided SSH access to developers who needed to view the logs. This setup was straightforward and efficiently handled a vast volume of logs at no cost. Journald's binary and structured format allowed for advanced searches. Additionally, I configured our primary Python application to log directly to journald, utilizing its structured logging features.