Ask Lemmy

This was recent, and it's still pretty sore for me. I doubt anyone will be able to pinpoint who I am, but if you for some reason are in this forum and recognize me please DM me. Try to count all of the red flags.

I was hired as a software engineer and was immediately thrown onto a "high-visibility project". My service was the middle man between two other mission critical services. Essentially downstream provided metrics and needed to get to upstream.

I laid out several different architectures that I recommended. First was prometheus. It's literally designed to do this, downstream is spread across many servers, prometheus is literally built to do this. Upstream then can scrape prometheus, any other future dependents can also scrape. This was rejected. "We did prometheus once, it didn't work." I check, it's a single tenant instance of Prometheus running on one 24XXL AWS VM. So, they didn't know how to properly configure prometheus. I tell them I can kill 2 birds with one stone. No, prometheus bad. Rejected.

Second, we use a highly reliable queue setup. Downstream publishes to queue, Upstream reads from queue. Seems simple enough, can have many producers and many subscribers, and we already have a kafka service. Rejected. For why, I ask. Literally "Upstream doesn't know how to work with queues". Literally got that as an answer. Read that as "We need to choose a subpar architecture because we openly admit our engineers don't have the necessary skills". I even offered to help them, to write that part of the code. Rejected.

Third option came straight from the CTO. We love datadog here. Everyone does. Datadog. Oh you feel that pit in your stomach don't you. The mandate came down from on high that Downstream would push metrics to Datadog. I then would need to periodically scrape Datadog, and then have an API that the upstream could then periodically scrape me. I looked into Datadog's API. They don't really support this. I reach out to Datadog, talk to their engineers, and they confirm this is a horrible pattern. I bring this up, say it's just not a good decision, there are better ways. Literally rejected by the CTO himself.

So, I build this rickity ass service, brand new built with thumbtacks and glue. Along the way more is mandated to me. We'll have literally 8x the number of metrics we originally planned for. We're well over Datadog's API limitations. I am mandated to put it into a Postgres instance. Every decision I am overridden.

On top of this, Downstream is completely overworked and doesn't have time to answer questions about specific metrics. Upstream then asks me, who has been there now for a grand total of 4 months, and I don't know the specific questions. I refer them to Downstream for helping describe what specific metrics are and do. They report to my superiors that I am not being a team player for this. They also don't know how to use my API, I have to explain concepts like GET and POST to them, how to serialize datetimes. I end up writing some of their code for them just to make it work.

In the end, we shipped late. There was an arbitrary deadline set by the CTO that we missed - we were not consulted on this deadline, there was no reason for the deadline beyond "We should be live on this date". We missed it by 5 days. During those 5 days I am online every waking moment, sleeping an average of 4-5 hours per night. I'm a walking zombie trying to patch this thing.

A week after release I'm called onto a meeting with my direct boss, who reports to the CTO. He tells me that due to my "Lack of Ownership" and "Lacking team spirit" they are letting me go. I'm stunned. This entire time literally any decision I tried to make was overridden. They chose the worst possible architecture, forced me to implement it, forced me to talk to third parties about designing this anti-pattern. I had 2 other teams actively work against me, and on top of that I had no support from anyone. I was alone, and isolated. I got off that call, and I just cried. I felt like such a failure.

I'm at a new job now, and I've realized what a toxic environment that was. Horrible engineering practices, way too much pressure on me alone. I had developed health issues that I wasn't even aware of that now have subsided. I literally tried my best, and they just let me go. I found out later that my boss who fired me was being chewed out over the horrid project, and he put 100% of the blame on me to save his own ass.

Thanks for listening