Huh, I've found The GitHub Copilot better. You still can't trust it when it talks about APIs, though. Or anything else really - you have to keep your wits about you. I use it for suggestions on where to start with things, or for testing my assumptions, or for generating boilerplate code, but not for copying and pasting anything critical.

Here is the problem that won’t change for me or my coworkers : we will never use GitHub and our source code is very private (medical devices or worse).

Also I asked a question that didn’t need any context or codebase. It was about a public API from an open-source project. It hallucinated a lot and failed.

Last but not least, I never needed an autocomplete on steroids. I would enjoy some kind of agent that can give precise answers on specific topics, but I understand that LLMs may never provide this.

I just cringe a lot when programmers tell me to use a tool that obviously can’t and will never be able to give me those answers.

I hate to say it but, I have to agree. GPT4 is a significant improvement over GPT3. I needed to use a Python library for something that was meant to be a small, simple CLI app. It turned into something bigger and accumulated technical debt. Eventually, I was having problems that were niche and hard to trace, even with logging and all the other approaches.

I eventually said fuck it, and so I threw a shit tonne of my code into it, explaining what I was doing, how I was doing it, why I wasn't doing it another way, and what I expected vs the actual result. Sometimes it suggests something that is on the right path or is entirely spot on. Other times, it thinks it knows better than you, to which you tell yourself it isn't, because you tried all its suggestions, and then you realise something that would technically allow GPT to say, "I told you so", but out of spite you just close the tab until the next issue.

For practical tasks, GPT has come pretty far. For technical ones, it is hit or miss, but it can give you some sound advice in place of a solution, sometimes.

I had another issue involving Matplotlib, converting to and from coordinate systems, and having plots that had artifacts due to something not quite right. The atan2 function catches many people out, but I'm experienced enough to know better..... Well, normally. In this particular case, it was a complex situation and I could not reason why the result was distorted. Spending hours with GPT4 lead me in circles. Sometimes it would tell me to do things I just said I did, or that I said don't work. Then, I say to it, "what if we represent this system of parametric equations as a single complex-valued function, instead of dealing with Cartesian to polar conversations?". Then it would zip up a whole lot of math (related to my problem). The damn thing handed me a solution and a half. In theory, it was a great solution. In practice, my code is illiterate, so it doesn't care.

All in all, while it failed to help me solve my issue, it was able to reason and provide feedback to a wide range of challenges. Sometimes it needed prompting to change the trajectory it intends to follow, and this is the part you need to learn as a skill. Until these LLMs are more capable of thinking for themselves. Give it time.