I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history. Agents cannot program, and it’s taking longer and longer to realize that they can’t. They are a highly sophisticated statistical model designed to mimic the distribution of programming. The output is broken, but in a way that’s getting harder and harder to detect. Which is exactly what you’d expect from an increasingly accurate statistical model.
This alarm’s being rung for over a year now, so “calling it now” means finally reading the writing on the wall
Let it be known that the first person to call it was actually Sam Altman when OpenAI’s paper on AI Scaling Laws in 2020 subtly showed that the diminishing returns will stop showing improvement with infinite power, compute time, and data before 94% accuracy is reached.
yeah I was kinda like. calling it?
The difference is it’s geohot saying it this time. His opinion will likely be taken more seriously by silicone valley elites. For reference, he was the original PlayStation hacker and lead at Comma.ai (FOSS self-driving).
This is very obvious unless you are in tech leadership, in which case your job is now to push this at all costs and suppress dissenting voices.
In tech leadership. I don’t have to push it. My talented engineers took to it immediately.
They learned quickly that it is a tool. Instead of using a shovel and a wheelbarrow, they have a backhoe now. If you don’t know how to dig a hole, the backhoe is just a way to make a mess faster. It doesn’t replace intelligence.
They can use it to do the scutwork while they focus on the important stuff.
The duds are still typing shit into spreadsheets and emailing them as attachments while their coworkers are getting stuff done.
It is a tool. You can learn to use it or you can just be mad that it exists. In either case it isn’t going away. Like the telephone, the car, the computer, the internet, it is here to stay.
What’s fascinating about this conversation is … how do people think software used to be made ? With talented and knowledgeable developers who would never “hallucinate” an API or a library function ? With cybersec experts who would never put their user’s data in jeopardy ? With performance investigators checking the computational complexity of each function ? Bitch please…
Software engineering is not the kind of mystical cathedral building these people have in mind, it’s more like a musty workshop in Pakistan where they make tractor tires with no safety equipment and a cigarette in their mouth. We’ve been throwing imperfect humans in various states of lucidity at every problem known to man for 30 years but suddenly people start believing that their bog standard CRUD software should be written by monks having attained cosmic godhood.
If you’re letting your engineers find uses for it instead of constantly demanding that they generate lengthy “user stories” and decision documents and deferring thinking to agents instead of quickly planning stuff out using their experience then you’re probably quite an outlier by now.
I am a very lazy man, micromanagement takes so much fucking energy. Hire talented people, give clear and unambiguous guidance and trust them to do the work. It is amazing how easy management can be when you don’t get in the way.
Being in leadership is way easier than being an engineer and the pay is better too. Some people really overcomplicate shit.
Typing shit into spreadsheets isn’t something software engineers typically do… And usually shit that’s typed into spreadsheets has to be accurate. There are major consequences if not, and AI is not reliable enough for that kind of work.
geohot
Now there’s a name I haven’t heard in a long time. George Hotz was the guy who first jailbroke iOS and the PlayStation 3 and made the towelroot exploit for early versions of Android, before legal threats drove him out of the scene.
And the self driving car company? Didn’t he also recently found another company for servers or something? I remember seeing him get like 1st place (or top 10) for one of the days on Advent of Code like a year or two ago. Thats very difficult, like Olympic swimmer of programmer type of thing.
They are not the automated from 0 to 100 coders that some people claim them to be. But they are quite capable, definitely much more capable than what anyone could have imagined ten years ago. Given well defined problems they can excel at even relatively complex tasks. I pointed Claude at a latex file of a somewhat complicated nonparametric statistical estimate calculation to look for any mistakes and it was actually able to find some. I then pointed it at a code that replicates the calculations and it was also able to correctly identify some issues with the code. I think this is the way one should use LLMs, not let it loose on coding tasks. In the former way you won’t even be able to burn through your first tier account quota where as in the latter the LLM will likely end up getting in weird loops burning tokens like there is no tomorrow. Also this method of sane usage of LLMs is much more suitable for open local LLMs. I don’t think there is any doubt anymore that LLMs can be very useful tools, not just for doing stuff but learning it too. People should move past the stage of invalid criticisms like “they are just stochastic parrots” and move to more serious matters like environmental impact, greedy fucking CEOs pretending LLMs are replacements for humans, degredation of skills, getting lazy at checking AI code, ethics of capitalizing on collective human knowledge and the unsustainable AI bubble that tech companies are pushing for.
invalid criticisms like “they are just stochastic parrots”
That’s not a criticism per se, it’s a description of how they work.
if it’s broken in a way that can’t be detected, is it actually broken?
all software is broken in some way. if the rate of bugs generated by llm and the severity of those bugs drops below the rate you would expect from a human programming team, then llm is offering something competitive.
It will eventually be detected, but it passes tests before hitting production, that is the problem.
broken in a way that can’t be detected
Is not what anyone said and you’re lying when you pretend they did.
deleted by creator
No, humans make less mistakes. Less. That’s the key here, statistical models are trained on human data so by pure logic can never, ever, under any circuimstance, reach 100% accuracy. With current understanding of LLMs with a focus on AI Scaling Laws, and more importantly of natural human language adaptation, they will never reach 94% accuracy with infinite power and infinite training. That’s what the curve shows us in OpenAI’s 2020 research paper on AI Scaling Laws and later Deepmind’s paper correcting their math, that the diminishing returns will hit a limit far before convergence.
In addition to that, the AI also cannot detect subtle changes to established problems or any new unaccounted for variables, because they’re a statistical model and not capable of actual thought. They also lack any sense of responsibility for their actions for the same reason.
You fucking sloppers always try to say “HuMAnS mAkE misTAKeS, TOO!” Yeah and the fucking slopbots are trained on those mistakes and make them again but worse.
But you’re forgetting the key difference that makes it so much worse - we can fix human mistakes especially if we can talk to the human to figure out how. With an llm we have no external reference, only poorly designed code where the comments are there to guide the writing, not describe what was written. So it’s much harder to debug an output, and the llm cannot be trusted to clean it up either.
A human can be held responsible. A machine cannot. If the machine writes bad code, and someone gets injured or killed because of it, who takes responsibility?
I state again: a machine cannot be held responsible.
It is never the coder that is responsible, it is the one who makes the code available to use. Often with humans, they are one and the same. With machines, they are not.
You can totally fix AI-written code with AI. You tell it something is wrong, it tries to fix it.
I did a recent experiment with AI writing a document format converter and that’s exactly what I did. It wrote some code, I checked the output, found a formatting issue or similar, asked it to fix it, repeat. It works unreasonably well and with Fable the final code isn’t even bad.
You can fix problems, if you know they are there and there is a model of that problem being fixed.
You can’t fix problems you don’t know are there, or do not have modeling.
That’s true for humans too surely? How would anyone fix problems they don’t know exist?
Humans generally don’t hallucinate libraries or documentation. If there is a bug or error on a human maintaine repo the human in charge will generally know what went wrong and how to fix it, the AI will just gaslight your ass because the AI has no idea.
This is true. But it doesn’t invalidate any of my points. Humans have unique failure modes too.
To add to the other response - it is much more difficult to work with Ai to debug inconsistent issues or similar unless you can understand the code and step through with a debugger to check for race conditions or similar.
Recently I was working with an Ai tool for some c code that depending on machine ran wildly differently. The Ai was unable to identify any issues, and kept recommending fixes for hardcoding values or similar that I had to revert. The fix ended up needing to use valgrind to create a different enough environment to see how a race condition was made to properly have one async call delay for the other.
AI can be powerful, and humans can be dumb. But if the code was human made, I would not have needed 3 hours to find a problem, and I wouldn’t have tried to turn to AI for a simple fix because I’d know what I was looking for to start with.
Now keep doing this for months, with non-trivial software that other people use.
Probably fine if you review the code carefully. And if you’re working in a domain that AI is decent at (e.g. web stuff). But even if it wasn’t it doesn’t mean AI cannot program.
Also, unlike say, writing articles, like 75% of coding is copy.pasting blindly from elsewhere anyway.
Not in my experience.
You know the feeling that you want to rewrite a project? But you know that most rewrites are a bad idea.
Be it your own, old code. Or code you inherited.
There is a small chance that the world realizes that they went in the wrong direction and nothing can get fixed. That will be the time of rewrites.
No, I don’t expect this to be very likely. The agent code will remain, and human programmers get yelled at for not fixing it fast enough.
Rewriting all code after everyone has been using AI tools to break it doesn’t sound any better than writing good code now, be it with or without LLMs.
No one paying the bills cares
It’s so nice to see some people speaking reason. If only any of those people ran multibillion dollar companies.
Let them fail and then scramble to rehire.
It’s going to be a wonderful time to be a Freelance Senior Developer and above in a few years.
remember, when you interview for a job and they ask you, “do you have any questions?”, you ask;
- has AI ever been used to develop your product?
- what percentage of your product has been written by agenetic AI?
- is the use of AI tracked as a performance indicator?
But they do work, maybe not as a full replacement but my god the amount of boilerplate I can avoid in creating unit tests from scratch. Extracting and finding information in the code base is also useful, not everything is an easy text search of tracing a few code paths. It’s an incredible tool for these kinds of work.
If it becomes harder to tell the difference then it also means it’s closer to matching reality. And todays AI can do very impressive “reasoning”, managing to debug complex issues I have had.
The most important part is that you as developer is fully responsible and can stand behind what they do and deliver using AI agents.
Right? The bottle has opened. It has taken so much mundane work out of programming. Also, I feel like a human is just as likely to create great looking code changes with a possible flaw. You just have to review the code. Whether it’s a person or a bot, “lgtm” can only be used sparingly.
Unpopular prediction: AI agents are going to get better at coding. Not great, but halfway decent at cranking out basic features. Once everything levels out in like 3-5 years, AI agents will be a cherished part of the toolbox most software developers. It will be useful for skimming code, it will be useful for tedious parts of tasks that are just a degree off from boilerplate.
People are definitely gonna try to use it for things more complicated than that, and that’ll be a mistake, and it will be costly, but the far side of it could be pretty cool actually. Admittedly I have an optimistic disposition.
I have yet to be impressed.I’m not very convinced*. I asked for format type mappings between Pipewire, WebGpu, and Vulcan and both ChatGPT and Gemini failed very badly only providing the most common type mappings. This should be a wildly easy task, something any programmer or even beginner programmers can complete. It’s just very boring, mundane, buffer shifting like work. It almost feels like pencil pushing.Why couldn’t they do it?
Are you using real agents or the free chats on the web, because the latter ones are really dumb. Even when you ask them to search the web for basis you don’t get much success.
I haven’t paid
edit: Well I did for grok, but not on purpose. Grok still failed too.
The bigger paid models or potentially the local ones if given access to search the web can probably get you the right answers. The big ones have pretty much memorized half the internet, but can still be wrong so pushing them to verify their answers.
But the harder part is trusting what they say regardless. I can’t just take an answer for truth, and unless I can verify the statement (fact checking myself, looking up the source, running/testing the code, etc) then it gets harder to do anything with AI. This is the thing I hate about AI in general, people just take whatever they say at face value. Lawyers with fake citations, random people asking chatgpt about random facts and such. Its a tool that people put too much faith in to do thinking for them.
Hence the “are going to get better” and “3-5 years”.
part of the issue as well is that when they get something completely broken, people just re roll the output until they get something that’s broken in ways they don’t notice. Or re roll parts of it, or tell the system to judge if the output is broken and re roll the parts that it judges are broken automatically. Or increase the size of the context window to get it closer to that upper limit of accuracy.
All this together can get a more functional output with less effort, and as people find these tricks it gives them the illusion of an upward trend in capability, like this is all solvable issues that will improve as time goes on. Big problem with that though, theses tricks and methods explode the compute cost rapidly. That’s all fine and dandy when everyone is getting their compute costs for these tools subsidized by these model providers, but eventually they will need to charge the real cost of running this. The compute providers that host the model providers are also running at a loss, trying to help grow the market segment and maximize their market share. And then places that have the datacenters in them are giving tax breaks and discount utilities to attract new construction.
Everyone except the people making the chips is selling at a loss, and as people pile on usage to make up for the fundamental limitations of these systems, the demand balloons, validating to the providers at all levels that this is a growing market they should invest more in to.
But eventually… they need to make money. The bill comes due on all the debt and investment. What happens to the people who have fully embraced these to run their businesses? Or to all the people who have built their skill set around using these systems? It’s a crisis, a series of crisis, each time a debt wall gets hit by someone in the supply chain. A half decade of technical debt that just got really expensive to deal with, and not enough experienced people to handle it, since all the grey beared retired and not enough new people got brought in to replace them because the entry level work was automated.
Don’t forget the vibe coders who can’t tell that the Ai is doing it wrong
The output is broken, but in a way that’s getting harder and harder to detect.
Introducing code homeopathy: “Yes those two programs are consisting of the exact same bytes, but this one contains pure virgin artisan essence (C20 distilled)”
