I read a retrospective from another team yesterday. They were preparing a demo for leadership and apparently it didn’t go so well. There were many reasons like last minute changes in requirements but one line was interesting for this discussion: The use of the LLM agents caused confusion about who is fixing which bug and taking on which tasks. The agents act like they’re alone in the codebase and keep adding and changing code broadly whenever they see an opportunity for improvement. And that often without specifically being prompted to do so. Now if you have multiple team members collaborating on a project with each their agents, chaos ensues because the agents are not coordinating and the diffs/commits become insane and the team doesn’t fully understand them.

This actually results in a necessity to test more and longer because nobody knows which bugs have really been fixed now and rapidly preparing for a release or a demo becomes more difficult than before.

In my opinion an LLM can be very useful to rapidly prototype something. Especially with an API or language you’re not familiar with. But for Production it’s still a huge risk to push code you don’t understand.