Closing the loop: the journey towards AI systems that improve themselves

Building a robust production AI system today means orchestrating hundreds of agents, tools, and prompts. So when something breaks or when you try to improve the system, an eval that verifies a single prompt tells you almost nothing about whether your change was an improvement. The hard part isn’t building these systems anymore, it’s understanding them.

I’ll share our journey at incident.io toward self-improving systems, and the tools we’ve built to get there. Measuring how these systems perform is only one part of it. What’s made the biggest difference is how quickly we can act on what we learn. We’ve spent the last year closing the gap between noticing something could be better and proving that a change actually makes it so, until that loop is fast enough to run constantly rather than once in a while. I’ll show how we do that today, how that’s shifted from 6 months ago, and where we think the next 6-12 months are heading.