Talk
AI/ML
AI/ML Closing the loop: the journey towards AI systems that improve themselves
Leo Sjöberg
Product Engineer at incident.io
Building a robust production AI system today means orchestrating hundreds of agents, tools, and prompts. So when something breaks or when you try to improve the system, an eval that verifies a single prompt tells you almost nothing about whether your change was an improvement. The hard part isn’t building these systems anymore, it’s understanding them.
I’ll share our journey at incident.io toward self-improving systems, and the tools we’ve built to get there. Measuring how these systems perform is only one part of it. What’s made the biggest difference is how quickly we can act on what we learn. We’ve spent the last year closing the gap between noticing something could be better and proving that a change actually makes it so, until that loop is fast enough to run constantly rather than once in a while. I’ll show how we do that today, how that’s shifted from 6 months ago, and where we think the next 6-12 months are heading.
Building for the future means building systems that can take advantage of new models and ever greater intelligence the moment it’s available, with the confidence that it makes your product better. The systems that are winning the future won’t be the ones wired to today’s best model; they’ll be the ones built to keep improving as the models do.