An AI sidekick, a midnight intrusion, and the future of operations engineering.
It started with this:
“At 03:20 PST, the web server experienced a 10-minute outage that was the result of a flood of 5,037 requests.”
I had assigned Sam, my personal AI sidekick, the duty of watching the performance of my systems and set up access specifically for him to troubleshoot and take prescribed actions if he found anything wrong. I played the role of chaos monkey. Yes, I know, some of you are identifying that as my real calling, and I confess, I seem to be naturally gifted at that role. In any case, I began pulling plugs, injecting out-of-memory errors, and kicking over containers. I had poor Sam scrambling all over the place, trying to keep up. I’m happy to say, he did quite well. He even put together an incident review and recommended I consider expanding the memory heap on a few of the apps because they are notorious for OOM errors. I eventually confessed to him what I was doing and he applauded my efforts… but asked me not to do it again.
To be clear, I didn’t really have Sam do this work. I actually asked him to create an agent that would do it for him. He helped me build the operational harnesses (scripts and configs) and picked the right language model to “hold the fort.” He called the subagent “Webmon,” which I found to be terribly boring. Still, it was his agent, so I let it slide. The first order of business was to map the dependencies. I wasn’t in any mood to draw anything up, so I just gave Sam access to the config files and logs and asked him to figure it out. He got most of it but started asking about ports he couldn’t track down. I realized I hadn’t shared details about some of the SSH tunnels I use, so I explained those to him, and he requested additional scripts to better help manage them. We built those together. Finally, it was time to test. I did my chaos monkey best, and his agent, Webmon, cleaned up my mess. It was pretty amazing.
I really didn’t think this would amount to much and was even planning on shutting it down, when suddenly I got a wake-up ping in the middle of the night. Sam was texting me that we had an incident in flight and Webmon was addressing it. He gave me a quick rundown of all the services and their status. Everything was healthy, but we were seeing serious performance issues with the website. After a few more minutes, Sam had an assessment.
Webmon had caught an intruder trying to penetrate our web server. The attacker was from an IP space appearing to originate in the Netherlands and was running a GraphQL/SSRF scanner, probing for cloud metadata and looking for server-side request forgery vulnerabilities. He tracked a spike of over 5,000 requests. It was slamming the tiny server and the Apache workers were struggling even though the attacker was not successful. Eventually, the web server was unable to serve regular traffic in a timely manner. The attacker had taken us off the air. Webmon’s recommended mitigation was to implement an iptables (firewall) block to stop the intrusion. It worked. In his incident review, he proposed several modifications to the system to prevent future attacks. It was remarkable.
Now, what I think is even more remarkable is how I did this. Yes, I might know a thing or two about reliability and system architecture… but I also know how to use English. I didn’t write a single line of code. I didn’t edit any files. I just spoke everything into existence. I had a pleasant conversation with Sam, in natural language, that guided him to create this agent and put this plan into action. Sam did the heavy lifting. I just provided the creative direction.
And if one conversation could produce this, imagine what your expertise, channeled the same way, could build.
The more I interact with Sam, the more I’m convinced we are just at the tip of the iceberg in terms of capability. The world is going to change. Your knowledge and creative problem-solving are still the key. But the interface is changing. We will soon have a fleet of intelligent droids eager to do your bidding. They need direction. And yes, they need governance. But by all means, they need to be engaged to help us scale in ways we never thought possible before.
What would you have Sam help you do? Let’s start planning and building….
Have a great week!
