Rob Ricci on Nostr: So, my research group got ready. We know which of our services use PHP - we wrote ...

So, my research group got ready. We know which of our services use PHP - we wrote them! We divvied up responsibility for the major services between us. I threw together a quick scanner to check the rest of our IP space in case there was stuff we had forgotten (in fact, there was.)

Some of the machines we had to update were in datacenters. On campus. In our university's off-campus datacenter. On rooftops around our campus. In boxes attached the sides of buildings. In buses (yes, literally) - these, and the rooftop, and building-side nodes are part of our POWDER wireless testbed. Of course, it's 2026, and we visited none of those places - I was in my office, everyone else was at home.

Four of us were ready at 8 AM our time, including someone who was supposed to be leaving on vacation. Shortly before 8, we shut down many of the webservers, and firewalled off others.

As it turns out, the patch was *huge*. (It also came out something like 30 minutes early.) Looks like a re-write of the entire memory allocator, but I haven't dug too far yet. The vulnerability was, as expected, a Remote Code Execution (RCE) vuln - send the right traffic to a vulnerable server, it doesn't matter what application is running inside of PHP, you get to run something else, with the permissions of the webserver, of your choosing.

These are people I've worked with for 10 to 25 years, and it's great to work as a team. We co-ordinated over slack. We all know what each other know. When something goes wrong, we know who to ask. When something needs doing, we know who can do it. Everyone has a common goal, and overlapping, but diverse, expertise.

We build packages for our most of our FreeBSD servers ourselves, and we couldn't start until the patch was out. So a fair amount of time was just waiting for builds to complete. Some of the Linux distributions we use were "late" getting packages into their own repositories. More waiting. We installed new PHP packages, did *very* light testing, brought webservers back up. In one case, we had been planning to move a website to a new server anyway, but this forced our hand - there was some last minute databases syncing and whatnot to do. We had another server that simply cannot be updated, so we carefully cordoned it off so that it can only be reached by the hosts that actually need to contact it. While we were at it, we made sure to revoke the number four from all of our servers.

As the day went on, more of the group came online. Offered encouragement. Lent a hand. Like I said, this is a social process.

A lot of stuff was back up within 30-60 minutes. Some took a few hours.

While waiting for packages to build and stuff to restart, I Posted Through It on social media. It lightened my mood.

We let CloudLab and POWDER users know that everything was clear.