Rob Ricci on Nostr: So what do I take away from this? First, we have had days like this before. Not ...

So what do I take away from this?

First, we have had days like this before. Not exactly like this, but kind of like this. The first one I personally remember is when the Code Red worm hit in 2001. We didn't run any vulnerable servers, but the huge increase in traffic all over the world just wrecked everything all day. In 2003 there was a remote root exploit in OpenSSH, get root on any machine in the world without a password. In 2008, it was discovered that most Debian machines had the same SSH keypair for the root user, due to a fixed seed when generating them "randomly." Again, free root on a ton of machines. In 2017, Spectre came out kicked off an era of side channel attacks. There are lots of others. So, the world being on fire, all at once, is a thing that has happened before, and a thing that will definitely happen again.

Does AI change this? Yes and no. It *is* being used to find new vulnerabilities. And, as we saw above, the act of fixing a vulnerability, itself, makes the exploit public knowledge too. But you have to do it: you have no idea who else might have already known about it, and even have been exploiting it, quietly. We've had other rounds of tools used to do mass scans of software looking for bug and vulnerabilities: program analysis, fuzzing, etc. Each have kicked off waves of bug reports, then eventually become part of the standard bug-hunting toolkit. That said, LLMs seem to be causing a much bigger wave than earlier tools.

But.

I believe that only *part* of LLMs' effectiveness in finding and exploiting vulnerabilities comes from the technology itself. Yes, they are surprisingly good on both fronts. They do seem to be getting better at it. But you cannot ignore the economic factor: there have been orders of magnitude more investment, over a much shorter time, than any other bug-finding tool. How much of their effectiveness comes from the fact that, right now, a whole lot of people - especially the frontier AI labs - are willing and able to spend tens of billions of dollars training new models and millions of dollars (probably more? a lot more? I haven't seen any estimates so I'll stick with a low number) pointing them at open-source software?

I really don't know the answer to this question, and I don't get the sense that anyone else *really* does, either - we're just going to have to wait and watch and learn. While the system is stuffed full of investors' cash, I expect we're going to get a lot more AI-assisted vulnerability and exploit finds. Does the system transition to being stuffed full of consumers' cash? Does it level off or contract? People have clearly placed their bets, but we have to wait to find out.

And this brings me to another point, which, I must say, is going a bit out on a limb, but which I think is very important - I'd really like to hear people's feedback on this.

Right now, finding and exploiting security vulnerabilities in software is good PR for the AI labs.

Right now, while they are spending money that was invested, not earned, and are desperately pushing for more investment, it makes a lot of sense for them to put special attention into training models for this purpose, and to spend inference money, themselves and with security research partners, on the task.

Look: today, hundreds of thousands? millions? more? of administrators of all kinds spent their day on defense. And they know it was because of AI. Hundreds of millions of other people have heard us complaining, or being awed, or being smug that we don't use NodeJS, or whatever.

It's a very, very good demonstration of the very real effectiveness of the technology.

At some point, the labs need to actually make money, though.

Is this going to be a good way for them to do it, long-term?

Is overall model improvement just going to be so effective that it doesn't matter if they spend money on this specific use case, it'll just keep being great?

It's worth noting that every other bug finding technology has generally followed a trajectory where it's *very* effective at first, opening up new classes of vulnerabilities. Once the kinds of bugs it's good at finding have mostly been found, it's a long, slow path of incremental improvement. Still useful, becomes part of the permanent toolkit, but the shine wears off. If the PR value of finding bugs with LLMs fades, and it becomes just another use case, will it start looking like the other tools in our toolbox? Or is this fundamentally different? I don't trust anyone who thinks they have a definitive answer to this question. My hunch is that we have a bunch of turbulence, then reach a new equilibrium, but I could be wrong.