In a recent post, I argued that models have become highly capable in the Security domain, and that we need to take the threat posed by them seriously. In this post, I want to explore what abundant AI could do for Blue Team, and what we probably must do in order to counter the threat posed by offensive uses of these models.
Recall that these models will continue to proliferate, mainly because there are labs in China producing nearly equivalent models, and releasing the weights, so the cat can't be put back into the bag. In a keynote talk at the SANS Cloud Security Summit, Jason Clinton from Anthropic made a few things clear:
1) Frontier LLMs will continue to scale for at least the next two years.
2) Inference at a particular level of capability is getting 10x cheaper per year as models become both larger and more efficient.
3) The open-weight models are only 6-12 months behind the US commercial frontier models.
The threat will grow, but...
The bad news then is that within 6-12 months there will be an open weight model with the same offensive security capabilities as Mythos.
The good news is that defensive capabilities of the commercial models are already good enough to defend at that same level, and will be up to 10x cheaper by the time of the open, no-guardrail release of a Mythos-level offensive model, and will continue to get cheaper from there.
As models continue to scale, will the threat continue to grow without bound? This is where there's room for optimism. Jason's opinion is (this is paraphrase, but it is his idea so I want to attribute it properly):
"I believe the offensive security capabilities of frontier LLM models are on an S-curve that will top out." -- Jason Clinton
When will they top out? Jason thinks that it's somewhere in the next 12-24 months maybe? Hard to tell. But there is a top where models will become maximally efficient at finding and exploiting vulnerabilities and exposures. It's bad that we'll reach that point, and we definitely need to be ready for it.
Offensive expertise is becoming a commodity
The strategic situation is the one we've always faced on blue team: we have to defend everywhere, but the attacker only has to get in one way. I believe that in the past, the rate limiter on hacking events was just expertise. The number of intrusions was only constrained by the number of people who had the skill to execute them.
Since we're now entering an era where offensive expertise will become abundant, this means that no amount of attack surface is OK. If you're vulnerable, you will be hacked, repeatedly, by persistent adversaries who know every vulnerability, and can chain low-severity vulnerabilities together in real time to get to their goal.
The legacy problem will kill us
Watch this talk by HD Moore to understand how bad the legacy problem is in real networks. If we focus only on current code bases, and don't address the legacy problem head on, then networks will remain vulnerable, and these advanced AI models will get in and move laterally freely.
But, here's the room for optimism, if AI becomes abundant and we leverage it in blue team, because:
Defensive capabilities will also become abundant
Now is the time for us to think about what abundant defensive AIcould look like. Security teams are typically personnel-constrained, i.e. there's not enough people to do all the work. What if that's no longer a problem, and we can apply cheap AI models to pretty much every low-level task that we could never get the time for?
This could mean using AI to:
Scan all existing code you depend on and fix all vulnerabilities.
Scan all code before release and fix vulnerabilities so in future there are no released bugs.
Refactor all IAM configurations to a more defensible least privilege configuration with as much simplicity as possible.
Create a just-in-time privilege oracle that evaluates the risk level of a requested level of access, creating short-term credentials for limited access cases that conform with an overall policy, while forwarding the riskiest requests to a human for approval.
Produce minimized access inbound zero-trust permissions with roles based on past access patterns (use it or lose it), with new authenticated connection requests evaluated on the fly for risk, allowed if consistent with an overall policy, or forwarded to a human for approval (with context) if risk is too high.
Reconfigure all internal networks with point-to-point connections (e.g. AWS Privatelink).
Create and maintain default deny egress configurations for repeated connections, and evaluate outbound requests on the fly and create new permissions as needed unless the risk is too high.
And last but definitely not least: Extract all existing legacy code bases and document interfaces and requirements, then reimplement and replace to modern, type safe programming languages. Build extensive test suites and test against the existing and new code to ensure they are equivalent, then do a canary, then a red/black failover to the new implementation. Continue to do this throughout time.
etc. (use your imagination)
Jason Clinton made the prediction in his talk that large, resourced organizations could reach a zero vulnerability state in 24 months. I believe this will be feasible (maybe not in the same time frame) for smaller organizations as well, for their own internally-written code. The smaller companies (and large non-tech companies) will also benefit from the efforts of the larger tech companies on open source software (which they also heavily depend on).
If security capabilities don't top out, or models are just improving more slowly, then new vulnerabilities could be discovered in already-patched code bases, but then we can fix those too, at a more manageable rate.
AI can do all that? You're funny.
Funny how? Am I a clown to you?
But seriously, there are a lot of things that could happen that would derail the current path for AI. But none of what I said above requires larger models than we already have today, it only requires them to get cheap enough that they are abundant. Will that happen? Of course I don't know. I do think that if the AI hype bubble pops in such a way that overall demand drops, then costs would drop further for Security applications specifically due to excess capacity on the supply side.
Hopefully the thought exercise helps get your creativity flowing. We all need to be thinking about this and making it real to the extent possible. Regardless of what happens from here on in w.r.t. AI investment, we're entering a new Security age. Let's take advantage to fix the things we know need to be fixed.
Allons-y! [1]
[1] My other option was "Excelsior!" Take your pick of which is less cringey.