Wasn't Y2K a non-issue?
The mug above was issued to a relative of mine, who was on an IT team at a large US telecom that worked on remediating Y2K issues before the year 1999 rolled over to 2000. If you heard that Y2K was a non-issue, that is true, but only because of the heroic work by teams like the one that got this mug. A large-scale effort, across many or even most companies, fixed the code where two-digit years were used, which could have caused issues as date math like Year2 - Year1 would break because 00 - 99 = -1, or in some cases 255, 32767, etc. instead of the desired answer 1.
Some credit card expiration math was already failing once credit cards were issued with expiration year of '00' because 97 > 00 therefore the card was deemed expired. There were other issues already happening such as with mortgage amortization calculations. There's a good Reddit thread about it.
But, the community rallied, companies spent what they had to spend, and the problem was mostly remediated, allowing New Year's 2000 to be a celebration rather than disaster response.
What's similar about offensive AI?
I've argued that AI's offensive cybersecurity capabilities have already reached a point where, once open-weight models fully catch up, a tipping point has been reached. Vulnerable networks will be breached, probably repeatedly, as offensive AI becomes abundant and democratized. Worse: networks will be breached even when not known to be vulnerable (i.e. patching known vulnerabilities will no longer cut it) due to AI's capabilities in finding zero day vulnerabilities.
Because of the dynamic and adaptive nature of offensive AI, we will need to not only remediate all known vulnerabilities, but discover all new vulnerabilities that can be discovered at each level of AI that is released, and remediate them all. (AI agents have proven they are very good at chaining medium/low vulnerabilities into a full compromise).
We will also need adaptive defenses that can detect and adjust to novel attack patterns, deploy defensive measures in real time, and drive patching and remediation on their own (with high-level human oversight). More detailed suggestions below.
What happens if we don't do this?
...dogs and cats living together, mass hysteria! -- Peter Venkman, Ghostbusters
When experts first warned about Y2K, business leaders delayed and procrastinated when they saw how much it would cost to fix the issues before the year 2000. But tabletop exercises, simulations and tests showed that there were real problems that would have caused quite a lot of havoc if they had been allowed to happen.
A lot has happened since Y2K. We have become absolutely, completely, intimately and irretrievably intertwined and dependent on computers and networks. There is not a single area of human life that would be unaffected by massive, at-scale compromise of computer networks.
I can't see the future obviously, but let's look at some of our critical systems:
The power network in the United States has outdated and bespoke hardware and software, lack of redundancy, and long-term lack of investment. If the power network went fully down it could stay down for 18 months or more.
Water systems in a dozen states have been hacked recently, so far without disastrous consequences (though one district did have to issue a boil water warning). A more determined attacker (or one persistent at causing damage but unaware of or indifferent to the consequences) could have done much worse.
Similar scenarios for other critical infrastructure could threaten food safety or availability, lead to toxic chemical releases or worse, create financial system impacts or crises including at-scale theft, expose the healthcare system to novel and systemic risks, etc.
Moving beyond critical system disasters, simple widespread outages of the Internet would bring modern life to a standstill. What aspect of your life can you fully accomplish without your computer or your phone? Even if you had the cash to buy groceries, would grocery stores even be able to accept the cash or account for how much of it they received, let alone run the supply chains that bring the food to the store?
Aren't you overreacting?
I sincerely hope so. I'm honestly not trying to be alarmist. But it's always been my belief that the rate of critical systems being hacked was only limited by the number of skilled and motivated people doing the hacking. With democratization and commodification of offensive AI, that bottleneck is removed. We would have to assume that all of these systems would be hacked, because they can be. The attackers wouldn't even need to intend to cause damage at this scale, they would just need to provide suitable AI agents with compute, ask for hacking, and point vaguely at the world.
Can we make AI agents that aren't good at hacking?
Not if they are also good at programming and analyzing systems. Hacking is just finding unexpected execution paths, which has always been a surprise, a delight, and a risk. Hacking cons like DEFCON attract a large audience because hacking is fun, and if you understand how systems work then you're also going to see how they can break.
We need AI that understands how systems work, and can build new systems. The hacking capabilities are a necessary and unavoidable side effect of those capabilities. Guard rails can be good (a topic for another time) but can never be foolproof.
Can we just make everyone review code?
For Y2K, COBOL programmers were brought out of retirement to comb over ancient code and fix the two-digit years (sometimes in their own code). Now that the scale of deployed code is far greater (Y2K was 26 years ago and the rate of code production has grown exponentially), and the legacy problem far worse since it covers all code and not just date calculations, we don't have the luxury of bringing people out of retirement to fix this. There just aren't enough people with the skills to do the remediation.
One estimate is that 2.8 Trillion lines of code have been written in the last 20 years. 30% of new code produced in 2024 was generated by AI, and that proportion has almost certainly increased since then. Unfortunately, that code contains security flaws more often than human-created code (though that should improve over time), mainly because the profit motive (as it always has) incentivizes functionality over security.
Even if we could put 2.8 million programmers onto the job (and they stopped writing new code), each would have to review 1 million lines of code in (for estimation purposes) 20 months. Let's say 20 working days per month, then they would each be reviewing 2,500 lines per day for flaws. While review speeds are hard to estimate, Google's experience is that their engineers spend about 3 hours a week reviewing 4 (mostly) small changes of ~40 lines. If they reviewed at that rate continuously (8 hrs x ~40 lines per hour = 320 lines per day) they would fall far short of the rate needed. Speeding up the rate of review leads to problems being missed, undermining the point of the whole enterprise. And you're certainly not going to 10x those numbers just using humans.
Plus, many systems (especially infrastructure) are running obscure and outdated software (often older than those 20 years) on hardware that few experts are familiar with. Reviewing and fixing all that code is beyond the skill of most programmers. Or more accurately: they just haven't looked at those systems ever before.
Besides, there's a lot more needed than just reviewing and fixing code. As mentioned, we need to build robust dynamic defenses also. Furthermore, not writing any new code would most likely crash the economy, and would certainly put a lot of software companies out of business as their feature delivery dropped to zero. Unless this work was mandated, companies that avoided the vulnerability work would have a distinct market advantage (for as long as they could avoid the consequences).
OK, you've convinced me. What do we need to do?
I think we'll need a concerted effort across all areas of Security, with significant investment from companies and governments. We will need regulatory requirements and incentives so that some organizations don't ignore the effort to gain temporary advantage.
We need to scale out and up applying AIto reviewing and fixing code, for all systems including legacy systems. Where code is not available (i.e. someone produced a binary at some point in the past and the code is lost), reverse engineer and decompile the code, analyze and fix it, and produce a new version that is secure. We need legislative and regulatory top-cover to make such efforts fully legal if the code owner cannot be found in a reasonable time frame, or is uncooperative.
We need to produce test harnesses that can run the old and new code bases and ensure that the behavior is the same. Yes, I realize that's the same as solving the halting problem. We'll need to make do with testing all the edge cases we observe on the actual networks, from code analyses and from system-level requirements. AI can assist in those efforts as well.
Design vulnerabilities will be a particular problem. Even when every component of a system is secure, emergent behaviors of the whole system can result in insecure or unsafe states. We need to perform holistic system-level analyses to uncover these kinds of weaknesses. We need to design AI-driven systems that perform these analyses and bring in context beyond the system being studied to estimate the consequences.
Every defensive system and capability will need to be reviewed, enhanced and in many cases rebuilt for the age of offensive AI. AI needs to be adopted aggressively across all Security domains.
Regulatory audits need to become dynamic and continuous. The age of checklists is over (if there ever was an age where that was effective). Instead systems need to be continuously examined, tested, and red-teamed by AI agents.
I'm sure there are important efforts missing from this list. I'm happy to hear further suggestions. We need all our best thinking applied to this problem.
All of these efforts will need to be directed and overseen by humans, but in ways that are scalable. Layers of analysis and visualization should produce high-level insight into the efforts so that humans can apply judgement and make good decisions. Humans are the only ones who can decide what should be done, but we will need to heavily rely on AI to turn that into action in order to do it at scale.
Why are you shilling for the AI companies?
I don't think I am. AI is now a global capability, being advanced by different labs across the world, and most notably across two completely independent geopolitical and regulatory domains: the US and China. Like it or not, the cat is out of the bag and no particular company is going to be the sole beneficiary of this kind of effort.
I don't work for, nor do I own (pre-IPO) stock in, either of the big 2 AI companies in the US. I am not disproportionately invested in AI any more than the big index funds (and I think actually less so if I've done things correctly). However, I did work for Databricks and have (pre-IPO) stock in it, so I do have an interest in them succeeding in the market. But Databricks is a provider-agnostic system that can plug in any model including the free/open ones.
I am co-founder and CEO of Perj Security, which is building a product that uses AI, which pretty much any new product would. To the extent that we become part of the solution (as I would like us to be), we would be only a small part of it.
Why is this happening?
The decision to release ChatGPT 3.5 in November 2022 was the watershed event. Everything that has happened since then is the snowball growing bigger. Think about how much has changed in those 3 years. For reasons understood by no one, AI-generated video of Will Smith eating spaghetti is an effective (and funny) illustration of how far we've come in one particular area. I can confirm this same rate of progress is happening in offensive AI cybersecurity.
Thankfully, defensive AI capabilities have grown along with the offensive capabilities (though perhaps defense lags behind, certainly in how it's being applied). We can figure out how to guide the snowball and build a robust track for it to roll on, or we can get rolled over and crushed. Now's the time in Security to build the track.