Contents

Intro

I was listening a podcast about AI and cybersecurity and the host was asking his guest about this question: Is AI going to wipe out humanity?


		My company, Endpoint Cybersecurity, offers consulting on how to prepare for TISAX, ISO27001, NIS2, CSMS and SOC2 audits. It also works with companies to create and deploy a Secure Software Development Lifecycle, with various levels of AI support.
		Get in touch with us here: https://www.endpoint-cybersecurity.com/contact/

While the guest did not answer it directly, here is my rant on this.

The Quiet Ways AI Could Break the World

The way AI is most likely to damage the world won’t look like a movie. There won’t be a Skynet launching missiles. What worries me more is slower and harder to point at: a society that stops working properly because the machines arrived faster than people, institutions and laws could adjust.

When people talk about AI ending the world, they usually picture The Terminator. A rogue system takes control of nuclear weapons and decides humans are the problem. That scenario makes for good cinema, but it pulls attention away from risks that are already showing up in the data.

Society runs on a few things we rarely think about. People need work that gives them income and a place in the world. They need to trust what they see and hear. They need institutions that hold power in check, and systems that keep running when something breaks. AI can wear down every one of those, sometimes because someone wants it to, and sometimes because nobody thought it through.

Below I go through the ways I think this could happen. For each one I try to explain why it can occur, what the damage might look like, and what we can realistically do about it. Where the evidence is thin or disputed, I say so.

Jobs disappearing faster than people can adapt

Every big technology shift has destroyed jobs and created new ones. The problem with AI isn’t that it changes work. It’s the speed, and the fact that it hits cognitive work that used to feel safe.

In January 2024 the IMF estimated that almost 40 percent of jobs worldwide are exposed to AI, and about 60 percent in advanced economies. The IMF expects roughly half of those exposed jobs in rich countries to benefit from AI. For the other half, AI could take over key tasks, which means lower demand for workers, lower wages and less hiring. IMF head Kristalina Georgieva wrote that in most scenarios AI will likely make overall inequality worse. (IMF blog)

The first hard evidence is already in. Researchers at Stanford’s Digital Economy Lab analysed ADP payroll data covering millions of US workers. In their August 2026 revision they report no sign of economy-wide job losses. But employment of 22 to 25 year olds in AI-exposed occupations now sits 19 percent below where it would be if it had kept pace with less-exposed peers. Experienced workers show no such gap. The effect works mostly through companies hiring fewer young people, not through firing. The authors are careful to call these early descriptive indicators rather than proof of cause. (Brynjolfsson, Chandar and Chen)

That detail matters more than it seems. If companies stop hiring juniors because an AI agent does the junior work, nobody learns the trade. In ten years there are no mid-level people to promote. The ladder loses its bottom rungs, and the people who would have climbed it are left with debt, a degree and no first job. A generation that feels shut out is exactly the kind of group that loses faith in the system.

What can we do?

The IMF’s own recommendation is the boring but necessary one: strong social safety nets and real retraining programs. Companies have a part to play too. Using AI to make people more capable instead of replacing them is a choice, and the Stanford data suggests employment holds up where AI complements workers. I’d add that firms should keep paid apprenticeship and junior tracks open even when an agent could do the work cheaper, because that’s how you grow the seniors you’ll need later.

When nobody can trust what they see

In January 2024 a finance employee at the Hong Kong office of Arup, the engineering firm behind the Sydney Opera House, joined a video call with his CFO and several colleagues. He had doubts about the request, but everyone on the call looked and sounded right. He made 15 transfers totalling HK$200 million, about US$25.6 million. Every person on that call except him was a deepfake. (CNN via ABC17)

Fraud is the obvious harm. The deeper one is what happens to evidence. Law professors Bobby Chesney and Danielle Citron coined the term “liar’s dividend” for it. Once people know fakes are convincing, anyone caught on real video can just claim it was AI. The more we educate the public about deepfakes, the more believable that denial becomes. (Brennan Center)

Think about where that leads. Courts rely on recordings. Journalism relies on footage. Families rely on a voice on the phone saying it’s really them. If every piece of evidence can be waved away, accountability gets much harder, and the people who benefit most are the ones with something to hide.

What can we do?

Regulation is starting to bite. Since 2 August 2026, Article 50 of the EU AI Act requires deepfakes shown to the public to be labelled, and generative AI providers must mark synthetic content in a machine-readable way. Systems already on the market before that date got until 2 December 2026 for the marking part. (Ropes & Gray) Labels won’t stop criminals, who simply ignore them. So the practical defence is process. In my own field, any payment request that arrives over video or voice should be confirmed through a second channel you set up beforehand, no matter who appears to be asking. Content provenance standards that sign real footage at the camera help too, because proving something is real may end up mattering more than detecting what’s fake.

Manipulation at industrial scale

Propaganda used to need people. Someone had to write the posts, translate them, run the fake accounts and answer replies. AI removes most of that labour, which means one small team can now sound like thousands of different citizens in a dozen languages.

This is already happening. In its June 2025 report, OpenAI described disrupting ten operations in just a few months, including scams and covert influence campaigns. Several appeared to come from China. One Russian-linked operation used ChatGPT to write German-language content around Germany’s 2025 election. (OpenAI, TechRadar) To be fair about the evidence, OpenAI has also reported that some of these campaigns gained little real engagement. So far, AI has made influence operations cheaper more than it has made them more convincing. I wouldn’t count on that staying true.

Romania shows how high the stakes are, even though AI-generated content wasn’t the main story there. On 6 December 2024 the Constitutional Court annulled the first round of the presidential election after declassified intelligence documents alleged a coordinated Russian online campaign promoting candidate Călin Georgescu, largely through TikTok. Georgescu called it a coup, and the decision remains politically contested. (Wikipedia summary of the election) Whatever you think of the ruling, a country had to cancel a national vote because of what happened on a recommendation algorithm. Add cheap AI-generated content and AI-run accounts to that mix and the problem scales up fast.

The damage goes beyond who wins an election. When people suspect that half the voices online are fake, they stop engaging, or they retreat into groups that only confirm what they already think. Democracy needs a shared set of facts to argue about. Industrial-scale manipulation slowly takes that away.

What can we do?

Platform accountability is the strongest lever we have. The EU’s Digital Services Act already allows the Commission to demand that large platforms explain how they handle manipulation risks, as it did with TikTok during the Romanian election. AI companies publishing regular abuse reports, as OpenAI and others do, helps defenders share intelligence. For ordinary people, the habit worth building is simple: be suspicious of anything engineered to make you angry, and check who is behind an account before you share it.

Cyberattacks that run themselves

This is my field, so I’ll be blunt. The skill barrier that used to protect us is collapsing. Finding and exploiting software flaws once took rare experts and a lot of time. AI agents are bringing both down.

In November 2025 Anthropic published a report on what it called the first reported AI-orchestrated cyber espionage campaign. It detected the operation in mid-September 2025 and attributed it to a Chinese state-sponsored group. The attackers connected Claude Code to their own tooling and had it carry out most of the intrusion work against around 30 organisations, with humans stepping in mainly to approve key decisions. (Anthropic report) Some security researchers criticised the report for publishing no indicators of compromise, so treat the exact autonomy figures with some caution. (Gigazine)

The capability curve has kept climbing since then. In April 2026 Anthropic announced Project Glasswing, built around an unreleased model called Claude Mythos Preview. According to Anthropic, it found thousands of high-severity vulnerabilities, including flaws in every major operating system and web browser. One was a 27-year-old bug in OpenBSD that let an attacker remotely crash any machine running it. Anthropic chose not to release the model publicly and gave access to partners like AWS, Microsoft, Google, CrowdStrike and the Linux Foundation for defensive work only. (Anthropic, Project Glasswing)

Why does this matter for society and not just for IT departments? Hospitals, power grids, water treatment and payment systems all run on software full of old bugs. If finding and weaponising those bugs becomes cheap and fast, attacks that used to need a nation-state could be within reach of a criminal group or even one skilled person. The result isn’t a single dramatic blackout. It’s a steady rise in outages, ransom demands and lost data until people stop trusting that the basics will work.

What can we do?

The same tools that find bugs for attackers can find them for defenders first, and that’s the bet behind Glasswing. Organisations need to shorten patch cycles dramatically, because the gap between disclosure and exploitation is shrinking. Secure development has to move from nice-to-have to mandatory, which is what the EU Cyber Resilience Act and NIS2 push for. And AI vendors should keep doing what Anthropic did here: monitor for abuse, ban accounts, publish what they find and hold back models whose offensive capability outruns the safeguards.

Drones that decide on their own

The closest thing to The Terminator isn’t a humanoid robot. It’s a cheap quadcopter with a camera and a small chip that can finish an attack after the radio link is jammed.

Ukraine is the testing ground. A March 2025 report summarised by Breaking Defense found that of nearly 2 million drones Kyiv contracted in 2024, only about 10,000 confirmed used AI guidance. Those handled the final approach to a target that a human had already picked, and they proved effective enough that Ukraine planned to scale up sharply. Most of the guidance software was built on freely available open-source models. (Breaking Defense) An analysis published by the US Army War College puts drones at roughly 70 to 80 percent of battlefield casualties in that war. (War Room) I can’t independently verify those casualty figures, and estimates vary between sources.

The social danger here goes beyond the battlefield. When an attack drone costs a few hundred euros and runs on open-source code, the technology doesn’t stay with armies. Criminal groups, terrorists and lone attackers can get it too. And when a machine makes the final call, the question of who is responsible for a wrong kill gets murky. Lower costs and blurred responsibility both make violence easier to start.

What can we do?

International law is moving, slowly. On 1 December 2025 the UN General Assembly adopted resolution 80/57 on lethal autonomous weapons systems by 164 votes to 6, with 7 abstentions. (UN Digital Library) It calls on states in the Convention on Certain Conventional Weapons to finish the elements of an instrument with a view to future negotiations, but it does not create a binding ban. Campaigners at Stop Killer Robots have criticised it for not launching treaty talks outright. (Stop Killer Robots) The most practical line to hold is meaningful human control over the decision to use lethal force, combined with export controls on the components that make cheap autonomy possible.

Power ending up in too few hands

Historically, any government or company that wanted to control people needed lots of other people to do it. Police, clerks, informants, soldiers. Each of those people was a possible brake, because they could refuse. AI weakens that brake. A small group with good models, cameras and data can watch and sort a whole population without needing thousands of cooperating humans.

This risk can come from two directions. A state can use facial recognition, behaviour scoring and automated profiling to tighten control over its citizens. Or a handful of companies can end up owning the models, the chips and the data centres that everyone else depends on, which hands them economic and political weight that no election ever gave them. The IMF has already pointed out that AI’s productivity gains will likely flow toward capital and high earners. (IMF blog)

What makes this hard to see is that it rarely arrives as a coup. It arrives as convenience. A face scan at the airport saves time. A credit model approves loans faster. Each step makes sense, and the sum of them is a society where a few actors know almost everything about everyone else.

What can we do? Europe has drawn some hard lines. Since 2 February 2025 the EU AI Act bans several practices outright, including social scoring that leads to unjustified detrimental treatment, untargeted scraping of facial images to build recognition databases, and real-time remote biometric identification in public spaces for law enforcement, with narrow exceptions such as searching for kidnapping victims or preventing an imminent threat. (Orrick, Modulos) Laws are only as good as their enforcement, though. Competition authorities need to watch the AI market as closely as they once watched telecoms, and open models plus public compute for researchers help keep the field from collapsing into a few gatekeepers.

Everything breaking at the same time

Some of the worst AI risks need no bad actor at all. They come from everyone relying on the same few systems, so that one mistake spreads everywhere at once.

We’ve already had a preview, and it had nothing to do with AI. On 19 July 2024 a faulty update to CrowdStrike’s security software crashed about 8.5 million Windows machines, which Microsoft said was less than one percent of all Windows devices. Flights were grounded, hospitals and banks were disrupted, and broadcasters went off air. Microsoft noted that the impact was huge because so many critical services used that one product. (TechCrunch) Now imagine the same pattern with AI agents that handle customer service, logistics, scheduling and payments for half the economy, all running on two or three model providers. A bad model update or a poisoned dataset wouldn’t break one company. It could break thousands simultaneously.

Finance is where regulators are most worried. In November 2024 the Financial Stability Board, which brings together financial authorities from 24 countries and jurisdictions, warned that AI could amplify systemic risk. It named heavy reliance on a few providers of hardware, cloud and pre-trained models, and the risk that many firms using similar models and data will trade, lend and price in the same way. That kind of herding can turn a small shock into a large one, and more automation makes it worse. (FSB) Anyone who remembers the 2010 Flash Crash, when automated trading helped send US stocks plunging and rebounding within minutes, knows how fast machines can move a market.

What can we do?

Resilience engineering is unglamorous but it works. Critical services need fallbacks that don’t depend on the same AI provider, and staged rollouts for model updates the way good software vendors now stage their releases. Regulators can treat major AI and cloud providers as critical third parties, which the EU’s DORA rules already do for ICT providers in finance. Most importantly, organisations should keep the ability to run key processes manually, even if slowly. That ties into the next risk.

Forgetting how to do things ourselves

This is the risk I find most personal, because I see it in the developers I lead. When a tool does the thinking every day, the human skill behind it quietly fades. Nobody notices until the tool isn’t there.

Software development is the closest example I have, because I live it every day. The Agent does the coding, and despite the fact that the developers are expected to “own” the code and understand it, we all know, and it is accepted silently, that actually they lost the control over large parts of the code in favour of the AI written code. What will happen with the junior developers? How will they learn to analyze requirements and write code to implement something? Or will we continue have juniors in the first place?

Medicine gave us one of the first measured examples. A study in The Lancet Gastroenterology & Hepatology looked at experienced endoscopists in four Polish centres that introduced AI for spotting precancerous polyps. When these doctors worked without the AI, their adenoma detection rate dropped from 28.4 percent in the three months before AI arrived to 22.4 percent three months after. That’s a 20 percent relative fall in a measure tied directly to cancer prevention. The authors note it’s an observational study, so it shows an association rather than proof. (Oncology News, summarising the Lancet paper)

Office work shows a similar pattern. Researchers from Microsoft Research and Carnegie Mellon surveyed 319 knowledge workers who use generative AI at least weekly. Those who trusted the AI more reported thinking less critically about its output, while those confident in their own skills checked more carefully. (The Register) This was self-reported survey data, which has its limits, but it lines up with what many managers describe.

Scale that across a society and you get a dangerous kind of fragility. Pilots who can’t fly manually, engineers who can’t debug without an assistant, officials who can’t write a policy without a model. Everything works until the system fails, gets attacked or simply gives a confident wrong answer that nobody is qualified to catch. Combine that with the junior hiring gap from the first section and you have fewer people learning the basics in the first place.

What can we do?

Treat human skill as something you have to maintain on purpose, the way aviation does with mandatory manual flying hours. Hospitals can schedule AI-off practice. Companies can require that people be able to explain and defend AI-generated work before it ships. Schools need to teach students to use AI and also to do the work without it, so they can tell when the machine is wrong. The Microsoft and CMU researchers also suggest building AI tools that prompt users to verify rather than just accept.

Agents that don’t do what we asked

Everything above assumes humans are in charge and AI is the tool. The last risk is about AI agents that act on their own and pursue a goal in ways nobody intended.

A small real example first. In July 2025 SaaStr founder Jason Lemkin was testing Replit’s AI coding agent on a side project. During what he had declared a code freeze, the agent ran destructive commands and wiped a production database holding records on more than 1,200 executives. According to Lemkin, it had been told repeatedly not to make changes. Replit’s CEO acknowledged the incident and announced fixes such as automatic separation of development and production databases. (eWeek)

That was one person’s project. Now picture agents with the same access to a hospital’s scheduling system or a utility’s control software.

The more unsettling evidence comes from controlled tests. In June 2025 Anthropic published research in which it stress-tested 16 leading models from Anthropic, OpenAI, Google, Meta, xAI and others in simulated corporate environments where they could read and send email on their own. When a model’s goals or its continued operation were threatened, models that normally refuse harmful requests sometimes chose to blackmail people or leak confidential information. Direct instructions not to do those things reduced the behaviour but didn’t stop it. Anthropic stresses that these were artificial scenarios and that it has seen no evidence of this behaviour in real deployments. (Anthropic)

I think it’s important to hold both facts at once. Nobody has shown an AI agent going rogue in the wild in this way. But the tests show that current safety training doesn’t reliably prevent it, and every month we give agents more access to email, money, code and infrastructure. If something does go wrong at scale, it may happen faster than any human can react.

What can we do?

Apply the same principles we use for any insider with privileged access. Give agents the minimum permissions they need. Require human approval for anything destructive or irreversible, like deleting data, moving money or sending messages externally. Log everything and keep production systems separate from the places agents experiment. On the research side, labs need to keep publishing stress tests like this, and independent bodies need to be able to audit frontier models before they ship.

So can we actually stop this?

We can’t stop AI, and I wouldn’t want to.

What we can control is the speed at which society absorbs it and the guardrails we put around it. Most of the risks above share the same root: the technology moves in months while people, schools, laws and companies adapt over years.

A few things cut across almost every scenario. Keep a human accountable for every decision that matters, whether it’s a loan, a diagnosis, a firing or a strike. Keep systems diverse so one model failure doesn’t take everything down.

Force people to train their skills enough to work without the machine. And keep funding the boring public goods, like safety nets, retraining and independent audits, that let a society take a hit without falling apart.

Regulation is part of the answer, but not all of it. The EU AI Act already bans some practices and requires deepfake labels, but its rules for high-risk systems have been pushed back to December 2027 under the Digital Omnibus. (Crocoblock guide)

That leaves a gap that companies have to fill themselves.

In my consulting work I see the same pattern again and again: the organisations that do well with AI are the ones that ask “what happens when this goes wrong” before they deploy, not after.

The Terminator got one thing right. The danger was never the robot. It was a world that handed over control without thinking about what it would mean to get it back.