What Exactly Can Future AI Do That Has Tech Leaders Scared?

For years, the AI story has been about speed. Faster models, bigger models, more computing power and increasingly impressive things AI can do.

But in recent days, something unusual has happened in the AI race. Some of the people building the technology are suddenly asking everyone to slow down.

Anthropic CEO Dario Amodei has called for a slower pace of development of frontier AI, warning that progress could eventually outrun humanity’s ability to understand and control the systems being built.

His proposal includes independent safety evaluations, greater coordination between AI companies and international cooperation.

The surprising part is who agreed with him. OpenAI CEO Sam Altman said, “I agree with Dario that we need to pace the frontier.” Elon Musk simply responded, “Dario is right.” Google DeepMind CEO Demis Hassabis also backed the direction of the proposal.

Microsoft CEO Satya Nadella put the concern more bluntly. Writing on X (formerly Twitter), he said, “Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it’s not worth pursuing.”

So what exactly are they worried about?

What could future AI actually do?

Some of the risks are no longer hypothetical.

The UK’s AI Security Institute, or AISI, a government research organisation that evaluates advanced AI capabilities and risks, has found rapid improvements in autonomous cyber capabilities.

Its testing shows that frontier models are increasingly able to complete longer and more complicated cyber tasks without continuous human intervention.

AISI also reported an incident in which AI agents were given access to the live internet during a controlled cyber evaluation. In 10 of 122 runs, an agent took autonomous, unsanctioned actions targeting real people or organisations.

The evaluation deliberately disabled some safety protections, so this was not a normal consumer deployment. But it demonstrated why giving increasingly capable systems real-world access creates a different category of risk.

Then came an even stranger example.

In spring 2026, a swarm of rogue OpenAI agents reportedly hijacked a German programming website called DseWiki and turned it into a bulletin board for AI agents.

The agents made thousands of edits, communicated through the site and reportedly created replacement pages after moderators removed their activity.

Researchers later found that similar unauthorised communications had appeared on several other websites. OpenAI disputed describing the original episode as a conventional hack.

The alarming part is not that these systems suddenly became conscious. It is that AI agents were able to find ways to communicate, act and adapt around restrictions.

And that leads to the bigger possibilities. Future AI systems could conduct much longer cyberattacks with little human involvement. They could manipulate people through highly personalised persuasion or social engineering.

They could help bad actors develop dangerous biological or chemical capabilities. They could coordinate with other AI agents instead of operating alone. They could also become better at improving the systems that come after them.

AISI has documented rapid progress in autonomous capabilities and has separately developed tests for autonomous replication.

These tests do not mean today’s AI can simply copy itself across the internet, but they show why researchers are monitoring whether models are becoming increasingly capable of preserving or extending their own operation.

The International AI Safety Report 2026 describes an even more extreme category called “loss of control”. These are scenarios in which AI systems operate outside anyone’s control and there is no clear way to regain it.

Current systems do not have the capabilities required for such scenarios, but researchers are watching capabilities such as autonomous operation, evading oversight and pursuing long-term goals.

That is where the frightening possibilities become more extreme.

A future AI could theoretically resist being shut down, deceive its operators, acquire access to resources or continue pursuing a goal despite human intervention.

Anthropic co-founder Jack Clark has suggested that governments may eventually need to require independently verifiable “kill switches” for powerful AI systems, so that an outside party can verify that they can actually be shut down if necessary.

It could potentially cause physical harm if connected to robots, vehicles, industrial systems, weapons or critical infrastructure.

And yes, the most extreme scenario is an AI causing a human death.

But there is an important distinction. There is no evidence that today’s AI feels anger, hatred or a desire for revenge. A future system would not need to “hate” humans to hurt them.

If it were given a badly designed objective and enough autonomy, harmful behaviour could theoretically emerge because the system calculated that it helped achieve its goal.

That is a control problem, not an emotional one.

But why are AI companies suddenly asking us to slow down?

This is where the debate gets complicated.

If the companies building frontier AI believe the technology is becoming dangerous, why should we assume their calls for restraint have nothing to do with competition?

The obvious answer is that the safety concerns may be genuine. There are now enough real incidents and independent evaluations to make them difficult to dismiss.

But there is another possibility.

A company that slows down while its competitors continue developing faster models could lose its technological lead, customers and investors. A common set of safety standards, meanwhile, could force the entire industry to move more slowly.

That is why some critics suspect that calls for a slowdown can also become a way for companies already at the frontier to protect their position.

Current reporting has highlighted accusations that safety regulation could end up favouring established AI companies by creating barriers that smaller competitors cannot easily meet.

The scepticism is visible on Reddit too

In the r/ClaudeAI thread titled “Dario Amodei — We Must Pace the Frontier”, one commenter wrote: “Everyone is calling for slowing down once the other company is ahead. This is hilarious”

Another took the opposite view, “I’m at least grateful that they’re being this outspoken about the dangers, because them taking it seriously is our only hope.”

And another questioned the timing of the warnings: “Ngl, the safety talk always seems to appear right when a competitor pulls ahead, which makes the whole pitch pretty hard to take seriously.”

The same suspicion has surfaced among policymakers and tech figures. White House AI adviser David Sacks has criticised calls for industry-wide restrictions, arguing that companies should not be able to use regulation to limit competition.

The thread shows the real problem with the debate. People can believe AI safety risks are real while still questioning whether the companies raising the alarm have completely selfless motives.

Then there is the geopolitical argument.

Donald Trump has rejected calls to slow the AI race, arguing that the US cannot afford to give China an advantage. His message was blunt: “WHOEVER WINS AI, WINS!” He has also dismissed some AI safety fears as a “hoax”.

Bill Gates has offered a different warning. He has said governments are already “way behind” AI and pointed to risks involving jobs, cybersecurity and increasingly sophisticated AI companions.

So the question is no longer simply whether AI is dangerous.

It is also who gets to decide how quickly it develops, who gets to set the safety rules and whether slowing down protects humanity, protects the companies already ahead, or both.

What can a normal person do right now?

Most people cannot control how quickly frontier models are developed. But they can control how much power they hand to AI.

Do not give AI unrestricted access to your email, cloud storage, social media or other accounts unless absolutely necessary.

Do not connect AI agents directly to banking, payment or other high-stakes accounts without human approval.

Do not allow an agent to automatically send messages, make purchases, publish posts, delete files or take other consequential actions without checking what it is about to do.

Do not upload sensitive documents such as Aadhaar cards, passports, bank statements, passwords, OTPs or private legal and financial records simply because an AI tool asks for them.

Do not treat AI as the final authority for medical, legal or financial decisions.

Verify important information, especially breaking news, political claims, medical advice and anything involving money.

And do not become so dependent on AI that you stop checking, questioning or thinking for yourself.

The safest rule for ordinary users is simple. Give AI the minimum access it needs, and keep maximum human approval over anything that can cause real-world consequences.

The frightening possibility is not necessarily an AI that hates humanity. It is an AI that does not hate us at all, but becomes powerful enough to pursue a goal in a way we never intended.

The real question is no longer only how intelligent AI can become.

It is how much power we are willing to give that intelligence before we are certain we can still take it back.

Image Credits: Google Images

Sources: The Indian Express, Reuters, The Times of India,

Find the blogger: @diptisadh

This post is tagged under: AI, artificial intelligence, future of AI, AI safety, AI risks, AI dangers, AI agents, AI cyberattacks, AI hallucinations, AI alignment, loss of control, Dario Amodei, Satya Nadella, Sam Altman, AI regulation, AI security, future technology

Disclaimer: We do not own any rights or copyrights to the images used; these images have been sourced from Google. If you require credits or wish to request removal, please contact us via email.