The people building AI are worried. What should the conservative response be?
Author: Matt Day
At what probability does a risk become something a government has a duty to take seriously?
If engineers told ministers there was a 10% change that a new nuclear reactor would destroy London, nobody would say that the other 90% looked encouraging.
Yet recently Anthropic’s Head of Alignment suggested there was around a 10% chance that advanced AI could wipe out humanity, and it caught Westminster’s attention. Anthropic CEO Dario Amodei then called explicitly for the industry to “pace the frontier” so that safety can keep up. President Trump, by contrast, has warned that slowing America risks handing the advantage to China.
In the AI world, this is referred to as “P(doom)” – the estimated probability that AI will lead to an existential catastrophe, such as human extinction or permanent human disempowerment.
Two notes of caution are necessary. Claims about future commercial capabilities should obviously be treated with caution. And P(doom) is not an exact science. It is difficult to attach precise percentages to an event that has never happened.
As a sci-fi nerd, I have perhaps watched too many films to understand instinctively why we would want to build something substantially more intelligent than ourselves. I struggle to think of a storyline where that has ended well.
But we do not need Terminators roaming the streets for advanced AI to cause serious harm.
Models operating increasingly independently of humans, circumventing safeguards or weaponised by bad actors could conduct sophisticated cyberattacks, assist the development of biological weapons or operate autonomous weapons.
Even if P(doom) were zero, these possibilities should demand the attention of government.
When reading the warnings from Anthropic, I was reminded of Jurassic Park. A sceptical Dr Ian Malcolm warns John Hammond that his scientists had become so preoccupied with whether they could create dinosaurs that they failed to consider whether they should.
The metaphor is imperfect. Unlike Hammond’s scientists, many of those developing frontier AI are thinking seriously about whether they should. Some of the most prominent warnings have come from the people building it. Anthropic employs researchers specifically to understand how advanced models might behave in dangerous or unexpected ways.
The problem is that these concerns have not slowed the race, at least so far.
That is because frontier AI increasingly resembles a prisoner’s dilemma. A company that unilaterally slows down may surrender the frontier to a competitor. A country that restrains its own development cannot know its geopolitical rivals will do the same.
So everyone has an incentive to keep running, even whilst some of the runners are warning that they do not know precisely where the finish line leads.
We have faced this problem before. Nuclear weapons eventually produced arms-control agreements between adversaries because both sides recognised that unconstrained competition could end badly for everyone.
There is an obvious temptation to imagine a similar settlement for advanced AI. We should certainly seek international agreement on the most dangerous capabilities, particularly among Britain and its allies.
But we should acknowledge it will be harder this time. AI is being developed primarily by private companies, driven by commercial as well as geopolitical competition. The infrastructure used to train frontier models also generates enormous economic value. And unlike a nuclear warhead, software can be copied.
There may therefore be no grand treaty that neatly removes the danger. Nor can Britain simply wait for Washington, Beijing or Silicon Valley to decide what level of risk we should tolerate.
That brings us to the Conservative response.
The stakes run both ways. I believe AI could be one of the greatest engines of prosperity and human progress since the Industrial Revolution. The challenge is to capture that extraordinary upside without being blind to risks at the frontier.
The wrong lesson would be that AI itself must be smothered in regulation. An SME using an AI assistant and a laboratory training a model capable of autonomous cyber operations plainly do not present the same risks. Allowing fears about superintelligence to justify sprawling regulation of ordinary AI would damage precisely the innovation and growth Britain desperately needs.
The principle should be simple: regulate capability, not the label “AI”.
Innovation should be the default and regulation the exception. But frontier AI may prove to be one of those exceptions.
Where a small number of companies are developing systems of unprecedented capability, government has a legitimate role that markets alone cannot fulfil – particularly when competitive pressures mean no individual company feels able to take its foot off the accelerator.
Amodei has proposed independent evaluators embedded inside frontier laboratories, alongside regulation as capabilities increase. It is a sensible starting point.
Britain should seek to make AISI the world’s leading independent frontier-AI evaluation institution and help build an international system of model evaluation around it.
The objective should not be another sprawling technology regulator policing every company using AI. It should be a highly technical capability focused narrowly on the frontier: independently testing the most powerful systems and establishing capability thresholds that trigger greater oversight.
Any regime must also avoid creating unnecessary barriers to entry. Regulation should therefore follow demonstrated capabilities rather than particular companies or business models, preserving competition as the market develops.
And there must ultimately be a point at which government is prepared to say: not yet.
If a future system demonstrates a credible ability to autonomously conduct devastating cyberattacks, assist the creation of biological weapons or defeat the safeguards designed to constrain it, it would be extraordinary to argue that deployment must continue simply in the name of innovation.
We should neither accept that geopolitical competition means there can be no brakes, nor pretend that safety policy can ignore geopolitical competition.
There is then a second question: whose hands are on the wheel?
Even perfectly safe AI would create a strategic vulnerability if our economy, public services and national security became too dependent on technologies controlled overseas. Today the frontier is overwhelmingly American, and we are fortunate that our closest ally is also the world’s leading technological power. But allies are essential; dependency is optional.
Sovereignty does not mean autarky or recreating the technology stack behind a national firewall. It means retaining sufficient domestic capability – compute, affordable energy, researchers, capital and technical expertise – for Britain to make meaningful choices for itself.
There is little sovereignty in adopting AI across Whitehall, the NHS or our armed forces if the British state cannot independently interrogate the underlying systems or another country or company could ultimately restrict our access to them.
This is why technology policy can no longer be treated as a niche interest in Westminster. For much of the past two decades, political debate about technology has tended to revolve around broadband speeds, social media and whichever app has most recently upset Parliament. Important issues, certainly, but usually left to specialist ministers, select committees and policy wonks.
Artificial intelligence changes that. AI is economic policy because it could transform productivity and the labour market. It is education policy because it changes what skills our children will need and even what it means to learn. It is energy policy because the infrastructure required to train and run these systems demands enormous amounts of power. It is defence and foreign policy because advanced models will increasingly shape cyber operations, intelligence and warfare. And it is constitutional policy because the more decisions we delegate to machines, the more important the question becomes of who is ultimately accountable for them.
An old IBM maxim put it well: “A computer can never be held accountable, therefore a computer must never make a management decision.”
The technology has changed beyond recognition, but the principle hasn’t.
Conservatism is not a belief that whatever is technologically possible should therefore be done. We are rightly sceptical of utopian schemes promising to redesign society from first principles. We value human judgement, institutions and accountability.
None of this requires pessimism about technology. AI is likely to help cure diseases, accelerate scientific discovery, create enormous new industries and deliver a more efficient state. Britain should seize those opportunities.
Perhaps the AI pessimists are wrong. I hope they are. Perhaps P(doom) is not 10% but 1%, or even 0.0001%.
Nobody knows. But hope is not a policy.
The Conservative response should be neither to stop technological progress nor blindly assume that every technological advance constitutes human progress. It should preserve human agency, insist on accountability, confront genuine dangers at the frontier and ensure Britain retains sovereignty over the technologies on which our future will depend.
P(doom) may be impossible to calculate, but the stakes are not.