{"id":2624,"date":"2025-03-11T10:30:53","date_gmt":"2025-03-11T10:30:53","guid":{"rendered":"https:\/\/blogs.bath.ac.uk\/iprblog\/?p=2624"},"modified":"2025-03-10T21:45:50","modified_gmt":"2025-03-10T21:45:50","slug":"how-the-risk-of-ai-weapons-could-spiral-out-of-control","status":"publish","type":"post","link":"https:\/\/blogs.bath.ac.uk\/iprblog\/2025\/03\/11\/how-the-risk-of-ai-weapons-could-spiral-out-of-control\/","title":{"rendered":"How the risk of AI weapons could spiral out of control"},"content":{"rendered":"

\"\"<\/a>AI\u2019s rapid advancement brings both promise and peril. Akhil Bhardwaj explores how flawed AI reasoning\u2014like mistaking rulers for cancer markers\u2014raises ethical concerns, especially in weapon development. He examines the risks of overconfidence, lack of accountability, and unintended consequences, arguing that a global ban on AI weapons, like past CFC regulations, could prevent catastrophic outcomes.<\/em><\/p>\n

Akhil Bhardwaj<\/a> is an Associate Professor (Strategy and Organisation) in the School of Management, University of Bath.<\/a> This article is republished from The Conversation<\/a> under a Creative Commons license. Read the original article here.<\/a><\/em><\/p>\n

\n

 <\/p>\n

Sometimes AI isn\u2019t as clever as we think it is. Researchers training an algorithm to identify skin cancer thought they had succeeded until they discovered<\/a> that it was using the presence of a ruler to help it make predictions. Specifically, their data set consisted of images where a pathologist had put in a ruler to measure the size of malignant lesions.<\/p>\n

It extended this logic for predicting malignancies to all images beyond the data set, consequently identifying benign tissue as malignant if a ruler was in the image.<\/p>\n

The problem here is not that the AI algorithm made a mistake. Rather, the concern stems from how the AI \u201cthinks\u201d. No human pathologist would arrive at this conclusion.<\/p>\n

These cases of flawed \u201creasoning\u201d abound \u2013 from HR algorithms that prefer<\/a> to hire men because the data set is skewed in their favour to propagating<\/a> racial disparities in medical treatment. Now that they know about these problems, researchers are scrambling to address them.<\/p>\n

Recently, Google decided to end its longstanding ban<\/a> on developing AI weapons. This potentially encompasses the use of AI to develop arms, as well as AI in surveillance and weapons that could be deployed autonomously on the battlefield. The decision came days after parent company Alphabet experienced a 6% drop<\/a> in its share price.<\/p>\n

This is not Google\u2019s first foray into murky waters. It worked with the US Department of Defense on the use of its AI technology for Project Maven<\/a>, which involved object recognition for drones.<\/p>\n

When news of this contract became public in 2018, it sparked backlash from employees who did not want the technology they developed to be used in wars. Ultimately, Google did not renew its contract, which was picked up by rival Palantir<\/a> instead.<\/p>\n

The speed with which Google\u2019s contract was renewed by a competitor led some to note the inevitability<\/a> of these developments, and that it was perhaps better to be on the inside to shape the future.<\/p>\n

Such arguments, of course, presume that firms and researchers will be able to shape<\/a> the future as they want to. But previous research has shown that this assumption is flawed for at least three reasons.<\/p>\n

The confidence trap<\/h2>\n

First, human beings are susceptible to falling into what is known as a \u201cconfidence trap\u201d<\/a>. I have researched this phenomenon, whereby people assume that since previous risk-taking paid off, taking more risks in the future is warranted.<\/p>\n

In the context of AI, this may mean incrementally extending the use of an algorithm beyond its training data set. For example, a driverless car may be used on a route has not been covered in its training.<\/p>\n

This can throw up problems. There is now an abundance of data that driverless car AI can draw on, and yet mistakes still occur<\/a>. Accidents like the Tesla car that drove into a \u00a32.75 million jet<\/a> when summoned by its owner in an unfamiliar setting, can still happen. For AI weapons, there isn\u2019t even much data to begin with.<\/p>\n

Second, AI can reason in ways that are alien to human understanding. This has led to the paperclip<\/a> thought experiment, where AI is asked to produce as many paper clips as possible. It does so while consuming all resources \u2013 including those necessary for human survival.<\/p>\n

Of course, this seems trivial. After all, humans can lay out ethical guidelines. But the problem lies in being unable to anticipate how an AI algorithm might achieve what humans have asked of it and thus losing control. This might even include \u201ccheating.\u201d In a recent experiment, AI cheated to win chess games<\/a> by modifying system files denoting positions of chess pieces, in effect enabling it to make illegal moves.<\/p>\n

But society may be willing to accept mistakes, as with civilian casualties<\/a> caused by drone strikes directed by humans. This tendency is something known as the \u201cbanality of extremes\u201d \u2013 humans normalise even the more extreme instances of evil<\/a> as a cognitive mechanism to cope. The \u201calienness\u201d of AI reasoning may simply provide more cover for doing so.<\/p>\n

Third, firms like Google that are associated with developing these weapons might be too big to fail<\/a>. As a consequence, even when there are clear instances of AI going wrong, they are unlikely to be held responsible. This lack of accountability creates a hazard<\/a> as it disincentivises learning and corrective actions.<\/p>\n

The \u201ccosying up\u201d<\/a> of tech executives with US president Donald Trump only exacerbates the problem as it further dilutes accountability.<\/p>\n

Rather than joining the race towards the development of AI weaponry, an alternative approach would be to work on a comprehensive ban on it\u2019s development and use.<\/p>\n

Although this might seem unachievable, consider the threat of the hole in the ozone layer. This brought rapid unified action in the form of banning the CFCs<\/a> that caused it. In fact, it took only two years for governments to agree on a global ban<\/a> on the chemicals. This stands as a testament to what can be achieved in the face of a clear, immediate and well-recognised threat.<\/p>\n

Unlike climate change \u2013 which despite overwhelming evidence continues to have detractors \u2013 recognition of the threat of AI weapons is nearly universal<\/a> and includes leading technology entrepreneurs and scientists<\/a>.<\/p>\n

In fact, banning the use and development of certain types of weapons has precedent \u2013 countries have after all done the same for biological weapons<\/a>. The problem lies in no country wanting another to have it before they do, and no business wanting to lose out in the process.<\/p>\n

In this sense, choosing to weaponise AI or disallowing it will mirror the wishes of humanity. The hope is that the better side of human nature will prevail.\"The<\/p>\n

All articles posted on this blog give the views of the author(s), and not the position of the IPR, nor of the University of Bath.<\/em><\/p>\n

 <\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

AI\u2019s rapid advancement brings both promise and peril. Akhil Bhardwaj explores how flawed AI reasoning\u2014like mistaking rulers for cancer markers\u2014raises ethical concerns, especially in weapon development. He examines the risks of overconfidence, lack of accountability, and unintended consequences, arguing that...<\/p>\n","protected":false},"author":1742,"featured_media":2625,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[150,128,109,143,133,126,127],"tags":[],"class_list":["post-2624","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-the-labour-market","category-data-politics-and-policy","category-emerging-technologies","category-global-politics","category-science-and-research-policy","category-security-and-defence"],"acf":[],"jetpack_featured_media_url":"https:\/\/blogs.bath.ac.uk\/iprblog\/wp-content\/uploads\/sites\/115\/2025\/03\/Blog-Images-136.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blogs.bath.ac.uk\/iprblog\/wp-json\/wp\/v2\/posts\/2624","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.bath.ac.uk\/iprblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.bath.ac.uk\/iprblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.bath.ac.uk\/iprblog\/wp-json\/wp\/v2\/users\/1742"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.bath.ac.uk\/iprblog\/wp-json\/wp\/v2\/comments?post=2624"}],"version-history":[{"count":0,"href":"https:\/\/blogs.bath.ac.uk\/iprblog\/wp-json\/wp\/v2\/posts\/2624\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.bath.ac.uk\/iprblog\/wp-json\/wp\/v2\/media\/2625"}],"wp:attachment":[{"href":"https:\/\/blogs.bath.ac.uk\/iprblog\/wp-json\/wp\/v2\/media?parent=2624"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.bath.ac.uk\/iprblog\/wp-json\/wp\/v2\/categories?post=2624"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.bath.ac.uk\/iprblog\/wp-json\/wp\/v2\/tags?post=2624"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}