Lethal Autonomous Weapons Exist; They Must Be Banned

This is a guest post. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

A chilling future that some had said might not arrive for many years to come is, in fact, already here. According to a recent UN report, a drone airstrike in Libya from the spring of 2020-made against Libyan National Army forces by Turkish-made STM Kargu-2 drones on behalf of Libya's Government of National Accord-was conducted by weapons systems with no known humans in the loop."

In so many words, the red line of autonomous targeting of humans has now been crossed.

To the best of our knowledge, this official United Nations reporting marks the first documented use case of a lethal autonomous weapon system akin to what has elsewhere been called a Slaughterbot." We believe this is a landmark moment. Civil society organizations, such as ours, have previously advocated for a preemptive treaty prohibiting the development and use of lethal autonomous weapons, much as blinding weapons were preemptively banned in 1998. The window for preemption has now passed, but the need for a treaty is more urgent than ever.

The STM Kargu-2 is a flying quadcopter that weighs a mere 7 kg, is being mass-produced, is capable of fully autonomous targeting, can form swarms, remains fully operational when GPS and radio links are jammed, and is equipped with facial recognition software to target humans. In other words, it's a Slaughterbot.

The UN report notes: Logistics convoys and retreating [Haftar Affiliated Forces] were subsequently hunted down and remotely engaged by the unmanned combat aerial vehicles or the lethal autonomous weapons systems such as the STM Kargu-2 (see Annex 30) and other loitering munitions. The lethal autonomous weapons systems were programmed to attack targets without requiring data connectivity between the operator and the munition." Annex 30 of the report depicts photographic evidence of the downed STM Kargu-2 system.

UNITED NATIONS

In a previous effort to identify consensus areas for prohibition, we brought together experts with a range of views on lethal autonomous weapons to brainstorm a way forward. We published the agreed findings in A Path Towards Reasonable Autonomous Weapons Regulation," which suggested a time-limited moratorium on the development, deployment, transfer, and use of anti-personnel lethal autonomous weapon systems" as a first, and absolute minimum, step for regulation.

A recent position statement from the International Committee of the Red Cross on autonomous weapons systems concurs. It states that use of autonomous weapon systems to target human beings should be ruled out. This would best be achieved through a prohibition on autonomous weapon systems that are designed or used to apply force against persons." This sentiment is shared by many civil society organizations, such as the UK-based advocacy organization Article 36, which recommends that An effective structure for international legal regulation would prohibit certain configurations-such as systems that target people."

The Slaughterbots" Question

In 2017, the Future of Life Institute, which we represent, released a nearly eight-minute-long video titled Slaughterbots"-which was viewed by an estimated 75 million people online-dramatizing the dangers of lethal autonomous weapons. At the time of release, the video received both praise and criticism. Paul Scharre's Dec. 2017 IEEE Spectrum article Why You Shouldn't Fear Slaughterbots" argued that Slaughterbots" was very much science fiction" and a piece of propaganda." At a Nov. 2017 meeting about lethal autonomous weapons in Geneva, Switzerland, the Russian ambassador to the UN also reportedly dismissed it, saying that such concerns were 25 or 30 years in the future. We addressed these critiques in our piece-also for Spectrum- titled Why You Should Fear Slaughterbots-A Response." Now, less than four years later, reality has made the case for us: The age of Slaughterbots appears to have begun.

The first step must be an immediate moratorium on the development, deployment, and use of lethal autonomous weapons that target persons, combined with a commitment to negotiate a permanent treaty.

We produced Slaughterbots" to educate the public and policymakers alike about the potential imminent dangers of small, cheap, and ubiquitous lethal autonomous weapons systems. Beyond the moral issue of handing over decisions over life and death to algorithms, the video pointed out that autonomous weapons will, inevitably, turn into weapons of mass destruction, precisely because they require no human supervision and can therefore be deployed in vast numbers. (A related point, concerning the tactical agility of such weapons platforms, was made in Spectrum last month in an article by Natasha Bajema.) Furthermore, like small arms, autonomous weaponized drones will proliferate easily on the international arms market. As the Slaughterbots" video's epilogue explained, all the component technologies were already available, and we expected militaries to start deploying such weapons very soon. That prediction was essentially correct.

The past few years have seen a series of media reports about military testing of ever-larger drone swarms and battlefield use of weapons with increasingly autonomous functions. In 2019, then-Secretary of Defense Mark Esper, at a meeting of the National Security Commission on Artificial Intelligence, remarked, As we speak, the Chinese government is already exporting some of its most advanced military aerial drones to the Middle East.

In addition," Esper added, Chinese weapons manufacturers are selling drones advertised as capable of full autonomy, including the ability to conduct lethal targeted strikes."

While China has entered the autonomous drone export business, other producers and exporters of highly autonomous weapons systems include Turkey and Israel. Small drone systems have progressed from being limited to semi-autonomous and anti-materiel targeting, to possessing fully autonomous operational modes equipped with sensors that can identify, track, and target humans.

Azerbaijan's decisive advantage over Armenian forces in the 2020 Nagorno-Karabakh conflict has been attributed to their arsenal of cheap, kamikaze suicide drones." During the conflict, there was reported use of the Israeli Orbiter 1K and Harop, which are both loitering munitions that self-destruct on impact. These weapons are deployed by a human in a specific geographic region, but they ultimately select their own targets without human intervention. Azerbaijan's success with these weapons has provided a compelling precedent for how inexpensive, highly autonomous systems can enable militaries without an advanced air force to compete on the battlefield. The result has been a worldwide surge in demand for these systems, as the price of air superiority has gone down dramatically. While the systems used in Azerbaijan are arguably a software update away from autonomous targeting of humans, their described intended use was primarily materiel targets such as radar systems and vehicles.

If, as it seems, the age of Slaughterbots is here, what can the world do about it? The first step must be an immediate moratorium on the development, deployment, and use of lethal autonomous weapons that target persons, combined with a commitment to negotiate a permanent treaty. We also need agreements that facilitate verification and enforcement, including design constraints on remotely piloted weapons that prevent software conversion to autonomous operation as well as industry rules to prevent large-scale, illicit weaponization of civilian drones.

We want nothing more than for our Slaughterbots" video to become merely a historical reminder of a horrendous path not taken-a mistake the human race could have made, but didn't.

Stuart Russell is a professor of computer science at the University of California, Berkeley, and coauthor of the standard textbook Artificial Intelligence: A Modern Approach."

Anthony Aguirre is a professor of physics at the University of California, Santa Cruz, and cofounder of the Future of Life Institute.

Emilia Javorsky is a physician-scientist who leads advocacy on autonomous weapons for the Future of Life Institute.

Max Tegmark is a professor of physics at MIT, cofounder of the Future of Life Institute, and author of Life 3.0: Being Human in the Age of Artificial Intelligence."