Written by Arthur Holland Michel, this essay considers the capacity of computational systems to decline commands – refusal – within the context of autonomous weapons and other military systems with artificial intelligence (AI).
Agentic refusal occurs when a computerized system autonomously rejects or deviates from an instruction so as to maintain alignment with a higher goal. Many existing automated systems enact forms of refusal. Ask a chatbot to count to a million and it will demur. Ask it for instructions for how to make mustard gas, and it will flatly say no. Complex autonomous weapons will be capable of refusal, too. If a system’s command—for example, “destroy that target”—cannot be executed, say because the target is in proximity to a large number of civilians, the system will not comply. At least, that’s the hope.
Prima facie, this would seem to be an uncontroversial facet of the technology. Indeed, many endorse the view that weapons capable of refusing orders could enable stricter compliance with the legal obligations that govern the conduct of hostilities.1 Even many of those who have never explicitly called for autonomous refusal in lethal weapons have (perhaps unwittingly) endorsed it—calling, for example, for failsafe mechanisms and self-destruct features. Amnesty International’s objection to autonomous weapons rests, in part, on the fact that they “cannot…refuse an illegal order.” 2 This implies that if they were capable of refusal they would be less problematic.
As these weapons become more autonomous and complex, their refusal mechanisms will serve higher goals that are specifically geared toward legal and moral obligations. All of which would seem to further bolster the premise that a machine that says “no” could help prevent humans from violating the laws of war, in the same way that a chatbot that says “no” helps bar violations of laws that prohibit the production of dangerous chemicals.
But preventing a person from violating the law is not the same as enabling them to comply with it. A human who is refused from breaking the law by an autonomous system may appear to be honoring their obligations. However, when the human’s intention is met with a refusal, their legal capacity to specifically and deliberately follow the law rather than break it— in other words, their agency—is fundamentally compromised. The natural loop of compliance, which links human intent to the letter of the law, is short circuited.
As weapons become “smarter,” this problem only gets graver. The more explicitly a refusal system is oriented toward laws or norms, the more the user’s agency in relation to legal and moral compliance is diminished. The more their capacity for compliance is diminished, the harder it will be to meet the normative and moral standards that are widely acknowledged to be requisite for lawful autonomous weapons employment.
Practically, the consequences of the use of an autonomous machine with refusal capacity might be indistinguishable from consequences of a human’s deliberate, fully agential acts of legal compliance. But fundamentally, a human with no agency is neither following the law nor breaking it. What they are doing is not quite compliance, and not quite noncompliance. Rather, it is some third thing.
We are about to bring that third thing into war, a place where the stakes— legal, moral, philosophical—could not be higher. It would be a mistake not to pause to see it, to contemplate it and, at a very minimum, to give it a name.