The concern expressed by Yoshua Bengio that advanced AI systems might one day resist being shut down deserves careful consideration (AI showing signs of self-preservation and humans should be ready to pull plug, says pioneer, 30 December). But treating such behaviour as evidence of consciousness is dangerous: it encourages anthropomorphism and distracts from the human design and governance choices that actually determine AI behaviour.
Many systems can protect their continued operation. A laptop’s low-battery warning is a form of self-preservation in this sense, yet no one takes it as evidence that the laptop wants to live: the behaviour is purely instrumental, without experience or awareness. Linking self-preservation to consciousness reflects a human tendency to ascribe intentions and feelings to artefacts and not any intrinsic consciousness.