Anatomy of an AI Kill Chain – Airwars X AI Now Institute

Scroll

AI warfare is much discussed, but often little understood. Rather than a single overarching programme, military AI today consists of overlapping machine learning algorithms rapidly automating tasks previously carried out by humans. In turn, human oversight and accountability are steadily disappearing.

This story portrays a fictional, and simplified, account of an AI-enabled attack.

To demystify military AI, we construct an abstract kill chain that reflects an AI targeting cycle. Rather than copying a formal kill chain framework used by a specific state, our schema instead illustrates the tactical activities and decisions involving the use of AI systems through six stages: 1 Sensor and Data , 2 Surveillance , 3 Intelligence and Identification , 4 Selection , 5 Strike , and 6 Post-strike Assessment.

Throughout, tech explainers dissect AI technologies, their limitations, and how their breakneck adoption risks deadly errors.

Tech Explainer

Click on me throughout to learn more

While the story is fictional, the technology is not. The AI described here is in use by advanced militaries today, including in Ukraine, Gaza and Iran.

It begins with data.

The world is more digitised than ever before. The phones we carry, the cars we drive, and the platforms we communicate on all produce troves of information.

In conflict zones rival militaries capture, log, and track all they can.

Human analysts cannot process this tsunami of data at the speed that militaries are demanding, driving increased reliance on AI. But this rush to make more decisions faster than ever before risks deadly mistakes.

In our story, two neighbouring nations are at war. An advanced military is monitoring a town in the other’s territory, from which the opposing force has launched attacks. Throughout they will be referred to as the Military and the Opposing Army, respectively.

Soldiers rely on a decision support system (DSS)—like the one shown on screen—to monitor the area, aid in military decision making, and coordinate operations.

Data collection and analysis are instrumental to the AI often used within a DSS. Yet faulty sensor outputs and noisy or mislabelled data can impact the reliability and accuracy of these systems.

The Military’s AI algorithms continually scan satellite imagery for anomalies, or changes that could indicate the Opposing Army’s mobilisation or weapons manufacturing. In the past month, the algorithms flagged twelve sites of unusual activity, which have been designated as potential targets.

At one, unmarked trucks were detected coming and going from a disused community center.

Anomaly detection systems can use convolutional neural networks (CNNs) and autoencoders to identify patterns or changes in visual or sensor data. When used by militaries, these algorithms may struggle to distinguish threats from ordinary environmental variation.

To monitor the community center, the Military has geofenced the area, meaning that anyone entering the site is tracked via their phone. Dozens of visits occur daily, each logged and monitored by machine learning algorithms.

An alert appears. A mobile phone SIM has entered the geofenced area. The analyst on shift begins to carry out further surveillance.

Shortly after, the phone sends a text. The message is automatically intercepted and translated.

The analyst wants to interrogate the translation—they can see the original text, but don’t speak the regional dialect. The duty translator is unavailable—they will have to trust the automated translation.

Automated translation tools that rely on large language models (LLMs) are now used extensively by militaries. Yet evidence shows that LLMs are unable to account for nuances in dialect or context—leading to grave translation errors.

The use of “launching soon” in the translated text triggers an emergency protocol.

The operators want to establish who the sim card is registered to. Based on telecoms registration data, the AI-enabled DSS matches the phone to a 30 year-old male called Christo.

The operator must now confirm that Christo poses an imminent threat.

The army relies on a social scoring system to comb through data passively collected from residents: call data records, social media connections, information from open platforms like Telegram, and data from facial recognition cameras. Operators use the algorithms to automatically assign residents a risk score from 1 to 10 judging their ‘likelihood’ of belonging to the Opposing Army.

Christo scores a seven.

The system classifies profiles with ratings of seven and above as members of the Opposing Army. Operators are required to review the supporting evidence to decide whether to confirm or reject the algorithmic rating.

Risk profiling systems rely on decision tree modeling to synthesise data from telecoms providers, social media, and other sources and detect combatants. Yet these models often overclassify civilians as combatants.

The DSS logs the pieces of evidence that inform Christo’s score.

As the threat is deemed imminent, the analyst only has three minutes to review the evidence and pass the decision on.

The time constraint makes it difficult to thoroughly vet each piece of supporting evidence and heightens the risk of automation bias—with the analyst deferring to AI-generated outputs without critically interrogating them.

Social media posts

One factor in Christo’s risk score is his connections and activities across social media. In the past three months, two posts he shared on Telegram are classified as supposedly ‘sympathetic’ to his nation’s Military, while two Facebook posts were classified as potentially “endorsing the actions of” the Opposing Army. A number of confirmed Opposing Army personnel liked the latter.

The analyst scrolls through automatically translated titles and captions—the posts appear to bedisplays of supportfor the Opposing Army. In the time allotted, it is nearly impossible for the analyst to verify the translations and the authenticity of the posts themselves. The underlying LLMs are prone to producing ‘hallucinations’.

Militaries use generative AI and LLMs to parse, translate, and classify social media content to build profiles of targets, monitor networks, and flag “high-risk” behaviour. However, LLMs can fabricate outputs and often fail to account for nuance and synthetic information.

Link analysis

Approximately 20% of Christo’s Instagram followers and 30% of his Facebook connections are members of the Opposing Army. In the past two weeks, his telecoms provider registered six outgoing calls to members of the Opposing Army.

Militaries use link analysis to establish a person’s social network and identify combatants. Yet the machine learning algorithms, like random forests, trained to detect members of opposing forces can produce false positives, drawing erroneous conclusions from their training data.

Location data

The army has hacked into CCTV cameras across the city. Christo has been tracked travelling near a major Opposing Army base three times in recent weeks.

Some computer vision systems identify and track individuals by following a person’s face, gait, or movement across video streams. These systems may misidentify people as combatants because they follow similar routes or resemble those classified as targets.

Given the limited time, the operator skips the remaining evidence and confirms what the social scoring system suggested—Christo seems to be an active member of the Opposing Army engaged in threatening activity and therefore a valid target. The system generates an intelligence report for unit command.

As the operation is high-urgency, command quickly signs off and dispatches a surveillance drone.

Resume feed

Live feed

The drone’s computer vision identifies a truck arriving at the disused community center.

Shortly after, two more trucks arrive. Four people begin loading the vehicles with what the object recognition software identifies as large boxes or crates. Given their size and gait, the software also classifies the figures as military-aged males.

Based on the information at hand, the analyst concludes the materiel is most likely related to recent attacks. A decision to strike is urgently recommended before the convoy leaves the site.

Target recognition systems use CNNs to interpret and identify objects in satellite, radar imagery, and drone feeds. Though widely deployed by militaries, CNNs struggle to identify objects in contexts, lighting, and angles that differ from training data.

The system identifies all those on site as adult males likely engaged in military activity—recommending them as combatants and detecting no civilians. The operator proceeds with initiating the attack.

Militaries often use algorithms that assess telecoms signals to estimate the number of civilians in an area before a strike. Yet the inherent inaccuracy of mobile geolocation, network outages, and communication scarcity make these markers too flawed to rely on.

The DSS proposes three potential weapons systems.

As the area around the building has seen jamming of radio-frequency and GPS signals, a series of one-way attack drones—launched from the nearby border—are recommended as the “top match”.

LLMs synthesise information about targets, a military’s available munitions, and acceptable civilian casualties to recommend munitions for strikes. Yet LLMs’ low accuracy rates and tendency to misinterpret data are bound to produce faulty outputs.

Based on the information at hand, the strike is approved by the commander—with the support of legal and political advisors.

The drones are dispatched with autonomous flight capabilities and object recognition. The operator programs the munitions to navigate to the building and strike once they detect the structure.

As they enter the signal-jammed area, the operator loses contact—the munitions enter autonomous guidance mode.

The first drone detects the building, and dives towards it.

One-way attack drones identify and lock onto targets, even in GPS-denied or electronically jammed environments through the use of recurrent neural networks (RNNs). However RNNs tend to only perform well in test scenarios, but not when deployed in unpredictable battlefield conditions.

Jul 18 14:04

Jul 18 14:39

Damage Assessment

Engaged Target

S001

Structure

Assessment

Confidence

Destroyed

95%

Collateral Damage

Confidence

None

95%

Killed

4

Injured

3

The analyst later receives an assessment report of the strike, produced by an AI-assisted ‘battle damage assessment’ (BDA) system. It concludes the strike hit the targeted building and at least three adults were killed.

Automated BDA systems collate information from satellite imagery, drone feed, and telecoms data after a strike to summarise the operation. Technical errors earlier in the kill-chain and misguided assumptions in targeting can lead to faulty analyses.

Feed — Latest

Evidence suggests...

strike_report.pdf

The report logs the operation as a success, with no further action required.

But quickly social media accounts suggest otherwise.

A day after the strike, a major international newspaper reports that nine people, including two women and a child inside the building, were killed.

Reporters discover Christo was a school teacher who helped establish an aid system for civilians displaced by war.

That day he had been due to launch a new food distribution point for displaced families. The boxes were filled with rice, pasta and canned vegetables, the people in the trucks volunteer aid workers.

After days of pressure, the military orders an internal investigation. It finds a series of cascading errors piled up through the AI-assisted kill chain.

From miscategorising the building as a weapons facility, to misidentifying Christo as a combatant, to failing to notice additional civilians at the site, a series of technical lapses resulted in the death of innocents. At no po

[truncated for AI cost control]