Using Fine-Tuned LLMs to Identify Indicators of Vulnerability in UK Police Incident Logs
This study explores adapting an LLM classification pipeline, originally developed on US police data, to estimate the prevalence of four vulnerability indicators (mental ill health, substance misuse, alcohol dependence, homelessness) in UK police incident narratives. Analyzing nearly 3,000 de-identified logs, the research finds that LLMs can provide meaningful prevalence estimates at scale, but naive deployment is unreliable, requiring substantial human input and statistical correction. The study underscores that LLM outputs cannot be treated as valid measurements without careful methodological support.
-->
[Submitted on 20 Jul 2026]
Title:Using Fine-Tuned LLMs to Identify Indicators of Vulnerability in UK Police Incident Logs
View a PDF of the paper titled Using Fine-Tuned LLMs to Identify Indicators of Vulnerability in UK Police Incident Logs, by Sam Relins and 1 other authors
View PDF HTML (experimental)
Abstract:Purpose: Understanding how much of routine policing involves vulnerable people could inform resourcing, training, and multi-agency response, yet administrative data provide limited insight. We explore whether an LLM-based classification pipeline, developed on open-source US police data, can be adapted to estimate the prevalence of four vulnerability indicators - mental ill health, substance misuse, alcohol dependence, and homelessness - in UK police incident narratives, and when outputs can be treated as defensible measurements.
Methods: We analyse nearly 3,000 de-identified incident logs from a UK police force, using a multi-stage pipeline combining repeated model inference, label aggregation, structured human review, and statistical correction. The pipeline runs on a locally hosted open-weight LLM, reflecting the secure environments police must work in.
Results: LLMs can produce meaningful, if imperfect, prevalence estimates at scale. Mental ill health indicators are present in approximately one in five incidents, with lower prevalence for other indicators. However, naive LLM deployment is unreliable: single-pass classifications are unstable, and aggregated outputs systematically over-assign indicators relative to human judgement. Correcting these biases required substantial human input and statistical adjustment, leaving considerable uncertainty.
Conclusions: While LLMs can extract information from unstructured police data, their outputs cannot be treated as valid measurements without careful methodological support. At the population level, defensible estimates are achievable but resource-intensive; at the individual level, errors remain frequent and unpredictable, limiting suitability for operational decisions. This study highlights both the potential and the constraints of LLM-based measurement in applied settings.
Comments: 24 pages, 4 figures. Preprint. Under review
Subjects:
Computation and Language (cs.CL); Computers and Society (cs.CY)
ACM classes: I.2.7; J.4
Cite as: arXiv:2607.18446 [cs.CL]
(or arXiv:2607.18446v1 [cs.CL] for this version)
https://doi.org/10.48550/arXiv.2607.18446
arXiv-issued DOI via DataCite (pending registration)
Submission history
From: Sam Relins [view email] [v1] Mon, 20 Jul 2026 18:58:33 UTC (1,818 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled Using Fine-Tuned LLMs to Identify Indicators of Vulnerability in UK Police Incident Logs, by Sam Relins and 1 other authors
View PDF
HTML (experimental)
TeX Source
view license
Current browse context:
cs.CL
new | recent | 2026-07
Change to browse by:
cs cs.CY
References & Citations
NASA ADS
Google Scholar
Semantic Scholar
Loading...
Data provided by:
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
Author
Venue
Institution
Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)