2026-05-22 04:00 UTCOriginal source2 min readUpdated: 2026-06-30 13:03 UTC

Probabilistic Attribution For Large Language Models

This paper proposes a model-agnostic probabilistic token attribution measure that uses Bayes' rule to invert next-token log-probabilities, capturing the model's internal representation of token sequence distributions and revealing LLM behavior through entropy and attribution scores.

SourcearXiv Computational LinguisticsAuthor: Shilpika Shilpika, Carlo Graziani, Bethany Lusch, Venkatram Vishwanath, Michael E. Papka

[2605.21726] Probabilistic Attribution For Large Language Models

[Submitted on 20 May 2026]

Title:Probabilistic Attribution For Large Language Models

View a PDF of the paper titled Probabilistic Attribution For Large Language Models, by Shilpika Shilpika and 4 other authors

View PDF HTML (experimental)

Abstract:The generative nature of Large Language Models (LLMs) is reflected in the conditional probabilities they compute to sample each response token given the previous tokens. These probabilities encode the distributional structure that the model learns in training and exploits in inference. In this work, we use these probabilities to situate LLMs within the mathematical theory of stochastic processes. We use this framework to design a model-agnostic probabilistic token attribution measure, using Bayes rule to invert the next-token log-probabilities so as to capture the models internal representation of the distribution over token sequences. The representation is independent of the models computational structure. This representation yields the conditional probability of the response given the prompt, and of the response given the prompt with a token marginalized away. Our attribution score is the log of the ratio of these probabilities. We further compute the entropies of a single prompts token distributions, conditioned on the remaining context. The interplay between entropy and attribution score sheds light on LLM behavior. We evaluate 8 models across 7 prompts and investigate anomalies, token sensitivity, response stability, model stability, and training convergence, thereby improving interpretability and guiding users to focus on uncertain or unstable parts of the generation.

Comments: 29 pages, 13 figures

Subjects:

Computation and Language (cs.CL); Artificial Intelligence (cs.AI)

Cite as: arXiv:2605.21726 [cs.CL]

(or arXiv:2605.21726v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2605.21726

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Fnu Shilpika [view email] [v1] Wed, 20 May 2026 20:36:26 UTC (19,560 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Probabilistic Attribution For Large Language Models, by Shilpika Shilpika and 4 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-05

Change to browse by:

cs cs.AI

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)