2026-06-16原文2 min readUpdated: 2026-06-16

Simplifying the Modeling of Arbitrary Conditionals in Natural Language

Causal Transformers, while efficient for left-to-right decoding, struggle with arbitrary conditionals (e.g., text conditioned on both past and future tokens). The proposed AC-GPT introduces a simple modification to standard causal Transformers, enabling evaluation and sampling from arbitrary conditionals—including past, future, and mixed contexts—in a single forward pass. It preserves left-to-right ordering and next-token prediction, allowing fine-tuning of existing LLMs. Empirical results show it outperforms baselines on arbitrary conditional modeling without degrading standard performance.

SourcearXiv Computational LinguisticsAuthor: Yinhan Lu, Eric Elmoznino, L\'eo Gagnon, Sarthak Mittal, Tejas Kasetty, Guillaume Lajoie

[2606.14943] Simplifying the Modeling of Arbitrary Conditionals in Natural Language

[Submitted on 12 Jun 2026]

Title:Simplifying the Modeling of Arbitrary Conditionals in Natural Language

View a PDF of the paper titled Simplifying the Modeling of Arbitrary Conditionals in Natural Language, by Yinhan Lu and 5 other authors

View PDF HTML (experimental)

Abstract:Causal Transformers model sequences through an autoregressive factorization of the joint distribution, which enables efficient left-to-right decoding and conditional likelihood computation. However, they cannot tractably sample from or evaluate arbitrary conditionals -- e.g., a block of text conditioned on past and future tokens. Recent work aims to solve this problem through novel architectures, but they often lead to sub-optimal modeling of such conditionals and degraded generations. We propose Arbitrary Conditionals GPT (AC-GPT) which introduces a simple modification to standard causal Transformers to enable evaluating and sampling from arbitrary conditionals -- including past, future, and mixed contexts -- within a single forward pass. Unlike prior approaches, our method preserves the standard left-to-right ordering and next-token prediction objective essential for both strong performance and efficient training on natural language. Crucially, this compatibility allows existing LLMs to be fine-tuned for arbitrary conditioning. Our empirical results indicate that our method outperforms baselines on modeling arbitrary conditionals, without degrading standard left-to-right performance.

Subjects:

Computation and Language (cs.CL); Machine Learning (cs.LG)

ACM classes: I.2.6; I.2.7

Cite as: arXiv:2606.14943 [cs.CL]

(or arXiv:2606.14943v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2606.14943

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yinhan Lu [view email] [v1] Fri, 12 Jun 2026 20:39:33 UTC (839 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Simplifying the Modeling of Arbitrary Conditionals in Natural Language, by Yinhan Lu and 5 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-06

Change to browse by:

cs cs.LG

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)