2026-07-15 04:00 UTCOriginal source2 min readUpdated: 2026-07-15 08:23 UTC

TAKE: Trajectory-Aware Knowledge Estimation for Text Dataset Distillation

This paper proposes a text dataset distillation framework using Trajectory-Aware Knowledge Estimation (TAKE), which leverages influence functions and training trajectories to compute knowledge scores for each sample, and selects prototypes via optimal transport, compressing corpora to 0.1% of original size while preserving downstream task fidelity. Evaluated on text classification and natural language inference tasks.

SourcearXiv Computational LinguisticsAuthor: Tri-Nhan Vo, Dang Nguyen, Sunil Gupta

-->

[Submitted on 13 Jun 2026]

Title:TAKE: Trajectory-Aware Knowledge Estimation for Text Dataset Distillation

View a PDF of the paper titled TAKE: Trajectory-Aware Knowledge Estimation for Text Dataset Distillation, by Tri-Nhan Vo and 2 other authors

View PDF HTML (experimental)

Abstract:Large-scale text corpora have become a quiet bottleneck in modern NLP, not just in storage, but in the accumulated cost of training, fine-tuning, and continual learning. We propose a text dataset distillation framework that reduces corpora to as little as 0.1% of their original size while preserving downstream task fidelity. We approach distillation through the lens of influence functions, which quantify each sample's contribution to the downstream objective, a natural and principled basis for selection. We introduce Trajectory-Aware Knowledge Estimation (TAKE), which convolves the knowledge-based influence along the training trajectory into a single per-sample knowledge score, capturing informative samples. These scores serve as sample weights within a discrete Optimal Transport objective, guiding prototype selection from a synthetically generated candidate pool. We evaluate TAKE on downstream accuracy across text classification and natural language inference tasks at extreme compression (0.1% or 20 samples/class), showing that data efficiency is achievable without sacrificing task fidelity. The approach is theoretically grounded, with broader implications for coreset construction and data-centric AI. We release our source code at this https URL.

Comments: To be published in ECML-PKDD 2026

Subjects:

Computation and Language (cs.CL); Machine Learning (cs.LG)

Cite as: arXiv:2607.11898 [cs.CL]

(or arXiv:2607.11898v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2607.11898

arXiv-issued DOI via DataCite

Submission history

From: Tri-Nhan Vo [view email] [v1] Sat, 13 Jun 2026 05:57:08 UTC (31 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled TAKE: Trajectory-Aware Knowledge Estimation for Text Dataset Distillation, by Tri-Nhan Vo and 2 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.CL

new | recent | 2026-07

Change to browse by:

cs cs.LG

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)