Skip to content
AI News HubLIVE
Original source2 min read

SharedSAE: One Feature Dictionary Across Language Models

Summary

The paper introduces SharedSAE, a single sparse autoencoder that replaces per-model SAEs. Across four 1B-scale models from different families and tokenizers, it retains 96.6% of dedicated SAEs' mean explained variance, shows cross-model latent correlations 1.8 times higher than post-hoc aligned separate SAEs, and supports efficient adaptation of new models once the dictionary is frozen.

SourcearXiv Machine LearningAuthor: Daniil Ognev, C\'elian Vasson, Lijie Hu, Kentaro Inui, Benjamin Heinzerling
SharedSAE: One Feature Dictionary Across Language Models
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 3 Sep 2026]

Title:SharedSAE: One Feature Dictionary Across Language Models

View a PDF of the paper titled SharedSAE: One Feature Dictionary Across Language Models, by Daniil Ognev and 4 other authors

View PDF HTML (experimental)

Abstract:Sparse autoencoders (SAEs) are widely used to interpret language model activations, but SAE training and latent labelling are typically repeated for every model. Here, we show that a single shared SAE can replace a collection of dedicated per-model SAEs. Our method, SharedSAE, combines a shared dictionary with model-specific encoder-decoder pairs. Unlike the closest prior method, which discards activation magnitudes and requires all models at inference, SharedSAE instead normalizes only selection scores, preserving magnitudes, and uses model dropout for single-model inference. We train SharedSAE on four 1B-scale base language models spanning distinct families and tokenizers. Despite sharing its latents across models, SharedSAE retains 96.6% of dedicated SAEs' mean explained variance; its latent activations exhibit cross-model correlations 1.8 times as high as separate SAEs aligned post-hoc, and its latent descriptions transfer across models. After the dictionary is frozen, new models can be efficiently adapted to it, achieving near-dedicated-SAE reconstruction quality while reusing the shared latent descriptions.

Subjects:

Machine Learning (cs.LG); Computation and Language (cs.CL)

Cite as: arXiv:2609.04344 [cs.LG]

(or arXiv:2609.04344v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2609.04344

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Daniil Ognev [view email] [v1] Thu, 3 Sep 2026 18:09:41 UTC (344 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled SharedSAE: One Feature Dictionary Across Language Models, by Daniil Ognev and 4 other authors

View PDF

HTML (experimental)

TeX Source

view license

Current browse context:

cs.LG

new | recent | 2026-09

Change to browse by:

cs cs.CL

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

IArxiv recommender toggle

IArxiv Recommender (What is IArxiv?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • SharedSAE pairs a shared latent dictionary with model-specific encoder-decoder pairs, so one dictionary serves multiple models.
  • Unlike the closest prior method, it normalizes only selection scores, preserves activation magnitudes, and uses model dropout for single-model inference.
  • On four 1B-scale models it retains 96.6% of dedicated SAEs' mean explained variance and yields 1.8x higher cross-model correlations than post-hoc aligned SAEs.
  • After freezing the dictionary, new models can be adapted efficiently to near-dedicated-SAE reconstruction quality while reusing shared latent descriptions.

Highlights and analysis are generated automatically and may contain errors. Check the original source.