Skip to content
AI News HubLIVE
Original source2 min read

baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity

Summary

A new arXiv preprint introduces baseVultra, a compact recurrent cell that shifts dense recurrent computation away from gate transforms and toward candidate generation. It concentrates the dense budget in a single H×H candidate-generation matrix and replaces recurrent gate matrices with O(H) diagonal interactions. On TinyShakespeare character-level modeling it reaches 1.594 validation loss versus 1.604 for a parameter-matched LSTM and 1.616 for a GRU, with dense recurrent matrix-vector work at about 54% of GRU; on the larger text8 benchmark it is merely competitive with full-gate cells, and its formal-language gains appear only under constrained hidden-state capacity and fade as hidden size grows.

SourcearXiv Computational LinguisticsAuthor: Linke Zhang
baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

[Submitted on 23 Aug 2026]

Title:baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity

View a PDF of the paper titled baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity, by Linke Zhang

View PDF HTML (experimental)

Abstract:Nonlinear recurrent networks remain useful for streaming sequence modeling, yet gated cell design is still dominated by LSTM and GRU. We study whether recurrent computation can be reallocated away from gate transforms and toward candidate generation. We propose baseVultra, a compact recurrent cell that concentrates the dense recurrent budget on a single H x H candidate-generation matrix, while replacing recurrent gate matrices with O(H) diagonal interactions. The candidate update combines this dense branch with two low-rank auxiliary branches, an EMA cache, and learnable output blending; the gates additionally receive lightweight second-order access to h_{t-2} through diagonal terms. On TinyShakespeare character-level language modeling, baseVultra achieves a validation loss of 1.594 in the compact one-layer 200K setting, compared with 1.604 for a parameter-matched LSTM and 1.616 for a parameter-matched GRU, while its dense recurrent matrix-vector computation is about 54% of GRU under matched hidden size for this configuration. On text8, a roughly 45x larger benchmark, baseVultra is competitive with full-gate cells (1.854 bpc vs. 1.849 for GRU and 1.837 for LSTM). On formal-language probes, the largest gains appear under constrained hidden-state capacity (0.991 vs. 0.761 GRU and 0.691 LSTM OOD2 accuracy on Counter at H=16) and vanish as hidden size grows. These results characterize where and when candidate-focused recurrent capacity allocation helps: the benefit is concentrated in compact regimes, while full-gate cells remain competitive at scale.

Comments: Preprint. 14 pages, 3 figures. Revised based on peer review feedback from WAIC Academic 2026 (1 weak accept, 2 borderline)

Subjects:

Computation and Language (cs.CL); Machine Learning (cs.LG)

Cite as: arXiv:2610.30154 [cs.CL]

(or arXiv:2610.30154v1 [cs.CL] for this version)

https://doi.org/10.48550/arXiv.2610.30154

arXiv-issued DOI via DataCite

Submission history

From: Linke Zhang [view email] [v1] Sun, 23 Aug 2026 14:31:35 UTC (46 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled baseVultra: Compact Nonlinear Recurrence with Reallocated Capacity, by Linke Zhang

View PDF

HTML (experimental)

TeX Source

view license

Additional Features

Audio Summary

Current browse context:

cs.CL

new | recent | 2026-10

Change to browse by:

cs cs.LG

References & Citations

SciX

Google Scholar

Semantic Scholar

Loading...

Data provided by:

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Related Papers

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

Author

Venue

Institution

Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • Reallocates recurrent capacity from gate transforms to a single dense H×H candidate-generation matrix, with gates reduced to O(H) diagonal interactions
  • Achieves 1.594 validation loss on TinyShakespeare in a one-layer 200K-parameter setting, beating matched LSTM and GRU while using about 54% of GRU's dense recurrent matrix-vector computation
  • On text8 it records 1.854 bpc, essentially on par with GRU (1.849) and LSTM (1.837)
  • Formal-language probe gains are largest at small hidden sizes such as H=16 and disappear as hidden capacity increases

Highlights and analysis are generated automatically and may contain errors. Check the original source.

More recent stories on this topic · Research