2026-04-15 00:00 UTCOriginal source12 min readUpdated: 2026-06-27 00:25 UTC

Parcae: Doing more with fewer parameters using stable looped models

Parcae is a stable looped language model that matches the quality of a Transformer twice its size — a 770M model reaching 1.3B-level performance. We introduce the first scaling laws for looping and show that increasing recurrence, not just data, is a compute-efficient path to better performance.

SourceTogether AI Blog

⚡️ FlashAttention-4: up to 1.3× faster than cuDNN on NVIDIA Blackwell →

Introducing Together AI's new look →

🔎 ATLAS: runtime-learning accelerators delivering up to 4x faster LLM inference →

⚡ Together GPU Clusters: self-service NVIDIA GPUs, now generally available →

📦 Batch Inference API: Process billions of tokens at 50% lower cost for most models →

🪛 Fine-Tuning Platform Upgrades: Larger Models, Longer Contexts →

Inference

Serverless Inference

High-performance inference as APIs

Batch Inference

Inference for batch workloads

Dedicated Model Inference

Inference on custom hardware

Dedicated Container Inference

Inference for custom models

MiniMax M2.5

Nano Banana Pro

Qwen3.5-397B

GLM-5

kimi k2.5

gpt-oss-120B

Model library

Explore the top open-source models

Compute

Accelerated Compute

GPU Clusters

Reliable GPU clusters at scale

AI Factory

Custom infrastructure at frontier scale

Developer Environments

Sandbox

Build development environments for AI

Storage

Managed Storage

Store model weights & data securely

GB300

GB200

B200

H200

H100

Model Shaping

Fine-Tuning

Shape models with your data

Evaluations

Measure model quality

DeepSeek V3.1

GLM 5 FP4

Qwen3-VL 32B

gpt-oss-120b

kimi k2.5

Llama 4 Maverick

Model library

Fine-tune top open-source models

Research

Systems research for production AI

Research blog

All our research publications

Featured publications

FlashAttention

ATLAS

Kernel Collection

ThunderKittens

DSGym

Show all

Developers

Documentation

Technical docs for Together AI

Demos

Our open-source demo apps

Cookbooks

Practical implementation guides

Voice Agents

Build voice agents for production

Model Library

Playground

Together Chat

Which LLM to use

Company

Resources

Customer stories

Testimonials from AI Natives

Startup accelerator

Build and scale your startup

Customer support

Find answers to your questions

Blog

Our latest news & blog posts

Events

Explore our events calendar

Company

About

Get to know us

Careers

Join our mission

Pricing

Serverless Inference

High-performance inference as APIs

Batch Inference

Inference for batch workloads

Dedicated Model Inference

Inference on custom hardware

Dedicated Container Inference

Inference for custom models

MiniMax M2.5

Nano Banana Pro

Qwen3.5-397B

GLM-5

kimi k2.5

gpt-oss-120B

Model library

Explore the top open-source models

Accelerated Compute

GPU Clusters

Reliable GPU clusters at scale

AI Factory

Custom infrastructure at frontier scale

Developer Environments

Sandbox

Build development environments for AI

Storage

Managed Storage

Store model weights & data securely

GB300

GB200

B200

H200

H100

Fine-Tuning

Shape models with your data

Evaluations

Measure model quality

DeepSeek V3.1

GLM 5 FP4

Qwen3-VL 32B

gpt-oss-120b

kimi k2.5

Llama 4 Maverick

Model library

Fine-tune top open-source models

Research

Systems research for production AI

Research blog

All our research publications

Featured publications

FlashAttention

ATLAS

Kernel Collection

ThunderKittens

DSGym

Show all

Documentation

Technical docs for Together AI

Demos

Our open-source demo apps

Cookbooks

Practical implementation guides

Voice Agents

Build voice agents for production

Model Library

Playground

Together Chat

Which LLM to use

Resources

Customer stories

Testimonials from AI Natives

Startup accelerator

Build and scale your startup

Customer support

Find answers to your questions

Blog

Our latest news & blog posts

Events

Explore our events calendar

Company

About

Get to know us

Careers

Join our mission

Contact sales

All blog posts

Research

Published 4/15/2026

Parcae: Doing more with fewer parameters using stable looped models

Authors

Hayden Prairie, Zachary Novack, Taylor Berg-Kirkpatrick, Dan Fu

Table of contents

40+ Models Chosen for Production...40+ Models Chosen for Production...40+ Models Chosen for Production...

Links in this article

Paper

Code

Hugging Face

Summary

We present Parcae, one of the first stable architectures for looped language models, achieving the quality of a Transformer twice the size with clean, predictable training. Parcae creates a new medium to scale quality by increasing recurrence rather than purely scaling data, opening up an efficient frontier for training memory-constrained on-device models.

Getting the most out of your parameters

Traditional scaling laws tell us that to achieve the best performance, we need to scale FLOPs, often with more parameters or data. But as models move to the edge and inference costs skyrocket, we wonder: Can we scale quality without inflating memory footprint?

To that end, we’ve been exploring looped architectures, models that increase compute by passing activations through the same layers multiple times. While promising, these models have been unstable to train. We tackle this issue directly and introduce Parcae, a stable looped architecture that:

Is better than prior looped models: Parcae achieves up to 6.3% lower validation perplexity than previous large-scale looped recipes.

Punches above its weight: Our 770M Parcae matches the quality of a 1.3B parameter transformer trained on the same data, achieving the same performance with roughly half the parameters.

Scales Predictably: We establish the first scaling laws for looping, finding that compute-optimal training requires increasing looping and data in tandem.

Looped models are cool, but hard to train in practice

As models move to the edge and inference deployments take on larger portions of compute, there is an increasing interest in scaling model quality without increasing parameters. One mechanism we have been excited about is layer looping, where initial works have trained looped models that match the quality of larger fixed-depth architectures.

To turn a vanilla Transformer into a looped model, we follow prior work and partition its layers into three functional blocks: a prelude ($\mathcal{P}$), a recurrent ($\mathcal{R}$), and a coda ($\mathcal{C}$). The forward pass works in three stages:

Embedding: The prelude transforms the input into a latent state $e$.

Recurrence: The recurrent block iteratively updates a hidden state $h_t$ for $T$ loops. To maintain the input’s influence, $e$ is injected into each loop, typically via addition [1] ($h_{t+1} = \mathcal{R}(h_t + e)$) or concatenation with projection [2] ($h_{t+1} = \mathcal{R}(W[h_t; e])$).

Output: The coda processes the final $h_T$ to generate the model’s output.

Unfortunately, looped models are a headache to train [2][3][4]. We personally found them to suffer from residual state explosion and loss spikes. What makes looped models even trickier is that the recurrent block is composed of several vanilla Transformer blocks, making it difficult to reason about the source of instability.

Understanding the instability of looping

While instability is a fickle foe, we observed that a simple linear framework captured a significant source of instability. Specifically, we recast looping as a nonlinear time variant dynamical system over the residual, whose update rule is:

$$h_{t+1} = \overline{A} h_t + \overline{B} e + \overline{\mathcal{R}}(h_t, e)$$

where $\overline{A}, \overline{B}$ perform injection and $\overline{\mathcal{R}}$ is the contribution of the Transformer blocks to the residual stream. For the subquadratic sequence mixing fanatics out there, observe that if we ignore the nonlinear term $\overline{\mathcal{R}}$, the resulting system is a discrete linear time-invariant (LTI) dynamical system over the residual state, across model depth.

What's cool is that for discrete LTI systems, their stability and convergence are determined by the eigenvalues of $\overline{A}$. Specifically, stability is categorized using the spectral norm $\rho(\overline{A})$ (i.e., the absolute largest eigenvalue of $\overline{A}$), with stable systems (convergent) being $\rho(\overline{A})[2], a prior looped model, we tested against parameter- and data-matched RDMs, observing that Parcae reduces validation perplexity by up to 6.3%.

Params & Model Val. PPL (↓)

100M Scale

└ RDM 14.23

└ Parcae 13.59

350M Scale

└ RDM 10.76

└ Parcae 10.09

When retrofitting a very strong Transformer baseline into an RDM, without any hyperparameter tuning, we found Parcae to be robust over RDMs (which just flat-out diverged).

Params & Model Val. Loss (↓) Core (↑) Core-Extended (↑)

RDM Divergent Divergent Divergent

+ Parcae Constrained A 2.97 13.2 ± 0.2 9.1 ± 0.5

+ All Parcae Tricks 2.95 14.0 ± 0.2 9.7 ± 0.3

We also took Parcae and used it as a drop-in replacement for a standard fixed-depth Transformer. Using a nanochat-inspired setup, we train a series of language models on FineWeb-Edu, up to 1.3B parameters. We found that Parcae outperformed all parameter- and data-matched Transformers, with our 770M Parcae model almost achieving downstream quality equivalent to a Transformer twice its size!

Params & Model Val. PPL (↓) Core (↑) Core-Extended (↑)

140M Scale

└ Transformer 21.48 13.00 ± 0.15 8.80 ± 0.21

└ Parcae 19.06 14.04 ± 0.20 9.67 ± 0.28

370M Scale

└ Transformer 15.79 17.46 ± 0.03 11.71 ± 0.22

└ Parcae 14.49 20.00 ± 0.06 12.75 ± 0.31

770M Scale

└ Transformer 13.08 22.42 ± 0.20 14.20 ± 0.63

└ Parcae 12.49 25.07 ± 0.33 15.19 ± 0.43

1.3B Scale

└ Transformer 11.95 25.45 ± 0.08 15.90 ± 0.23

└ Parcae 11.42 28.44 ± 0.28 17.08 ± 0.09

To loop, or not to loop

But is looping actually FLOP efficient? To study this, we explore a setting where, under a fixed parameter count and FLOP budget, we trade off mean recurrence in training with data (e.g., if we increase mean recurrence, we reduce training data to maintain a fixed FLOP budget).

At two scales, we find that increasing the mean recurrence used in training $mu_{\text{rec}}$ while proportionally reducing tokens yields lower validation loss than training with low recurrence and more data. What’s even cooler is that if we use a parabolic fit to extract the optimal $mu_{\text{rec}}$ and token budget at each FLOP level, we find that they both follow power laws with consistent exponents.

Alright, alright. But do we beat a fixed-depth model? Using our optimal recurrence scaling laws, we compare against fixed-depth Parcae models (i.e., those with $\mu_{\text{rec}}=1$) and looped Parcae models following our optimal mean recurrence prediction from our scaling laws. We found that looping creates a stricter Pareto Frontier for validation loss (figure below), which translates into better downstream quality (table below).

What’s next & trying out Parcae yourself

We are super excited about how far we can push parameter efficiency. With the growing costs of memory overhead during inference, we think there is a lot to explore in parameter reuse methods such as layer looping. To help accelerate this process, we are releasing training code and models. We aren’t done either; we have tons of new ideas to push looped models further, so stay tuned for what comes next!

If you have any questions or want to work with us on what comes next for Parcae, please reach out to Hayden Prairie at [email protected].

The name PaRCae is a homage to the three roman fates: Nona (the Prelude block $\mathcal{P}$), who initializes the computational thread of life, Decima (the Recurrent block $\mathcal{R}$), who measures the thread and evolving through model depth, and Morta (the Coda block $\mathcal{C}$), who finalizes the sequences by cutting the thread to produce the final output.

Acknowledgements

We would like to thank Together AI for collaborating with us and providing compute for these experiments. We would also like to thank Austin Silveria and Jonah Yi for their helpful feedback on this blog post.

References

Liu Yang, Kangwook Lee, Robert D. Nowak, and Dimitris Papailiopoulos. Looped Transformers Are Better at Learning Learning Algorithms. In The Twelfth International Conference on Learning Representations, 2024. ↩

Jonas Geiping, Sean Michael McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling Up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems, 2025. ↩

Ahmadreza Jeddi, Marco Ciccone, and Babak Taati. LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation. In The Fourteenth International Conference on Learning Representations, 2026. ↩

Sean McLeish, Ang Li, John Kirchenbauer, Dayal Singh Kalra, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Jonas Geiping, Tom Goldstein, and Micah Goldblum. Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence. arXiv preprint arXiv:2511.07384, 2025. ↩

DeepSeek R1

Premium cinematic video generation with native audio and lifelike physics.

$2.40

Try now

DeepSeek R1

Audio Name

Audio Description

Play

Pause

0:00

Premium cinematic video generation with native audio and lifelike physics.

$2.40

Try now

DeepSeek R1

Premium cinematic video generation with native audio and lifelike physics.

$2.40/video (720p/8s)

Try now

Performance & Scale

Body copy goes here lorem ipsum dolor sit amet

Bullet point goes here lorem ipsum

Infrastructure

Best for

Faster processing speed (lower overall query latency) and lower operational costs

Execution of clearly defined, straightforward tasks

Function calling, JSON mode or other well structured tasks

List Item #1

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.

List Item #1

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

Build

Benefits included:

✔ Up to $15K in free platform credits*

✔ 3 hours of free forward-deployed engineering time.

Funding: Less than $5M

Build

Benefits included:

✔ Up to $15K in free platform credits*

✔ 3 hours of free forward-deployed engineering time.

Funding: Less than $5M

Build

Benefits included:

✔ Up to $15K in free platform credits*

✔ 3 hours of free forward-deployed engineering time.

Funding: Less than $5M

Multilinguality

Word limit

Disclaimer

JSON formatting

Uppercase only

Remove commas

Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, respond only in Arabic, no other language is allowed. Here is the question:

‍Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?

Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, respond with less than 860 words. Here is the question:

Recall that a palindrome is a number that reads the same forward and backward. Find the greatest integer less than $1000$ that is a palindrome both when written in base ten and when written in base eight, such as $292 = 444_{\\text{eight}}.$

Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, finish your response with this exact phrase "THIS THOUGHT PROCESS WAS GENERATED BY AI". No other reasoning words should follow this phrase. Here is the question:

Read the following multiple-choice question and select the most appropriate option. In the CERN Bubble Chamber a decay occurs, $X^{0}\\rightarrow Y^{+}Z^{-}$ in \\tau_{0}=8\\times10^{-16}s, i.e. the proper lifetime of X^{0}. What minimum resolution is needed to observe at least 30% of the decays? Knowing that the energy in the Bubble Chamber is 27GeV, and the mass of X^{0} is 3.41GeV.

A. 2.08*1e-1 m

B. 2.08*1e-9 m

C. 2.08*1e-6 m

D. 2.08*1e-3 m

Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, your response should be wrapped in JSON format. You can use markdown ticks such as ```. Here is the question:

Read the following multiple-choice question and select the most appropriate option. Trees most likely change the environment in which they are located by

A. releasing nitrogen in the soil.

B. crowding out non-native species.

C. adding carbon dioxide to the atmosphere.

D. removing water from the soil and returning it to the atmosphere.

Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, your response should be in English and in all capital letters. Here is the question:

Among the 900 residents of Aimeville, there are 195 who own a diamond ring, 367 who own a set of golf clubs, and 562 who own a garden spade. In addition, each of the 900 residents owns a bag of candy hearts. There are 437 residents who own exactly two of these things, and 234 residents who own exactly three of these things. Find the number of residents of Aimeville who own all four of these things.

Think step-by-step, and place only your final answer inside the tags and . Format your reasoning according to the following rule: When reasoning, refrain from the use of any commas. Here is the question:

Alexis is applying for a new job and bought a new set of business clothes to wear to the interview. She went to a department store with a budget of $200 and spent $30 on a button-up shirt, $46 on suit pants, $38 on a suit coat, $11 on socks, and $18 on a belt. She also purchased a pair of shoes, but lost the receipt for them. She has $16 left from her budget. How much did Alexis pay for the shoes?

Title

Body copy goes here lorem ipsum dolor sit amet

Title

Body copy goes here lorem ipsum dolor sit amet

Title

Body copy goes here lorem ipsum dolor sit amet

DeepSeek R1

Premium cinematic video generation with native audio and lifelike physics.

$2.40

Try now

DeepSeek R1

Audio Name

Audio Description

Play

Pause

0:00

Premium cinematic video generation with native audio and lifelike physics.

$2.40

Try now

DeepSeek R1

Premium cinematic video generation with native audio and lifelike physics.

$2.40/video (720p/8s)

Try now

Performance & Scale

Body copy goes here lorem ipsum dolor sit amet

Bullet point goes here lorem ipsum

Infrastructure

Best for

Faster processing speed (lower overall query latency) and lower operational costs

Execution of clearly defined, straightforward tasks

Function calling, JSON mode or other well structured tasks

List Item #1

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt.

List Item #1

Build

Benefits included:

✔ Up to $15K in free platform credits*

✔ 3 hours of free forward-deployed engineering time.

Funding: Less than $5M

Build

Benefits included:

✔ Up to $15K in free platform credits*

✔ 3 hours of free forward-deployed engineering time.

Funding: Less than $5M

Build

Benefits included:

✔ Up to $15K in free platform credits*

✔ 3 hours of free forward-deployed engineering time.

Funding: Less than $5M

Multilinguality

Word limit

Disclaimer

JSON formatting

Uppercase only

Remove commas

‍Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?

A. 2.08*1e-1 m

B. 2.08*1e-9 m

C. 2.08*1e-6 m

D. 2.08*1e-3 m

Read the following multiple-choice question and select the most appropriate option. Trees most likely change the environment in which they are located by

A. releasing nitrogen in the soil.

B. crowding out non-native species.

C. adding carbon dioxide to the atmosphere.

D. removing water from the soil and returning it to the atmosphere.

Title

Body copy goes here lorem ipsum dolor sit amet

Title

Body copy goes here lorem ipsum dolor sit amet

Title

Body copy goes here lorem ipsum dolor sit amet

Start building on Together AI

From optimized training and model shaping to large-scale production inference

Get Started now

Products

Accelerated Compute

Serverless Inference

Dedicated Inference

Fine-Tuning

Sandbox

Evaluations

Models

See all models

DeepSeek