Skip to content
AI News HubLIVE
Original source2 min read

REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs

Summary

Apple ML researchers introduce REFACTOR-VLA, a wake/sleep system that lets vision-language-action models build reusable libraries of typed motor programs without supervision. Sleep-phase clustering uses a behavioral-equivalence kernel in a learned latent world model; wake-phase synthesis produces Hindley–Milner-inspired typed lambda terms that a rectified-flow decoder turns into actions. On full LIBERO, scaling the world model from 188M to 430M parameters hurt all four suites, while adding InfoNCE during warmup sharply improved skill-clustering NMI, up to 0.915.

REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs
Report an error

The correction channel is not available yet. You can copy the article reference below for later.

Correction instructions
Read article

content type paperpublished September 2026

REFACTOR-VLA: Unsupervised Library Learning of Typed Motor Programs

AuthorsRiyaaz Shaik, Chandru Venkataraman

View publication

Most current vision-language-action (VLA) models—such as OpenVLA, π0, RT-2, and RDT-1B—are “monolithic.” This means they generate raw motor commands or very short sequences of actions, without organizing behaviors into reusable, well-defined abstractions. As a result, these models perform poorly on long-horizon (multi-step) tasks, and it’s difficult to interpret what they have learned. Existing approaches for discovering skills often avoid the core problem of deciding when two action sequences are “behaviorally equivalent.” For example, AtomicVLA and AtomSkill group action sequences by clustering their contrastive embeddings. In contrast, BLADE and LRLL rely on a large language model (LLM) to judge whether two sequences are equivalent, but these LLMs are not calibrated to the robot’s own dynamics. We introduce REFACTOR-VLA, a system that learns reusable skills using a “wake/sleep” architecture. In the sleep phase, the system clusters segments of motor programs using a Behavioral-Equivalence Kernel (BEK). This BEK is based on the outcomes of rolling out actions in a learned latent world model, Mφ. In the wake phase, the system generates typed lambda terms (simple, structured programs) from a vocabulary inspired by the Hindley–Milner type system. These lambda terms are then used by a library-conditioned rectified-flow action decoder to produce actions. Only abstractions that pass both a Minimum Description Length (MDL) criterion and a return-preservation gate are accepted as skills. To train REFACTOR-VLA, we use a three-phase schedule: • Phase A (World-model warmup): The latent world model Mφ is trained. • Phase B (Wake-phase policy optimization): The policy that uses the library of skills is optimized. • Phase C (Sleep-phase skill discovery): The system clusters action fragments into reusable skills. We evaluated REFACTOR-VLA on the full LIBERO benchmark suite. Our results show two main findings. First, simply increasing the size of the world model—from 188 million to 430 million parameters—worsened performance on 4 out of 4 benchmark suites, disproving the idea that just making the world model bigger always helps. Second, changing the training objective makes a big difference: adding an auxiliary supervised contrastive loss (specifically, InfoNCE loss) during the world-model warmup (Phase A) greatly improved the quality of skill clustering in the sleep phase (Phase C). We measured this using Normalized Mutual Information (NMI) under n = 3 multi-seeding: • Object suite: 0.462 ± 0.021 • Spatial suite: 0.867 ± 0.025 • Goal suite: 0.915 ± 0.013 • LIBERO-10 suite: 0.754 ± 0.010

Multimodal Large Language Models (MLLMs) have demonstrated a wide range of capabilities across many domains, including Embodied AI. In this work, we study how to best ground a MLLM into different embodiments and their associated action spaces, with the goal of leveraging the multimodal world knowledge of the MLLM. We first generalize a number of methods through a unified architecture and the lens of action space adaptors. For continuous actions,…

Read more

KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation

May 14, 2024research area Methods and Algorithms, research area Speech and Natural Language Processingconference ICML

Large Language Model or LLM inference has two phases, the prompt (or prefill) phase to output the first token and the extension (or decoding) phase to the generate subsequent tokens. In this work, we propose an efficient parallelization scheme, KV-Runahead to accelerate the prompt phase. The key observation is that the extension phase generates tokens faster than the prompt phase because of key-value cache (KV-cache). Hence, KV-Runahead…

Read more

Key points and analysis

Article intelligence

EngineersAdvanced

Key points

  • Monolithic VLA models like OpenVLA, π0, RT-2, and RDT-1B struggle with long-horizon tasks and interpretability; REFACTOR-VLA learns reusable skills via a wake/sleep architecture.
  • The sleep phase clusters action fragments using a behavioral-equivalence kernel in a latent world model, and the wake phase generates typed lambda terms inspired by Hindley–Milner types.
  • Scaling the world model from 188M to 430M parameters worsened performance on all four LIBERO suites, challenging the assumption that bigger world models are always better.
  • Adding InfoNCE contrastive loss during Phase A substantially improved sleep-phase skill clustering, with NMI reaching 0.915 on the Goal suite.

Highlights and analysis are generated automatically and may contain errors. Check the original source.