Getting video models to learn better, faster

Image and video models have gotten a lot better over the last few years, even though the internals of these models haven't changed much since Stable Diffusion 3.Of course, there have been small variants like the auto-regressive diffusion that GPT-Image popularized. But at a high level, it's pretty much all flow matching with a transformer backbone and a v-prediction objective. In our experience, most of the gains are directly attributable to 3 flavors of data improvements:RL has driven some improvement, but it's only started truly working for image and video in the past few months. Data Filtering & Rebalancing: Remove noisy data and resample your data strategically so your model learns more effectively Data Annotation: Gather better annotations like richer captions, bounding boxes, and font details so that it's easier for your model to disambiguate visual conceptsLLMs have gotten infinitely better at image captioning in the last 12 months. Less so for video, but that's for another time. Synthetic Data Generation: Finetune an ensemble of existing generative models to create training data for which there is little-to-no naturally occurring data (e.g. image editing / reference-conditioning for Nano-Banana style models)Often a collection of brittle LoRAs trained for very specific tasks and an LLM finetuned to filter out bad synthetic data. A couple of years ago, the prevailing wisdom across all generative models (be it text, image, audio) was to aggregate as much data as humanly possible for pre-training. Luckily, the field has gotten a lot smarter about this. If you throw a bunch of low-quality data (e.g. heavily compressed JPEGs) into pre-training, your model is going to waste a significant amount of its capacity learning how to mimic this slice of data. If you filter your dataset well, your model will have a lot easier time learning what you want it to learn. We know this sounds obvious, but it's a lot harder to do in practice.Then again, all good advice should seem obvious in retrospect. Today we're going to walk you through how our approach to data filtering has evolved since 2024. And, hopefully we'll save you from a couple of headaches if you end up training your own generative models down the line. 2024Old-school CV on CPUs