Mid-Training Language Models on Raw Video

A vision encoder \(g_\phi\) and a projector \(p_\psi\) map each frame \(x_t\) to \(N\) continuous visual tokens in the hidden dimension \(d\) of the language model, \(z_t=p_\psi(g_\phi(x_t))\in\mathbb{R}^{N\times d}\). Each token grid is serialized in row-major order with a learned newline embedding after every row, and the frames are concatenated in temporal order into one sequence \(V=[v_1,\dots,v_L]\). The language model \(f_\theta\) reads this sequence causally, and a two-layer MLP head \(r_\omega\) predicts the next visual token from each hidden state \(h_i=f_\theta(v_{1:i})\). Training minimizes the cosine distance \(\mathcal{D}\) between the prediction and the next token, with a stop-gradient \(\mathrm{sg}(\cdot)\) on the target,

Because attention is causal, the same loss trains next-patch prediction within a frame and prediction across frame boundaries. The objective needs no captions, no discrete codebook, and no text loss, and the prediction head is used only during mid-training.

Setup. Following Cambrian-S, a SigLIP SO400M/14 encoder at 384×384 resolution is connected to Qwen3-1.7B by a two-layer MLP projector. Mid-training uses 16-frame clips sampled at 0.2 frames per second from YT-Temporal-1B, a collection of public YouTube videos, which gives 6.99M clips and 64.4B visual tokens. All parameters are trained end to end for one epoch (27,300 steps) on 128 H100 GPUs, with 256 clips per batch and a 16k-token context. The mid-trained model and the model without mid-training then receive the same instruction tuning on LLaVA-OneVision-Data (3.9M images), so the two differ only in mid-training.

Mid-training improves every video and image benchmark. The largest video gain is on long-form egocentric understanding, where EgoSchema rises from 45.2 to 50.0. The largest image gains are on chart, document, and diagram tasks such as ChartQA (+8.2), DocVQA (+6.4), and AI2D (+6.1), even though mid-training uses no image-text pairs.

Mid-training contains no text, yet after instruction tuning the mid-trained model reaches a text average of 48.9 over 14 benchmarks, compared with 48.0 for the model without mid-training. Every knowledge and commonsense score stays within 1.4 points of the model without mid-training. Both models fall below the original Qwen3-1.7B on GSM8K, MATH500, and BBH. For the model without mid-training, this loss comes from instruction tuning alone, whose data contain only image-text examples, and the mid-trained model loses less on all three benchmarks.

Image and video averages rise by 4.9 and 3.1 points within the first 0.3 epochs (19.5B visual tokens) and change by less than 0.5 points over the remaining 0.7 epochs. The text average stays within 0.9 points of the model without mid-training at every checkpoint. The paper reports the trajectory of each benchmark.

We replace the visual target with a text caption of the current frame or of the next frame. Qwen3-VL-30B-A3B captions consecutive frame pairs with a progress-aware prompt, so the two captions of a pair differ where the action has progressed. All models receive the same instruction tuning, and the table reports the average accuracy on each benchmark suite.

Next-caption prediction reaches video and image averages within 0.3 points of visual next-token prediction, but its text average is 1.5 points lower. At a matched training budget, the 0.5-epoch checkpoint of visual next-token prediction (52.37 video, 53.76 image, 48.32 text) stays within 0.3 points of next-caption prediction on video and image and scores 0.9 points higher on text. Current-caption prediction falls 5.0 points behind on images. Video mid-training can therefore remain fully self-supervised, without the computational cost and label noise of automated captioning.