Large Pre-Training Datasets Don't Guarantee Robustness after Fine-Tuning in Image Classification

Radar chart of OOD accuracy of ViT-B/16 pretrained on LAION-2B and IN-21K, before and after fine-tuning on ImageNet-R — **OOD accuracy (robustness) of a ViT-B/16 model pretrained on two different datasets (LAION-2B, IN-21K), before and after fine-tuning on ImageNet-R.** LAION-2B pretrained model shows worse OOD generalization after fine-tuning compared to ImageNet-21K pretrained model although both were similar before training.

Abstract

Large-scale pretrained models are widely leveraged as foundations for learning new specialized tasks via fine-tuning, with the goal of maintaining the general performance of the model while allowing it to gain new skills. A valuable goal for all such models is robustness: the ability to perform well on out-of-distribution (OOD) tasks. We assess whether fine-tuning preserves the overall robustness of the pretrained model in image classification, and observed that models pretrained on large datasets exhibited strong catastrophic forgetting and loss of OOD generalization. To systematically assess robustness preservation in fine-tuned models, we propose the Robustness Inheritance Benchmark (ImageNet-RIB). The benchmark, which can be applied to any pretrained model, consists of a set of related but distinct OOD (downstream) tasks and involves fine-tuning on one of the OOD tasks in the set then testing on the rest. We find that though continual learning methods help, fine-tuning reduces robustness across pretrained models. Surprisingly, models pretrained on the largest and most diverse datasets (e.g., LAION-2B) exhibit both larger robustness losses and lower absolute robustness after fine-tuning on small datasets, relative to models pretrained on smaller datasets. We observe this collapse in contrastively pretrained (CLIP) models and their fine-tuned variants, where it grows with pretraining scale; the supervised models we test do not exhibit it. These findings suggest that starting with the strongest foundation model is not necessarily the best approach for performance on specialist tasks.

ImageNet-RIB Benchmark

Evaluation Metrics

The robustness improvement (RI) on i-th downstream dataset is defined as the average accuracy difference between fine-tuned model and pre-trained models:

$$RI_i = \frac{1}{N-1} \sum_{j=1, j \neq i}^N A^{(j)}_i - A_\text{pre}^{(j)}$$ $$mRI = \sum_i^N RI_i$$

Experiments on ImageNet-RIB

Average accuracy on OOD datasets of pretrained and fine-tuned models for each pretraining dataset — There are huge performance drops after fine-tuning on downstream datasets using OpenAI or LAION-2B pre-trained models.

Average accuracy on downstream and OOD datasets over fine-tuning epochs for each pretraining dataset — IN-21K with AugReg pre-trained models overfits on the downstream dataset and OpenAI pre-trained models learns the slowest. However, only OpenAI and LAION-2B pre-trained models suffer huge robustness degradation.

Accuracy on each dataset versus the number of samples per class of the ImageNet-1K training set — Rather, fine-tuning on small dataset leads severe catastrophic forgetting regardless of hyperparameter tuning.

This forgetting only happens when pretraining dataset is larger than 100M and the fine-tuning dataset is small

ViT-B/32	LAION-100M	LAION-400M	LAION-2B
$mRI$	−7.3	−23.8	−33.0

Citation

@article{hwang2026large,
    title   = {Large Pre-Training Datasets Don't Guarantee Robustness after
               Fine-Tuning in Image Classification},
    author  = {Hwang, Jaedong and Cheung, Brian and Hong, Zhang-Wei
               and Boopathy, Akhilan and Agrawal, Pulkit and Fiete, Ila R},
    journal = {TMLR},
    year    = {2026},
}

Large Pre-Training Datasets Don’t Guarantee Robustness after Fine-Tuning in Image Classification

Abstract

ImageNet-RIB Benchmark

Evaluation Metrics

Experiments on ImageNet-RIB

Citation

More research