Tackling Dimensional Collapse toward Comprehensive Universal Domain Adaptation

Abstract

Universal Domain Adaptation (UniDA) addresses unsupervised domain adaptation where target classes may differ arbitrarily from source ones, except for a shared subset. A widely used approach, partial domain matching (PDM), aligns only shared classes but struggles in extreme cases where many source classes are absent in the target domain, underperforming the most naive baseline that trains on only source data. In this work, we identify that the failure of PDM for extreme UniDA stems from dimensional collapse (DC) in target representations. To address target DC, we propose to jointly leverage the alignment and uniformity techniques in self-supervised learning on the unlabeled target data to preserve the intrinsic structure of the learned representations. Our experimental results confirm that SSL consistently advances PDM and delivers new state-of-the-art results across a broader benchmark of UniDA scenarios with different portions of shared classes, representing a crucial step toward truly comprehensive UniDA.

Video

Motivation

Left: Universal domain adaptation addresses both domain shift and category shift. However, category shift scenarios with high source-private ratios remain under-explored. Right: In such extreme cases, existing partial domain matching (PDM) methods that align source and target data on shared classes fail to outperform the simplest source-only baseline.

Key Findings

Dimensional Collapse under Extreme UniDA

Our analysis reveals that dimensional collapse occurs when the source-private ratio is high. We demonstrate this phenomenon through both a toy example for intuitive visualization and rigorous analysis of singular value spectrum.

Left: Toy example for visualization, where we sample different numbers of source-private data to study the effect under extreme UniDA. The target representations collapse to a single line under high source-private ratios. Right: Singular value spectrum of target representations under different source-private ratios. Several singular values drop to zero under high source-private ratios.

Degraded Representation Quality Impairs PDM

Our analysis reveals that dimensional collapse occurs when the source-private ratio is high. We demonstrate this phenomenon through both a toy example for intuitive visualization and rigorous analysis of singular value spectrum.

Partial domain matching relies on accurate uncertainty estimation, which could be compromised by poor representation quality.

Under high source-private ratios, uncertainty estimation must be highly accurate to outperform the source-only baseline, yet the estimation error is substantial. Conversely, under low source-private ratios, moderate accuracy suffices and the estimation error is low.

Address Dimensional Collapse without Labels

De-collapse Techniques from Self-Supervised Learning

Dimensional collapse is a well-known issue in self-supervised learning, primarily caused by the contrastive alignment term (Li et al., 2020). Various methods have been developed to address this problem, including contrastive learning approaches (e.g., AlignUniform, SimCLR), asymmetric models (e.g., SimSiam, BYOL), and redundancy reduction techniques (e.g., VICReg, Barlow Twins). Here, we show how the uniformity term from self-supervised learning can effectively prevent DC in UniDA. We further show that these SSL approaches also work in the paper.

The alignment term alone worsens the dimensional collapse, while the uniformity term effectively prevents it. Combining both terms achieves the best performance.

Adding SSL consistently improves PDM performance across the whole spectrum, and substantially outperforms baselines in extreme UniDA scenarios where DC is severe.

@inproceedings{dcunida_fang2025,
    title={Tackling Dimensional Collapse toward Comprehensive Universal Domain Adaptation},
    author={Hung-Chieh Fang and Po-Yi Lu and Hsuan-Tien Lin},
    booktitle={Proceedings of the International Conference on Machine Learning (ICML)},
    year={2025},
}