[PDF] GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost | Semantic Scholar (2024)

[PDF] Semantic Reader

This work proposes RDED, a novel computationally-efficient yet effective data distillation paradigm, to enable both diversity and realism of the distilled data in large-scale real-world applications.

This work proposes a procedure to exactly compute the unrolled gradient with constant memory complexity, which allows to scale MTT to ImageNet-1K seamlessly with ~6x reduction in memory footprint, and discovers that it is challenging for M TT to handle datasets with a large number of classes, and proposes a novel soft label assignment that drastically improves its convergence.

This work proposes to align the difficulty of the generated patterns with the size of the synthetic dataset, and successfully scale trajectory matching-based methods to larger synthetic datasets, achieving lossless dataset distillation for the first time.

The proposed dataset condensation framework demonstrates flexibility across diverse dataset scales and exhibits multiple advantages in terms of arbitrary resolutions of synthesized images, low training cost and memory consumption with high-resolution synthesis, and the ability to scale up to arbitrary evaluation network architectures.

This work shows that it can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model and introduces a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse.

G-VBSM is the first algorithm to obtain strong performance across both small-scale and large-scale datasets and surpasses all SOTA methods by margins of 3.9%, 6.5%, and 10.1%, respectively.

This work proposes a simple yet effective method that synthesizes condensed images by matching feature distributions of the synthetic and original training images in many sampled embedding spaces and significantly reduces the synthesis cost while achieving comparable or better performance.

This paper identifies two important shortcomings of naive distribution matching and addresses them with three novel techniques and outperforms most previous optimization-oriented methods with much fewer computational resources, thereby scaling data condensation to larger datasets and models.

A new optimization algorithm is presented that distills a large number of images into a few intermediate feature vectors in the generative model's latent space, significantly improving cross-architecture generalization in all settings.

This work unifies the formulations of the two tasks by decomposing and reorganizing the generic KD loss into a Normalized KD (NKD) loss and customized soft labels for both target class (image’s category) and non-target classes named Universal Self-KD (USKD).

...

...