CARES: Coverage-Aware Replay Exemplar Selection for Continual Learning

Abstract

Replay-based Continual Learning mitigates catastrophic forgetting by keeping a buffer of past samples, making buffer composition critical under a fixed memory budget. Recent coreset-based replay methods improve buffer quality using model-derived gradients, losses, or auxiliary optimization, but incur substantial computational overhead as memory is repeatedly updated. We propose CARES, a continual model-optimization-free approach for buffer construction that instead exploits the geometry of a frozen pretrained feature space using class-aware, density-weighted coverage. We evaluate CARES across balanced, imbalanced, and label-noisy streams under online class- and task-incremental settings. On Tiny-ImageNet, CARES is 2.3 to 8.8 times faster than existing coreset-based selectors. Although CARES achieves this speedup using no training signal from the continual learner, it ranks among the state of the art across balanced, imbalanced, and label-noisy streams under online class- and task-incremental settings, outperforming the strongest baseline by up to 7.58 percentage points under class imbalance while also attaining higher average accuracy on Tiny-ImageNet. Code will be released upon publication

Citation

If you use this work, please cite our paper:

@misc{shanbhag2026cares,
  title = {CARES: Coverage-Aware Replay Exemplar Selection for Continual
           Learning},
  author = {Arundhati S. Shanbhag and Brian B. Moser and Federico Raue and
            Tobias C. Nauen and Andreas Dengel},
  year = {2026},
  eprint = {2610.32799},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
}