RIP : A Simple Black-box Attack on Continual Test-time Adaptation

Trung-Hieu Hoang1, Duc Minh Vo2, Minh N. Do1,
1University of Illinois at Urbana-Champaign, 2The University of Tokyo
arXiv Code

rip_overview

An illustration of our Reusing of Incorrect Predictions (RIP) attack against a continual test-time adaptation (TTA) method. Here the attacker intentionally reuses samples that were incorrectly predicted in the subsequent rounds to make the model more confident in these erroneous predictions. RIP is the first black-box attack that can realistically collapse a TTA model.

Abstract

Test-time adaptation (TTA) has emerged as a promising solution to tackle the continual domain shift in machine learning by allowing model parameters to change at test time, via self-supervised learning on unlabeled testing data. At the same time, it unfortunately opens the door to unforeseen vulnerabilities for degradation over time. Through a simple theoretical continual TTA model, we successfully identify a risk in the sampling process of testing data that could easily degrade the performance of a continual TTA model. We name this risk as Reusing of Incorrect Prediction (RIP) that TTA attackers can employ or as a result of the unintended query from general TTA users. The risk posed by RIP is also highly realistic, as it does not require prior knowledge of model parameters or modification of testing samples. This simple requirement makes RIP as the first black-box TTA attack algorithm that stands out from existing white-box attempts. We extensively benchmark the performance of the most recent continual TTA approaches when facing the RIP attack, providing insights on its success, and laying out potential roadmaps that could enhance the resilience of future continual TTA systems.

BibTeX

@misc{hoang2024rip,
        title={{R.I.P.}: A Simple Black-box Attack on Continual Test-time Adaptation}, 
        author={Trung-Hieu Hoang and Duc Minh Vo and Minh N. Do},
        year={2024},
        eprint={2412.01154},
        archivePrefix={arXiv},
        primaryClass={cs.LG},
        url={https://arxiv.org/abs/2412.01154}, 
      }