PhD defence of Naz Mohammadi Sepahvand – Knowledge Transfer for Learning and Unlearning in Deep Neural Networks

Abstract

This thesis explores how knowledge transfer techniques can enhance both learning and unlearning in deep neural networks. The work begins by investigating knowledge distillation (KD), with a focus on the under-explored role of the capacity gap between teacher and student models. Through empirical studies, we show that tailoring teacher-student capacities to data difficulty improves performance. Inspired by these findings, a novel KD method is developed that dynamically adjusts teacher capacity, outperforming existing approaches. The focus then shifts to machine unlearning, a process enabling models to forget specific data points, which is critical for applications such as data privacy, copyright concerns, and the removal of harmful information. To address the limitations of existing exact and approximate unlearning methods, this thesis introduces transfer unlearning, which replaces data flagged for future removal with carefully selected auxiliary samples. This approach avoids retraining, ensures model performance, and provides practical unlearning guarantees. Finally, this thesis introduces the hypothesis that residual information left in internal layers can undermine the effectiveness and security of unlearning methods. To address this, a method inspired by domain adaptation is proposed, shifting unlearning to the representation space using domain adversarial training. This method aligns the unlearned model’s internal representations with those of an oracle model trained without the data to be forgotten. Empirical results demonstrate that the proposed approaches outperform competing methods across benchmarks and strengthen defenses against both black-box and white-box attacks, advancing the state of unlearning in deep neural networks.