Dissertation Defense of CMSE Xitong Zhang
Department of Computational Mathematics, Science & Engineering
Michigan State University
Dissertation Defense Notice
June 5, 2024 10:00AM—11:00AM (EST)
Zoom (Only): https://msu.zoom.us/j/98801161872
Meeting ID: 988 0116 1872
Passcode: CMSE
Deep Learning Regularization: Theory and Data Perspectives
Xitong Zhang
Abstract:
Generalization is a central research topic in deep learning. To enhance the test performance
of well-trained models on unseen data, it is essential to apply regularization techniques
that refine the model's expressive capabilities and the training process.
This dissertation categorizes regularization into theory-driven and data-driven approaches. Theory-driven regularization encompasses methods broadly applicable across various contexts, including conventional techniques such as weight decay and dropout. Conversely, data-driven regularization involves techniques specifically designed for particular data sets and applications. For instance, different neural network architectures can be developed to capture various useful patterns in data for specific applications.
The presentation will mainly focus on theory-driven regularization. Specifically, a training algorithm based on the PAC-Bayes bound will be introduced. The PAC-Bayes bound is the estimated upper bound of the test error of a machine learning model. Although minimizing the upper bound of the test error should intuitively benefit generalization, minimizing existing PAC-Bayes bounds often results in compromised test performance compared to empirical risk minimization (ERM) with commonly used regularization techniques such as weight decay, large learning rates, and small batch sizes. The designed algorithm seeks to bridge the gap between theoretical tightness and practical effectiveness of the PAC-Bayes bound in boosting test accuracy for classification tasks.
Committee Members:
Rongrong Wang
Saiprasad Ravishankar
Jianrong Wang
Youzuo Lin
