Bio

Maksym Andriushchenko is a second-year PhD student in computer science at EPFL (École Polytechnique Fédérale de Lausanne) in Switzerland. He obtained his MSc from Saarland University, Germany. His research mainly focuses on how to make machine learning algorithms adversarially robust and improve their reliability. Maksym has published seven papers at major machine learning and computer vision conferences (NeurIPS, ICLR, AISTATS, CVPR, and ECCV).

On the Stability of Fine-Tuning BERT

Data Science and Machine Learning

  • Session Language |English
Fine-tuning pre-trained Transformer-based language models such as BERT has become a common practice, dominating leaderboards across various NLP benchmarks. Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process—training the same model with multiple random seeds can result in a large variance of the task performance. Previous literature identified two potential reasons for the observed instability: catastrophic forgetting and the small size of the fine-tuning datasets. In our paper, we show that both hypotheses fail to explain the fine-tuning instability. We analyze BERT, RoBERTa, and ALBERT, which are fine-tuned on three commonly used datasets from the GLUE benchmark, and show that the observed instability is caused by optimization difficulties that lead to vanishing gradients. Additionally, we show that the remaining variance of the downstream task performance can be attributed to differences in generalization, where fine-tuned models with the same training loss exhibit noticeably different test performance. Based on our analysis, we present a simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than the previously proposed approaches.

    Flexible Pricing

    We made sure that no matter what package you choose, you are an active participant in the event and get the best professional experience.
    Students

    Students

    Sold out

    • All Streams
    • All Presentations
    • Q&A Session / Interactive
    • All Videos
    • Networking

    ₴Sold out

    Student ID number is required
    Base

    Base

    Sold out

    • All Streams
    • All Presentations
    • Q&A Session / Interactive
    • All Videos
    • Networking

    ₴Sold out

    Full

    Full

    Sold out

    • All Streams
    • All Presentations
    • Q&A Session / Interactive
    • All Videos
    • Networking

    ₴Sold out

    If you want to buy tickets from the company for a group of colleagues of 5 people, contact our manager to get individual discounts. Terms of return of tickets, see the link.