Register & buy
Register & buy

Data Science and Machine Learning

Learn about the latest algorithms and their application to solve various Data Science and Machine Learning problems. Immerse yourself in modern developments in the field of Deep Learning for Computer Vision, Natural Language Processing, Speech recognition and other areas.

Artificial Intelligence for Business & Products

You will learn how to use AI to optimize business processes in various industries, increase profits, create new products for the market, as well as how to start startups and manage the AI teams.

Data Engineering & Analytics

This is a stream focused on examples of solutions in the field of data engineering, business intelligence and data analysis. It will be considered the issues of data storage, management, processing, analysis and visualization.

Natalya Avanesova

Head of NLP

Preste | Kyiv, Ukraine

Automatic Text summarization: is GPT3 the best and only?

10:00

Data Science and Machine Learning

  • Session Language |Ukraine
Automatic text summarization is still one of the most challenging and interesting problems in the field of Natural Language Processing. The demand for automatic text summarization systems is spiking these days thanks to the availability of large amounts of textual data, and it really can aid many downstream applications such as creating news digests, report generation, users opinion summarization and so on. Sophisticated abilities that are crucial to high-quality summarization are possible only in an abstractive framework. Meanwhile, GPT-3 could help to generate human-like summaries of desired quality. But is it really the best and the only way to generate a high-quality summary? In speech, we will explore the realms of abstractive summarization using GPT3 and alternative techniques and we will compare them according to their performance for achieving the required summary.

10:45

Maksym Andriushchenko

Research Intern / PhD Student

Adobe/EPFL | Lausanne, Switzerland

On the Stability of Fine-Tuning BERT

10:50

Data Science and Machine Learning

  • Session Language |English
This talk will be based on our recent paper accepted to ICLR 2021. Fine-tuning pre-trained Transformer-based language models such as BERT has become a common practice, dominating leaderboards across various NLP benchmarks. Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process—training the same model with multiple random seeds can result in a large variance of the task performance. Previous literature identified two potential reasons for the observed instability: catastrophic forgetting and the small size of the fine-tuning datasets. In our paper, we show that both hypotheses fail to explain the fine-tuning instability. We analyze BERT, RoBERTa, and ALBERT, which are fine-tuned on three commonly used datasets from the GLUE benchmark, and show that the observed instability is caused by optimization difficulties that lead to vanishing gradients. Additionally, we show that the remaining variance of the downstream task performance can be attributed to differences in generalization, where fine-tuned models with the same training loss exhibit noticeably different test performance. Based on our analysis, we present a simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than the previously proposed approaches.