Data Science and Machine Learning
-
Session Language |Ukraine
Web application testing is an important step to improve the predictability of the user experience. Modern web programs typically contain a huge
variety of functionalities, and they all need to be tested. This leads to inefficiencies and increased costs of manual testing and requires automated testing. In this talk, we demonstrate the ability of machine learning to perform automated testing of web applications using only screenshots. Imitation learning is the main idea behind our approach.
We created a set of more than 4000 demonstrations of people (screenshots and related actions) on the website.
The training set includes about 300 different scenarios and 43 types of actions (e.g., "click", "enter email"). We experimented with two different models. As the first
model, we used a simple convolutional neural network (CNN) to classify 43 actions and regress the coordinates of mouse clicks on the screen. Using whole screenshots, we were able to simulate a demonstration with 99.9% accuracy in action classification and 26-pixel RMSE in coordinate regression. As a second approach, we used Siamese CNN to match the patterns between the elements that are clicked, and the entire screenshot. This model successfully distinguished 95% of the patterns.
The resulting models can be used to test web applications or perform tasks using only screenshots of websites. In the near future
we plan to develop more intelligent models based on reinforcement learning in addition to the machine learning approaches used