Session

Anna Nikulina
21 Sep 2019
17:55-18:40
AI for Business and Products

How to test your chatbot

Would you happen to know how a chatbot works? Is there an artificial intelligence inside? How to train chatbot? How to test that it is “smart” enough?

Our team now is integrating google chatbot to our application. The bot supports small talks, sends FAQ articles, sends dynamic responses depending on the input parameters – device model, etc. The bot even can do actions, such as cancellation of an account.

Our team did a great job by automating manual efforts to train and deploy the bot. We have several instances of the bot and we built a continuous delivery process based on testing.

The tricky part is that there is no expected result, we basically do benchmarking. We calculate metrics and evaluate results based on the comparison.

The threshold indicates whether the bot was successfully trained or not, is defined by myself and this is the most essential part. My experience is interesting for people who test chatbots and responsible for the quality of bot training models. I will tell you about the challenges we faced while training the bot, how we adjusted our approach when dialogues became huge and more context was added.