From Chatbots to Augmented Conversational Assistants

May 27, 2021 11:00 AM (PT)

Download Slides

From Chatbots to Augmented Conversational Assistants: An Experimental Study combining AI and Crowdsourcing

Developing technology, including Natural Language Understanding and Machine Learning are taking Chatbots, Virtual Assistants, Conversational Bots… to an extent unheard of in the past. And yet, we cannot but notice that efforts are still needed for these interactive agents to attain a satisfactory level of conversational abilities. As a matter of fact, Bots still often cause a classic frustration effect when they reach limits in the scope of their powers.  

In order to take Bots beyond their technical boundaries:  

  1. We augmented them with technologies such as advanced data processing and Artificial Intelligence, including computer vision specific capabilities to mention just a few.
  2. We designed a unique approach based on crowdsourcing methods complementing and enhancing existing techniques to AI / Deep Learning.

In this talk, we will present the result of our research based on two tracks: 

  1. As part as our first iteration of research (aka “AVA”), we focused our efforts on identifying the most relevant AI/ML features that bring advanced conversational capabilities to Chatbots.  
  2. In a second iteration (aka “CIVA”), we concentrated our research on bringing more understanding and replying abilities to Bots defining a progressive methodology where data are improved. 

In this presentation you will see how we augmented Chatbots capabilities with: 

  1. Machine Learning 
  2. Computer vision
  3. Sentiment analysis
  4. Crowdsourcing
  5. Conversational methods
In this session watch:
Frederic Jacquet, Chief Innovation Officer, Talan
Lao-dja Songdou Tchala, Data Scientist, Talan



Frederic: Hello everybody. You going to attend to our session from chatbots to augmented conversational assistant. I am Frederic Jacquet, and Lao-dja will talk with me on this session. In 2016, Gartner came up with this prediction. By 2020, the average person will have more conversation with bots than with their spouse. Well, we’re not there yet, but it is clear that virtual assistants are now very present in our lives, both on the personal and professional aspects. This is why 18 months ago we decided to search how we could take them to the next level leading to augmented conversational tools. You know that developing technologies, making bots more or less mainstream now. They are used in various fields including health care, call centers, travel booking, online shopping, [inaudible], information and services et cetera. But chatbots still needs improvements. As a matter of facto bots still often cause a classic frustration effect when they reached limits in the scope of their abilities.
According to a 2019 Forester Study, major areas needs to be addressed, including the ability for a bot to have knowledge of user’s history and previous interactions, generate automated action based on user’s responses, lead users will clear conversation just to mention a few. A conversational tool aims to serve two different purposes. From the company point of view, this is efficiency, while from the customer’s point of view, this is satisfaction. What could be the main frustrations areas I mentioned? First, bots don’t understand my question. Number two, bots don’t give me the answer I expect. And then the third point is the means for interaction with bots are way too limited. So what did we do? We augmented them with technology such as advanced data processing and AI. And we designed a unique approach based on no crowdsourcing methods of complimenting and improving AI techniques.
Next slide. So in this presentation, what you’re going to see how AI and machine learning can improve chatbots, conversational capabilities, how human can improve machine learning capabilities based on our crowdsourcing method. And finally how the combination of technologies and methods and human can lead to an inclusive bot. And now I’m going to pass to Lao-dja.

Lao-dja: Okay, thank you Frederic. Now we were going to see how machine learning and AI can be used to augment chatbot capabilities. So the fact AI competent that we identified to augment chatbot capabilities is the computer vision multiple choices, which will allow the user to choose from different options that the boat could be showing on the screen. We think it will make the conversation between the user and the chatbot more natural and that will increase the satisfactory level we are aiming for. So let’s see how it works.
So at the beginning we feed and try to model with thousands of images that represent hand in different position. We then use that model in the second step to locate a hand in the picture. Then we do some calculation to draw a circle around the hand and find this center of that circle. And then finally we write code to specify which action to do depending on the position of the targeted center. It’s good to see a demo now. Now just to mention, this feature isn’t integrated to the bot yet, but here we can see how it would look like. We execute the code that will lead in the screen with different options to choose from and clearly we can see that we have two points, one in yellow that followed the hand movement. And the other one, which is across that follow the mouse movement.
We can also see that those two points are moving quite independently. That’s showing that we really have hand on one side and the growth in the other side. Now let’s choose the top right option to open the submit page in order to register. Now the page is opened, let’s close it and let’s put the application. So that’s it. That’s all for the demo of multiple choices that would make the conversation a little more natural. Let’s see the second feature that we identify to augment the chatbot. Now, the second feature that we identify to augment a chatbot, the second machine learning feature is recommendation algorithm, which is basically the possibility for the bot to predict the user’s intention based on very few words or few tips that the user will be given. So we think it will allow the user to make himself understood in a better way. Now let’s see how it works.
So at the beginning, you start with users sending messages to the bot in full text. In a second step, the bot will give answers to those questions that was asked. And then in the third question, the bot will grab the fact that the user liked the answers or not, and for those answers that was liked by the user there will be a mapping between the users questions and the bot’s answers. And then in the fourth step, by doing some frequency calculation, the bot will extract the keywords from those questions so that it will fit a model to predict the user question whenever the user would give those keywords. And according to that, the bot will give the corresponding answer. Now a use case for this algorithm is a help desk use case where some users will be asking from some equipments and the bot using their keywords will predict their needs, just some supplement [inaudible] equipments that they might need.
So it starts with users that are authenticated at the very beginning of the conversation. So we can have an administrator and manager or a developer, they send a message in full text to the bot and the bot going through all the steps that we’ve seen above, bot giving an answer, doing the mapping, extracting keywords will predict the user’s role and then in the future, when the user will have another conversation with the bot, the bots will use the few words that the user will be given to predict its role and suggest some supplementary elements or equipments that the user might need just because other users with the same role has needed that previously. So that’s all. Let’s see a demo of the recommendation.
So here, basically I will be asking for some equipment and the boat will be based on what I asked to predict my role. So first let’s see, I asked for a calendar access and a mouse. The bot will say, Okay, you asked me about calendar access and mouse and keyboard. So you must be a manager and I thought you must need a computer and a smartphone.” Now, if I change my mind and ask, instead of counter access, I asked for a styling VS Code, the bot will this time say, “Okay, now you ask me about VS Code and a computer. So I guess you must be a developer. And I saw you must need a smartphone, dev tools, a keyboard and a mouse.” So that’s how it works. The bot will predict our induction and our role, and we’ll suggest supplementary element that we might need. That’s all for the recommendation algorithm.
Now let’s see how human and conversational methods can be used to increase both capabilities. So how does humans contribution augment a chatbot? The chatbot, but itself doesn’t have the necessarily imagination to extend it’s replying and understanding abilities up to that satisfactory level that most of people will be in right to wait for. So we think by bringing in the human’s emotion and creativity we can make that happen. That’s why we created a crowding platform where different people of any kind can come and propose alternative ways of asking questions and different ways of replying to those questions. And they can also correct each other’s proposition and remove them if necessary. So let’s see how it works.
So at the beginning we have a Q&A when each question is linked uniquely to one and only one answer, then Crowders working together, will propose for each question different ways of asking it and for each answer, different way of reply. And then in a second step, again working together, they will correct each other’s questions and others, and that work will lead to augmented and clear database. In the third step, we will use that clean and augmented database to feed the chatbot. So it will be able to adapt its answers based on what user need that we’ll be seeing soon, to ask for clarification if the user’s question isn’t clear enough and reformulate its answers if the user doesn’t get it.
Let’s see quickly a demo. So here, what we’ll be seeing at the first step, the Crowder will be proposing different ways of asking some questions. And then afterwards they will be giving scores to those questions based on few criteria, such as data’s complexity, and afterward those notation and proposition will be stored in the database. Now what they did with the question, they will be doing exactly the same thing, but now with others. So they will propose alternative answers and they will give notations for those answers based on same criteria and then after, all that will be stored in database again and will be feeding the chatbots. Now, alongside this crowding platform, we also have a monitoring system to make sure that the quality of the work will be high. So what does it consist in? Each Crowder can see his level, his score, how many questions he has proposed, how many answers they have given.
And we have levels such as a rookie, intermediate or an expert. And the idea of gamification is that when a Crowder see that, “Okay, I am a rookie and I want to do better than my mates. So I would work better.” And that’s the process of gamification. So it worked quite well. And in the same time, we have the possibility for the manager or the administrator to remove a Crowder when he’s not doing his job correctly. For instance, if he has offending words in his proposition. So he can be suspended and the administrator can also give him the right back. So that’s how the crowding work alongside this monitoring. Now, let’s see what the human’s contribution has brought to the bot at features. Now, the first feature that the human’s work has brought to the bot is the possibility for the boat to adapt his other to the user’s need.
So basically if the user say, “Okay, now I want short answers, the bot will take that in account and break short answers to the user. And anytime of the conversation, the user can change his mind and say, “Now I want more details because I don’t want to go straight to the point. I want to know everything you can propose.” And then the bot will take that in account and give data to the user. So let’s see how it works. At the beginning, the user sent a request to the chatbot in four texts. And the second step, the bot will give answers to the user. Once it does that in the third step, it will grab the fact that the user like his others are not, and will store that information in his memory in the forth step.
Alongside that, the bot for the answers, that it doesn’t know anything, it will do a prediction based on his memory to say, if the user would like or not the answer that he’s about to give and over conversation, the bot will be updating it’s model. So that at the end, it will be almost certain that of which answer the user will like. So that’s what we are aiming to do. But in the demo, what we’ll be seeing is the simple example where the user needs will be only short if he wants short answers or long answers. So let’s get to see a demo. So here at the beginning of the conversation, I will specify to the bot that I want short answers.
So we can say that all answers that the bot will be giving will be short. And now if I change my mind and say, now I want long answers, we will see that the answers that they worth we’ll take that in account and answers that you will be giving me will be long. So that’s all for adaptive answers. Let’s see other features that the human’s work has brought. The second feature that the human work have brought to the boat is the possibility for the chatbot to reformulate his also any time the user doesn’t understand. It can seem basic, but actually you can imagine during a conversation, when you say you don’t understand, and the bot keeps saying the same thing, or even worse, reply saying it doesn’t understand, it can be quite frustrating.
And we think this feature can get rid of that undesired behavior. So let’s see how it works. First the user sends the question to the chatbot and the chatbot will choose among the answer’s of it’s knowledge based with one answer that is linked to the question that the user have asked. And in the same time, the bot will store in his memory the given answers, and will tag it as already given. So now in the third step, when in the future, the user sends not the same question to the chatbot, the chatbot will again choose one answers from those of its knowledge base, but this time among those, that wasn’t targets that will choose responses that wasn’t already targeted as already given and as the conversation will be going, the bot will be updating its memory until it doesn’t have any other response for this specific question.
And that’s where the bot will say, “Okay, sorry for this specific question, I have no other answer to propose you. Do you want me to propose you an answer that you have already seen?” And the user can say yes or no. Depending on his answer, the bot will act accordingly. So let’s quickly see a demo now. So basically what I will be saying doing here is to say each time the bot will give me an answer that I don’t understand. So let’s go, I ask a question about a visa and they bot give me an answer. And I keep saying, I don’t understand. I don’t understand. And again, I don’t understand. And each time I will be saying that the bot would propose a different answer. So he could propose a different way of saying the same answer until there is no other answer to give. And then the bot will say, “Sorry for this specific question, I have no other answer to propose you.”
That’s all for their reformulating capability. Let’s see the third and last feature that the Crowder’s work has brought to the bots. It is the possibility for the chatbot to ask for clarification, whenever the user’s question, isn’t clear enough. So it’s something different from the bot saying, I don’t understand. It is okay. I’m a bit confused. What are you saying? Are you talking about this thing or this other things? So that’s why we’ll be seeing here. First, let’s see how it works. So the state of that is that at the beginning, we have at least two questions that are really similar in their meanings. So when the user asks one of those questions, the bot is confused and doesn’t know which answer to give, because he had at least two possibilities.
So what the bot will be doing is to extract keywords from each of those questions that were similar and alongside that it will also extract keywords for the alternative ways of asking those questions that we grab from Crowder’s work. And then by doing frequency calculation, it will grab the most frequent words that characterized the question, and it will use those questions in the future conversation to ask for clarification to the user saying, “Do you mean this bunch of keywords or rather this other bunch of keywords and the user will choose from that.
Let’s see a demo. So just to clarify things here we are in the context of making a travel to Bali. So I’ll be asking a question about animals and possibility to cancel my travel. So let’s go, I ask a question saying, “Can I cancel without any other information?” And then the bot will say, “I’m not sure I’ve understood what you want. Do you mean you want to cancel your reservation, or you want to cancel your trip?” I will choose the first one. Say, “I want to cancel my reservation.” I would choose the first option saying, I want to consume a reservation. And the bot will give me the answer. Now, what I will be showing here is that we can bring together or the features that we discussed above.
So asking for clarification, adaptive answer, and now what I will do, I will ask for more details. So we can see that you can also use adaptive features. So I say, “Can you give me more details?” And the bot give a bit more details as for this specific question, we don’t have in it’s knowledge base much more details. So let’s keep doing that and ask about animals this time. And the bot will say, “I just grabbed 32% of what you just said.” So can you give me more details? And I keep asking about animals. And this time the bot say, “I’m not sure I’ve understood what you just said. Are you talking about animals in the plane or other things?” And now instead of choosing between one, two and three, I will write it in full text saying, okay, I go for the first option. And by doing so, the bot will give me an answer, a very detailed one. And if, once again, I’d say, can you give me a shorter version of your answer?
The bot will take that in account and will give me a shorter answer. So that’s all for how the human contribution can improve chatbot capabilities. I will now hand on to Frederic. So he could walk you through the perspective of making an inclusive chatbot.

Frederic: Thank you Lao-dja. So about inclusive bot. Well, digital transformation and bringing new technologies, new methods and bringing new communication ways is expected to play a leading role to address the biggest developmental environmental and social challenges of the 21st century. According to the human resources’ research from Gartner, technology will drive equality for people with disabilities. By 2023 Gartner predicts that the number of people with disabilities employed will triple due to AI on emerging technologies reducing barriers to job access. CDC states disability inclusion means understanding the relationship between the way people function and how they participate in society and making sure everybody has the same opportunities to participate in every aspect of life to the best of their abilities and desires back to chatbots. You know that texting can be challenging to many people based on their morphology or due to symptoms such as the tremor, limited mobility in upper limbs.
For example, depending on age, physical trauma, disease or any efficiency, when putting together the results of our research, it will appear to us that we just came up with a set of tools and methods of bringing much easier access to the world of business to everybody. At some point, we have obviously made it clear in our team that we need to conduit a specific branch of research in order to take advantage of AI. Or bot that would lead to facilitating integration of disabled persons in the working world. Just like everybody else. So what did you see today? We provide you with explanation on how artificial intelligence and machine learning can improve chatbots’ conversational capabilities, and also explanations on how you men can improve machine learning capabilities based on the crowdsourcing method. And you could see a mini short demonstration to show what we were saying. So now I know we still have a five minutes and we will be very happy to answer to your questions if there was a question you may have.

Frederic Jacquet

Frederic Jacquet is Chief Innovation Officer within Talan, a digital company that designs and executes enterprise business transformation for a wide range of international customers. Frederic has b...
Read more

Lao-dja Songdou Tchala

Lao-dja Songdou Tchala is a Data-Scientist at Talan, a digital company that designs and executes enterprise business transformation for a wide range of international customers.In his role Lao is worki...
Read more