My robot understands me

Case

My robot understands me

VU Amsterdam researchers Elly Konijn and Piek Vossen see an enormous potential for robots helping lonely elderly people and children in the classroom. The biggest challenge is: the speech technology. Social sciences, linguistics and artificial intelligence come together in an NWO project aimed at vastly improving that.

Robots and old people are a superb match. Back in 2015, communication researchers from VU Amsterdam showed that in the documentary Ik ben Alice [I am Alice]. A simple conversation gave three lonely ladies genuine pleasure. They quickly found it quite normal to thumb through their photo album with the robot.

Language is a very personal

However, what Alice said was sometimes inspired via the laptop of one of the researchers. The biggest problem in the interaction between people and robots is the speech technology. Professor of Media Psychology Elly Konijn: ‘The intelligibility is sometimes an issue, certainly in Dutch. This software is nearly always in English. And we mainly understand language in context, in how you know each other. We need to develop the software in such a way that the robot knows who it has in front of it.’

The way in which we are sitting here now talking to each other involves so many different aspects
- Piek Vossen

Therefore Konijn and her colleague Johan Hoorn are now collaborating with Piek Vossen, Professor of Computational Lexicology at the Faculty of Humanities. For many years, his holy grail has been a computer that genuinely understands language. ‘We’re still a long way off from that’, he says. ‘The way in which we are sitting here now, talking to each other, involves so many different aspects. We know a lot about the world and about each other, and I know quite well what you will and won’t understand. Language is very personal. I not only understand what spectacles are, but what your spectacles are as well.’

Siri is the best, but understands nothing

But even if we put that to one side, merely acquiring a language has proven to be far more complex than we thought. Vossen: ‘We now know that language cannot simply be reduced to a system of rules. Trying to teach a computer language in that manner was an empirical failure. We do not even really know what a noun is.’

The absolute best in speech technology at present is Siri from Apple, says Vossen. Google is based on the same technology. You can ask it anything and the answers are pretty accurate, but it understands nothing. Vossen is trying to change that, and Konijn is examining which elements of the dialogue you actually need to feel a human relationship. And, conversely, how a relationship helps to understand language.

Did you know? Leolani is the Hawaiian word for “voice from heaven”

Leolani: voice from heaven

Now that programming with rules has failed, the world of artificial intelligence is trying a different approach to teach computers to understand language. Vossen: ‘The latest hype is the neural network. Humans as programmers have now largely been removed and machine learning software teaches itself a language by processing lots of texts.’

Vossen has developed software that talks with you in a natural language, and which wants to learn about the world and its conversation partner and therefore asks questions. What are "speculaas"? Who is your family? If the information obtained conflicts with previous information or if it does not know something for certain, it will point this out and ask for clarification. When you say horse do you mean the chess piece or the animal? This software is called Leolani, which is the Hawaiian word for “voice from heaven”. Vossen, Hoorn and Konijn want to train Leolani via conversations in such a way that the conversations become increasingly better.

In my computer, I already have twenty brains
- Piek Vossen

Leolani is a collection of components: a camera, a microphone and a speech generator − in other words, the eyes, ears and mouth − in addition to software that processes the incoming signals. Just like our brain, that software consists of different areas. It “thinks”: I hear something. Is it a person or is it a filter coffee machine? I also see a person. So it could be the case that the person is saying something. However, what that person says does not agree with what I already know or it elicits other questions. That is reflection. Furthermore, Leolani is motivated to fill in uncertainties and information gaps. It therefore poses questions. You say this, but somebody else told me something else. So what’s right?

Leolani talks via Pepper

You can connect Leolani to whatever robot you want, so that you don’t have to talk to a computer. Robot Pepper had already been purchased in 2017 and it is therefore the embodiment to practice with. Vossen, Hoorn and Konijn are now investigating how you can train Leolani in such a way that the conversation becomes increasingly more natural and it builds up a better relationship with you.

Robot Pepper

‘In my computer, I already have twenty brains’, says Vossen. Each brain is a collection of data from previously inputted knowledge and conversations the researcher X or Y held with Leolani in his or her office. Those conversations were held according to a scenario in which the robot knew a lot or a little about somebody in advance, did or did not know where it was, knew more or less objects, et cetera.

Alice meets Leolani

Hoorn and Konijn observe how realistic the person found the conversation, whether he or she had the feeling that the robot genuinely understood them, or that the robot perhaps asked too many questions and allowed to few moments of silence. Konijn: ‘We do that in the media lab, but also at a care home. We visit people with robot Alice 2.0 that is equipped with combined software from Leolani and from Alice.’

Robot Alice visiting an elderly woman

Konijn is struck by how willing elderly people are to participate in the research. ‘For them it is a welcome distraction. That they’re supposedly not interested in new technology is definitely a prejudice. You know what we hear from them? “Why has it taken you so long to come up with this?”’

By Rianne Lindhout
Read the full version of this article in Ad Valvas [in Dutch].

More information

Piek Vossen, Elly Konijn and Johan Hoorn from VU Amsterdam are doing research into better conversations and relationships with robots within the NWO project Communicating with and Relating to Social Robots: Alice Meets Leolani.

Computer as fully fledged co-author

Software that learns by working with people is called hybrid intelligence. This is a very promising direction within artificial intelligence. A Dutch research project about this led by VU Amsterdam received a Gravitation grant worth 19 million euros from NWO. In 2030, the team wants to publish the first scientific article with a computer as a fully fledged co-author. Ethics and privacy will also receive attention in this project. The principal researcher is Frank Harmelen, professor and computer scientist at VU Amsterdam in the Department of Artificial Intelligence. Piek Vossen is also involved in this project.