Bot teacher that impressed and fooled everyone

An ever-present problem in teaching, especially online, is the very many queries and questions from students. In the Georgia Tech online course this was up to 10,000 per semester from a class of 350 students (300 online, 50 on campus). It’s hard to get your head round that number but Ashok Goel, the course leader, estimates that it is one year's work for a full time teacher.

The good news is that Ashok Goel is an AI guy and saw his own subject as a possible solution to this problem. If he could get a bot to handle the predictable, commonplace questions, his teaching assistants could focus on the more interesting, creative and critical questions. This is an interesting development as it brings tech back to the Socratic, conversational, dialogue model that most see as lying at the heart of teaching.

Jill Watson – fortunate error

How does it work? It all started with a mistake. Her name, Jill Watson, came from the mistaken belief that Tom Watson’s (IBM’s legendary CEO) wife was called Jill - her name was actually Jeanette. Four semesters of query data, 40,000 questions and answers, and other chat data, were uploaded and, using Bluemix (IBM’s app development environment for Watson and other software), Jill was ready to be trained. Initial efforts produced answers that were wrong, even bizarre, but with lots of training and agile software development, Jill got a lot better and was launched upon her unsuspecting students in the spring semester 2016.

Bot solution

Jill solved a serious problem – workload. But the problem is not just scale. Students ask the same questions over and over again, but in may different forms, so you need to deal with lots of variation in natural language. This lies at the heart of the chatbot' solution -  more natural, flowing, frictionless, Socratic form of dialogue with students. The database therefore, had many species of questions, categorized, and as a new question came in Jill was trained to categorise the new question and find an answer.

With such systems it sometimes gets it right, sometimes wrong. So a mirror forum was used, moderated by a human tutor. Rather than relying on memory, they added context and structure, and performance jumped to 97%. At that point they decided to remove the Mirror Forum. Interestingly, they had to put a time delay in to avoid Jill seeming too good. In opractive academics are rather slow at responding to student queries, so thay had to replicate bad performance. Interesting, that in comparing automated with human performance, it wasn;t a metter of living up to expectations but dumbing down to the human level.

These were questions about coding, timetables, file format, data usage, the sort of questions that have definite answers. Note that she has not replaced the whole teaching tasks, only made teaching and learning more efficient, scalable and cheaper. This is likely to be the primary use of chatbots in the short to medium term - tutor and learner support. That’s admirable.

Student reactions

The students admitted they couldn’t tell, even in classes run after Jill Watson’s cover was blown – it’s that good. What’s more, they like it, because they know it delivers better information, often better expressed and (importantly) faster than human tutors. Despite the name, and an undiscovered run of 3 months, the original class never twigged. Turing test passed.

In Berlin this month, I chaired Tarek Richard Besold, of the University of Bremen, who gave a fascinating talk through some of the actual dialogue between the students and Jill Watson. It was illuminating. The real tutors, who often find themselves frustrated by student queries, sometimes got slightly annoyed and tetchy, as opposed to Jill, who comes in with personal but always polite advice. This is important. Chatbots don;t get angry, annoyed, tired and irritable. They are also free from the sort of beliefs and biases that wee humans always have. They don't have that condescending, and often misplaced. rolling-of-the-eyes reaction that an academic sometimes has towards simple mistakes and werrors by novice learners. The students found her useful, the person who would remind them of dues dates and things they really needed to know, then and there, not days later. She would also ask stimulating questions during the course. She was described as an “outstanding TA” albeit “somewhat serious”.  Of course, some got a little suspicious. They were, after all, AI students.

“Her name is Watson ;)” he added. They checked LinkedIn and Facebook, where they found a real Jill Watson, who was somewhat puzzled by the attention. What finally blew her cover was interesting, she was too good. Her responses were just too fast (even though Goel had introduced a time delay), compared to other TAs. When they did discover the truth the reaction was positive “This is incredibly cool.”

A student even wanted to put her up for a teaching award. Indeed Goel has submitted Jill for just such an award to Georgia Tech.

Bot v TA

Tarek points out that the qualities students expect of a tutor are that they are honest, flexible, patient, confident, a good listener, professional, willing to share and use available resources. Sure there are many things a good teacher can do that a bot cannot but there are qualities a bot has that teachers do not possess. Tarek showed a response from a real teacher on the course, who was clearly a little tetchy and annoyed, compared to the clear and objective reply by the bot. This relentless patience and objectivity is something a good bot can deliver. Remember that the sheer scale of the questions by students was beyond the ability of the teachers to respond and as the bot is massively scalable (hence their use in MOOCs), it is intrinsically superior on this point as something is always better than nothing. It’s all a matter of finding the right balance.

Education is also expensive, scarce and difficult to scale. In classes with a hundred or more students, few get any personal attention. So can we have personal attention at scale? In the short-term we can have scale for some functions. In the long-term we can certainly forsee this sort of technology, with other advances, as yet unknown, make inroads in all aspects of teaching – subject matter knowledge, feedback, planning, content creation, content delivery, assessment. These are already possible.

Attribution of human qualities

In The Media Equation, Nass and Reeves did 35 studies to show that we have a tendency to attribute human qualities and agency to technology, especially computers and especially computers than engage us in dialogue. If bots deliver useful help, supportm answers and even deeper teaching experiences, then this is a bonus. Indeed, I think that the natural language approch through bots accelerates this willingness to attribute agance to the bot. Natural language is our normal UI. As AI provides better and better natural language processing, along with trained and smart databases of answers and smart responses, so AI will become the new AI. One could argue that learners already have, in Google, Facebook, Twitter, Amazon, Netflix and dozens of other online services, AI-driven UI. This must surely be an advantage in learning, where, the mroe frinctionless the interface, the better the outcomes.

What’s next?

The following semester they created two new bots as AI assistants (Ian & Stacey). Stacey was more conversational. This is a natural but technically difficult evolution of bots in teaching – to be more Socratic. This time the students were on the look out for bots, but even then only 50% identified Stacey and only 16% identified Ian as AI. The next semester there were four AI assistants and the whole team (including humans) used pseudonyms to avoid detection.

Jill Watson is being turned into a commercial product. Make no mistake, IBM, Apple, Microsoft and Amazon see education as a market for AI, as do Pearson and others. Bot-based teaching is with us now and will only get better, faster and more widespread. There are chatbots now teaching Englsih and other languages. Some are fully integrated with human tutors. Teacherbots allow a real tutor to deal with many more students, increasing productivity, which, I think, is the greatest prize of all. It is not the case of dispensing with teachers, but raising their game. Teachers should welcome something that takes away all the admin and pain. It's something we hear teachers complaining about all the time, so let's grasp the solution. The next stage will be bots that provide more than responses to questions and queries but also provide real tutor plans, advice and play a more subsatntive teaching role. Differ is a Danish bot that encourages student engagement - vthere will be many others with lots of different roles. Remember also, the role of bots in ro-bots. We are now seeing the emergence of real working robots that are also communicating bots in social care and education.

The idea that professionals like Doctors, Bankers, Accountants and Lawyers will be, to a degree, replaced by AI, but teachers will not, is a conceit. It has already happened and will happen a lot more. A recent Parson/Watson tool uses AI to enhance textbooks, opening up dialogue with the student, with formative assessment and personalized help. CogBooks provide adaptive, personalised learning in realtime. WildFire actually creates online learning from documents, PowerPoints, podcasts or videos. This area is moving fast.

Conclusion

What I love about this story is the fact that a Professor used his own subject and skill to improve the overall teaching and learning experience of his students. That’s admirable. With all the dystopian talk around AI, we need to make sure that AI is a force for good. If it does, as seems increasingly certain, replace many jobs, resulting in increased unemployment, we need to make sure that, like the God Shiva, AI creates as it destroys. Bringing meaning to our lives through education and learning is surely an admirable goal. AI is perhaps the only way to bring scalable, automated, personalised teaching to create that future.