Now that the Jeopardy “Man vs Machine” challenge is over, lets spend a little time evaluating what Watson did (and what it didn’t) to get a better understanding of how applications like IBM’s Watson will “Keep Pace” in the future
So what did we witness ?
Regardless of your knowledge of the underlying technology, the performance of Watson was impressive (and entertaining). Just the steps needed to answer a question in a span of approximately 3 seconds (the time to read the question) consisted of:
- Break apart the words of the “answer” into discreet pieces
- Analyze the the pieces against its database
- Identify relationships and possible answers
- Calculate “degrees of confidence”
- Make the decision to answer the question – meaning “press” the button
Very impressive to say the least. But should mankind be worried ?
What Watson did REALLY well !
Some of the things that stood out:
1. When it “knew” the answer Watson could press the button instantaneously. Several articles have stated that it took Watson exactly four milliseconds – So a key problem for the human contestants was trying to beat Watson to the punch (the contestants looked very frustrated at times with this aspect).
2. The ability to derive CORRECT answers .. QUICKLY !
Ignore the man behind the curtain ..
Now if you were watching closely, subtle flaws could be identified with Watson’s performance. Some of those flaws are based on:
- Computers have an IQ of ZERO
- Computers still can’t “think” OR reason like human beings
- Watson wasn’t designed to THINK – it was designed to play Jeopardy (and understand the games nuances)
So with that as working background, some elements to watch on the eventual rerun:
1. Short answers – If you watch day 2 of the show – somewhere around the middle of the show was a period of “answers” that were very short (i.e. a few words). It appeared as though Watson didn’t have enough information to get an answer with a high enough confidence – and this allowed the human players to catch up.
2. Reason-ability. In the first “Final Jeopardy” the category was US Cities and Watson eventually came back with an answer of “Toronto”. The response from IBM was that “Cities” was not specifically mentioned in the “Answer” – but both humans were able to tie the category and question together. Watson couldn’t
3. Limitations – Watson was programmed to answer questions based on the rules of “Jeopardy”. You would be hard pressed to make it do something else – OR – answer questions that weren’t in its database. It is likely that Watson didn’t have enough information to associate the US Cities referenced in the “answer” s to World War II battles
We can rebuild him .. Better, Stronger, Faster …
The beauty of the Watson demonstration is that the technology is at an interesting point where it may be able to apply this approach to tasks that are truly data intensive. It can evolve to be yet another tool that people use to get their jobs done. It will be VERY interesting to see what IBM does next with Watson.
But as we know – technology marches on. And right after the Jeopardy challenge, the following Wall Street Journal blog discusses something that should be BETTER than Watson and is only 4-5 years away !
To “Keep Pace” with future technologies