IBM "Watson" Demonstration on Jeopardy is a Fraud

edited for snarkiness

Originally posted by Hemisphere
There will be no link. I saw this in real time. Tonight on Jeopardy the final question category was "U.S. Cities". Both human contestants got the answer Chicago correctly. The IBM computer answered "Toronto?????????" This answer and no challenge by Trabek? No grumbling in the audience? Likely the audience was filled with IBM employees. Just a few chuckles. Things that make you go...... WTF?

This had to be a programmed "miss". Why? You tell me. Trying to make it seem fallible? So as not to seem so sinister? This thing has a level of computation and control over vast systems and corporations never before seen. Hundreds of thousands more jobs will be lost when computers of this level enter the corporate market place. Not very appetizing an advertisement in the current job market. Just my opinion.

"Don't believe anything you hear and only half of what you see."

There're a lot of cities named Toronto.

Originally posted by Hemisphere
reply to post by DeltaPan

 

Thanks for the tips on the vids DP! I think that some categories, as hard as it is to believe, are/were blindspots for Watson. Precious few categories though. Whether an entire category was worded more idiomatically I can't say. I missed Day 1. There also might be holes in the data base. They will correct that off of the show performance I would think.

The category only serves as a guide. If you watch some of the commentaries on Watson on youtube you will see an IBM guy mention how categories are only one part of the deep analytics.

Watson wouldn't hit the buzzer if he wasn't confident. Yet, when you look at the times whne Watson didn't win, a lot of times he was at least 50% sure and had the correct answer too. What i've read tells me that Watson won't alwasy hit the buzzer if he's over 50% on an answer.

Not only that, but several times he took half a second too long to compute an answer. With more computing power he would have won those questions and buzzed in quicker than the others.

Btw, Watson was 14% for Toronto and his next guess was Chicago at 11%.

If it had been a normal question Watson would not have hit the buzzer.

And humans... I want to give us some applause. Bret and Ken did well for us. Not only that, they showed us how much computing power goes on in the brain that humans aren't aware of. When we're consciously looking for answer we're largely unaware of the unconscious processing our brain is doing to give us answers. It's remarkable. And you know.. Watson used 90 IBM Power 750 servers and a couple refrigeration units and probably took up room about 10 times the size of a refrigerator. Yet Ken and Bret did it with something the size of a human head using very low power requirements.

I also found out that Ken Jennings and others like him can answer Jeapordy questions 67% of the time. Watson was only able to be competitive with this in 2010. Even in 2007 he only got 15% right.

The machines will eventualy prevail, though. It'll be a strange world when the data is too complex for humans to comprehend timely and we'll have to depend on computers to answer our questions. We won't be able to live without them. All of the benefits will be from the computers answering questions we cannot but we'll be unable to preserve the benefits without the computers...

Look at this:
news.yahoo.com...

Some of that future is already here. According to news reports, IBM was to announce Thursday plans to work with Columbia University and the University of Maryland to create a diagnostic version of Watson for physicians, with voice recognition. The service could be launched by the end of next year. There are also discussions about a version of Watson that could help customers make decisions about consumer electronics.

If Watson's software was computed on a normal PC it would take over 2 hours to compute an answer. This doesn't seem long to me. 11 to 12 doublings for 2 second response.

This is a 'puter!!!

Think this will outdo Watson with no stress what so ever...

www.extremetech.com...#

Computer's by IBM as well.

Not only that, but several times he took half a second too long to compute an answer. With more computing power he would have won those questions and buzzed in quicker than the others.

That's an important point...the REAL determiner of Jeopardy is the buzzer.
For anyone who's ever played a similar game (like on a cruise ship, etc.), knowing the answer is only part of it. You have to be the first on the buzzer too. THIS is the real factor. Jennings always did so well because he had both, exceptionally quick on the buzzer, and knew the answers. I actually worked with a guy who was on the show up against Jennings. In fact, he did very well (and his end amount would have been enough to win any other night). He commented to me about the buzzer thing, and I got the same experience when recently on a cruise quiz show. I knew the answers, but it was ALL about the buzzer and who buzzed in first.

Originally posted by jonnywhite

And humans... I want to give us some applause. Bret and Ken did well for us. Not only that, they showed us how much computing power goes on in the brain that humans aren't aware of. When we're consciously looking for answer we're largely unaware of the unconscious processing our brain is doing to give us answers. It's remarkable. And you know.. Watson used 90 IBM Power 750 servers and a couple refrigeration units and probably took up room about 10 times the size of a refrigerator. Yet Ken and Bret did it with something the size of a human head using very low power requirements.

I also found out that Ken Jennings and others like him can answer Jeapordy questions 67% of the time. Watson was only able to be competitive with this in 2010. Even in 2007 he only got 15% right.

The machines will eventualy prevail, though. It'll be a strange world when the data is too complex for humans to comprehend timely and we'll have to depend on computers to answer our questions. We won't be able to live without them. All of the benefits will be from the computers answering questions we cannot but we'll be unable to preserve the benefits without the computers...

Look at this:
news.yahoo.com...

Some of that future is already here. According to news reports, IBM was to announce Thursday plans to work with Columbia University and the University of Maryland to create a diagnostic version of Watson for physicians, with voice recognition. The service could be launched by the end of next year. There are also discussions about a version of Watson that could help customers make decisions about consumer electronics.

Thanks jonny, some good points. Yes the two men did well but keep in mind their advantages. They are able to cross-reference idiomatically where the computer can't. That's part of why the computer downgrades the category titles. The other advantage I noticed is that in some instances I think that Bret was focusing on the buzzer and not the question. He ran into a category he was confident in and didn't bother listening to the question and focused on the buzzer timing. He then read the question quickly and answered. You will notice that he comes close to taking too much time on some of those. That's my opinion on that, I don't know for certain. In any case, for the humans this is a demonstration of mostly useless retention. Impressive? Yes. Some measure of intellect? Yes. Useful beyond Jeopardy? Questionable. Extremely questionable with the advent of PCs and the Internet. What do they know that could not be looked up? That someone could answer correctly in the medicine or physicist category would not necessarily make them candidates to be doctors or physicists.

This Jeopardy thing is just a demonstration/commercial. When IBM takes this into the medical field much of the idiomatic language difficulties will be eliminated. How? The doctors will be trained in how to input the information so as not to be idiomatic. It’s very easy and it's already been done with computers answering telephones. The computer will instruct you on how to answer diagnostic questions. If your answer is not understood the computer asks more questions to pin you down. Sometimes the computer will voice a scenario to you and ask you if that is what you meant and you reply with a "yes or no". This and variations of this are already in use as you likely know.

reply to post by Hemisphere


I agree it would have to be a fraud. Those kinds of questions a computer should whip the crap out of a human, easily, with one Gig of RAM tied behind its back.

Its not strategy, or anything that requires intelligence. Jeopardy is a contest of memory. Pure memorization of facts. And NO human can out perform a computer there.

Edit to add;

Well I take that back. The wording of the questions, that clever little allusion stuff they do, that would favor humans. Ok. Maybe its not a fraud.

Originally posted by Gazrok

Not only that, but several times he took half a second too long to compute an answer. With more computing power he would have won those questions and buzzed in quicker than the others.

That's an important point...the REAL determiner of Jeopardy is the buzzer.
For anyone who's ever played a similar game (like on a cruise ship, etc.), knowing the answer is only part of it. You have to be the first on the buzzer too. THIS is the real factor. Jennings always did so well because he had both, exceptionally quick on the buzzer, and knew the answers. I actually worked with a guy who was on the show up against Jennings. In fact, he did very well (and his end amount would have been enough to win any other night). He commented to me about the buzzer thing, and I got the same experience when recently on a cruise quiz show. I knew the answers, but it was ALL about the buzzer and who buzzed in first.

Yes. And it was likely when Ken knew the answer long before the question was finished being read. I know there are times when I blurt out the answer with only a couple of words read. I've done it from another room in the house, where I can't see the written question. My wife swears I cheat. In these instances where Ken had near instant recognition, he could ignore the end of the question and concentrate on the buzzer timing. There are often a short list of answers that fit a category. There are times when I can list a number of correct responses before anything is revealed. I bet you've all done that. And so knowing that short list would give you a buzzer advantage. I think this happens all the time. Not so much for Watson as he de-emphasizes the category.

Originally posted by Illusionsaregrander
reply to post by Hemisphere

 

I agree it would have to be a fraud. Those kinds of questions a computer should whip the crap out of a human, easily, with one Gig of RAM tied behind its back.

Its not strategy, or anything that requires intelligence. Jeopardy is a contest of memory. Pure memorization of facts. And NO human can out perform a computer there.

Edit to add;

Well I take that back. The wording of the questions, that clever little allusion stuff they do, that would favor humans. Ok. Maybe its not a fraud.

Illusions, that is the precisely the quandary that caused this thread. The computer has the bulk memory advantage. The humans know the subtlety of language. This must be lived to be fully understood and used to advantage. Everybody knew each others limitations going in. Yet the computer wagered in a pattern that would have had it beaten up in the back room of any Vegas Casino. If you catch my drift. The fraud is that this has been largely ignored and conveniently explained away by wagering logarithms. The computer wagered just a little with no knowledge it would be skunked on a final category. The following night the computer wagered recklessly having been skunked on an idiomatic question the previous night when a modest wager or even no wager at all would have caused a win. The dramatics this caused were not ignored by me. What nobody will discuss is that this in fact was a three day 90 minute commercial for IBM. Far be it from a corporation fixing the outcome of a commercial. If this did not turn out positively for the computer, what would the payoff for IBM be? An embarrassing and costly set-back the likes never seen from a paid advertisement.

Keep in mind I explained earlier that the language problem would be eliminated by retraining humans to communicate more clearly with the IBM computer in any future endeavors. This was not a side project for kicks or the betterment of mankind by IBM. This was the heart of an ad campaign.

Originally posted by Gazrok

Not only that, but several times he took half a second too long to compute an answer. With more computing power he would have won those questions and buzzed in quicker than the others.

That's an important point...the REAL determiner of Jeopardy is the buzzer.
For anyone who's ever played a similar game (like on a cruise ship, etc.), knowing the answer is only part of it. You have to be the first on the buzzer too. THIS is the real factor. Jennings always did so well because he had both, exceptionally quick on the buzzer, and knew the answers. I actually worked with a guy who was on the show up against Jennings. In fact, he did very well (and his end amount would have been enough to win any other night). He commented to me about the buzzer thing, and I got the same experience when recently on a cruise quiz show. I knew the answers, but it was ALL about the buzzer and who buzzed in first.

Good points you make here, but this really wasn't what I was explaining. Watson actually took half a second or more too long. This isn't about hitting the buzzer within a 1 microsecond response time. This is about Watson taking too long to compute an answer and then hitting the buzzer 1 microsecond later but still losing.

There's no question Watson would always have a better response time, though. Probably much better than even a microsecond (1/1000th of a second). But without an answer, it doesn't matter.

I see far too much skepticism here. Yes, we should be skeptical because IBM is for-profit. But they didn't make any money from Watson's earnings. Yes, this is an ad campaign for the technology that underlies Watson. But there's actually technology underneath it all. This isn't as simple as some people here are claiming. Walking seems simple but it's incredibly difficult to master it on a computer. Believe it or not, but Watson uses machine learning and many techniques we're learned over the years in AI and natural language processing. This is not something that's as simple as a database query because computers have always been masters of simple lookups or simple calculations. What Watson did will be a part of history because what he did wasn't simple.

Watson answered natural language questions with a similar hit/miss ratio to humans. People claiming that Watson cannot think idiomatically are missing the point. Watson is programmed to learn and can form 'ideas'. They're not like ours, but they can change over time and that's the purpose of them grilling Watson over the past several years with staged jeapordy competitions to prepare him for the real thing. Yes, human programmers influence his learning and change some things like his confidence levels and various other things but you're skipping the part where Watson has his own ideas that can change - that's very important not to miss. Ultimately, Watson is a collection of his own ideas and the ideas of his developers. If he can fake it and still win jeapordy competitions or pass a turing test, then it doesn't matter whether it's faked or not. What matters is his success rate.

Don't be surprised if in 10 years you're talking to computers for real. You had lots of warning. If you want to worry about it don't wait 10 years. Worry about it now before companies start using it to bring down costs and people are unemployed and back in school to find work that keeps getting replaced by machines and computers. Can we keep ahead of the progress by inventing new jobs?

Misplaced skepticism can be deadlier than a rampaging robot in new york.

BUT a better competition would be one that allows a set time, 3 seconds, to answer a question. Response time is not considered (pressing the buzzer). Instead the players would be scored on their answers.

Originally posted by jonnywhite
I see far too much skepticism here. Yes, we should be skeptical because IBM is for-profit. But they didn't make any money from Watson's earnings. Yes, this is an ad campaign for the technology that underlies Watson.

That is what they're selling my friend. The winnings? A million? That's lunch for the IBM execs. We're talking selling Watson to the AIG size corporations and nations too. We're talking potentially billions if not trillions tied to this demonstration. They do nothing out of the goodness of their hearts.

I actually got a couple of questions correct that Watson didn't. They did however, give you the multiple choice thing that they usually don't do, so that definitely helped.

Originally posted by maybee
I actually got a couple of questions correct that Watson didn't. They did however, give you the multiple choice thing that they usually don't do, so that definitely helped.

These were not among the easiest of Jeopardy categories maybee. Anyone that answered correctly did very well in my opinion. That however was not a multiple choice screen. That was a read-out of possible responses that Watson was considering. The upper most was his "best bet" percentage wise. The other two answers being the 2nd and 3rd most likely choices based on its computations. Often the correct answer did not so much as make the computer's top 3.

Originally posted by MetalCoffeeL
reply to post by CosmicCitizen

 

Toronto's airport is named Pearson International.

That's only one of Toronto's airports. The reason Watson picked Toronto was because of Billy Bishop Toronto City Airport and the fact that Bishop's name is almost synonymous with 'World War flying ace' and 'World War hero'.

The only issue here is that the computer didn't correctly eliminate all cities outside the US (as it should have). Based on the algorithm as it was presented in the NOVA special, Watson simply picked Toronto because of the overwhelming frequency of Bishop's name associated with World War and being a war hero. It is interesting, however, that Bishop was a WWI hero, rather than WWII, as the question specifically called for, but still, given the context it's not overly surprising that Watson picked Toronto if it wasn't going to eliminate non-US cities.

Originally posted by jonnywhite
There're a lot of cities named Toronto.

Oh, come on!
How many of those cities have a 'major' airport?
How many of those cities have two airports?
How many of those cities biggest airport is named for a war hero?
How many of those cities' second airport is named after a wwii battle.
Just searching for a second airport named for a wwii battle would eliminate all of them.
I haven't done it, but I'm a gambling man.

Chicago's 1st = O'hare
Chicago's 2nd = Midway

If any of those other Toronto's have a major airport named after a wwii hero and a second airport named after a battle then watson was correct, but I doubt any Toronto meets those requirements which should have been easy for any simple program to answer, no less an AI machine. Shame on IBM!

I'd laugh if someplace like Toronto, Ohio actually had two small airports at one time in its history that would also fit the question. Something as small as a grass field airstrip may be somewhere in the database.

Might even be a more obscure WWII Hero. Could be an airstrip named after somebody with a Purple Heart or something like that which would be hard to narrow down. Or perhaps something more generic like "Veteran's Field", which many towns have. And many could agree that WWII heroes were also veterans.

This would call for more research to see if there's a valid reason as to why the computer came up with such an answer.

Anyone willing to look over the list of U.S. cities named Toronto?

I haven't read the comments yet, I just can't wait to say that this thread is compelling. Surely, either the correct information is in the database with the appropriate contexts, or it's not.
Basically, I would guess that a computer would give the correct answer or no answer at all. I would guess that it would be impossible for a computer to give a wrong answer, unless it had been loaded with incorrect data.

Maybe I'm way out of date and my assumptions on how something like this Watson would work are entirely wrong...

A compelling thread indeed... I’ll go read the pages of comments now…

Well, I've finished reading all the comments.

I see I grossly underestimated the achievement that the Watson system represents. Good grief, it's spooky to say the least. Now with just a little bit of refining, it's going to be a system that can answer questions better than most people...
But what about after quite a lot of refinement? It's going to be able to answer questions better than trained professionals. It seems inevitable also, that the limitations of speech recognition will be eroded over a short period of time.
As a commenter in this thread already alluded to, this will be like that cheese playing system. It got better and better until it one year it won against the world’s best and then it far surpassed any and all human cheese players by an enormous margin. Surely, Watson is history repeating.


Editing to add:
There's a book by Isaac Asimov called "The Complete Robot" and this thread is reminding me of some of the ideas Isaac had about the role a "thinking machine" might have in society. The scenario that comes to mind is illustrated well in the short stories "Escape!" and "The Evitable Conflict". Massively powerful computers with highly advance AI (not just brilliant logic, proper AI) are used essentially to answer questions. In "The Evitable Conflict" society only followed the courses as directed in the answers supplied by such machines and therefore society was willingly completely under the control of the machines.