IBM "Watson" Demonstration on Jeopardy is a Fraud

Originally posted by grey580
reply to post by Hemisphere

 

What people don't seem to understand is that this thing is programmed by people.
And people make mistakes.
So ergo it will make mistakes too.

The question arises then. Can it transcend it's programming and learn from it's mistake.

I for one understand the deal. Mistakes, i.e. the "leg" answer. The media is used to sell products and ideas. I question to some extent all that I view. Why would I not question a poorly explained hiccup in a computers' programming that shows up in what is blatantly a 30 minute commercial posing as entertainment? I’m not ignoring the programming explanation, I am simply not buying it in this instance for the plausible reasons stated earlier.

My overall opinion about Watson: unimpressed.

Viewing it as another player, it appears it gets to the buzzer quicker. Ol' Jennings had to rapid-fire the plunger and worry about the answer later. If Watson knew the answer (ie, the color of his top answer was green), then he was basically given the opportunity to answer first.

Also, there weren't any play-on-words categories or the like, "before and after" would have been another good one. Those would have baffled Watson.

Even if a human did win, I don't think it should even have been as close as it was.

The whole thing seems like a semi-sham. I do believe many of us were expecting a little more.

There is no possible way a computer can fail at trivia..it simply can't unless programmed to fail.

The only advantage humans have over robots would be answering before the question was completed, or if the input (voice recognition) was nerfed to give falty information to begin with.

If there was a mistake given by the computer, then yes..I call hoax. Computers do not make mistakes. either the information was there, or it wasn't. computers don't guess incorrectly....especially to that degree (there are likelyhood algorithms that could be at play)

Originally posted by Hemisphere

Originally posted by Arbitrageur

Originally posted by _Del_
thenumerati.net...

Read the entire thread next time. Thanks for the duplicate info. Glad you buy the IBM line. It is after all a half-hour commercial.

You tell me to read the thread and the link was already posted which I acknowledged, but I still wonder if you read the link when you make posts like this:

Originally posted by Hemisphere
Don't you think that the term "U.S." would have or should have been explored prior to such a demonstration? Have you ever watched this program? Categories with "U.S." preceeding other terms are extremely common. "U.S. Presidents", "U.S. History", "U.S. Navy" and so on. Of all idioms to flubb on, "U.S."? If this was not intended it shows extreme incompetence.

The link I reposted for you explained that it was because of examining previous shows that they chose to largely ignore the category:

Category titles cannot be trusted. I blogged about this earlier, in a post How Watson Thinks. It has learned through exhaustive statistical analysis that many clues do not jibe with categories. A category about US novelists, for example, can ask about J.D. Salinger's masterpiece. Catcher in the Rye is a novel, not a novelist! These things happen time and again, and Watson notices. So it pays scant attention to the categories.

Even if they did test the U.S. if they then decided to ignore it because it's a category heading, how will recognizing it help when they ignore the category anyway?

Originally posted by Hemisphere
Those "Torontos" should have been ignored as they don't fit with the other caveats. I have yet to see IBM state that Watson meant Toronto, Iowa as the correct answer. Mark Twain once referred to Berlin as "The Chicago of Europe". That doesn't make Berlin a viable answer for a super computer to come up with.

I don't think you understand how Watson "reads" or understands the questions. I'm not at all trying to be argumentative; it just seems like you're a bit confused on how the process works.

Originally posted by Arbitrageur
The link I reposted for you explained that it was because of examining previous shows that they chose to largely ignore the category:

Category titles cannot be trusted. I blogged about this earlier, in a post How Watson Thinks. It has learned through exhaustive statistical analysis that many clues do not jibe with categories. A category about US novelists, for example, can ask about J.D. Salinger's masterpiece. Catcher in the Rye is a novel, not a novelist! These things happen time and again, and Watson notices. So it pays scant attention to the categories.

Even if they did test the U.S. if they then decided to ignore it because it's a category heading, how will recognizing it help when they ignore the category anyway?

edit on 16-2-2011 by Arbitrageur because: clarification

In this case they chose wrong. "Many clues do not jibe with categories" and many do jibe. Now what? The Day 2 final category title directly influenced the possible correct answers. "US novelists" also narrows down the possible correct answers even if the answer in the form of a question is not the author's name. A computer would have a huge advantage for a final question in the category "US Novelists". If that category were given to the computer while going to the final commercial break, tens of thousands of novelists could be accessed with every known work cross referenced for all possible tie ins for three minutes by the computer whereas a very well read human could come up with how many names? 100? De-stressing the category is a mistake. More to the point, de-stressing "US" in the Day 2 title was a glaring mistake. To the point where I think it was suspicious. Furthermore, the final wagers in Jeopardy are made on the basis of having only the final category title and the running totals of you and your opponents.

Watson's round two wager for the final question was just $947 coinciding with a question worded using idioms that confused the computer especially while mostly disregarding the information provided in the category title. Or so we are told. Round three the wager went to $17,973 with a question that contained all necessary information for the computer without needing the category title. The convenient change in the wagering pattern is suspect. If Watson hung on another idiom filled final question and Jennings had went all in on the final instead of wagering $1000, the chance was there for a Jennings win. Why would the computer risk a loss on the final wager, possibly ruining this three day 90 minute IBM commercial? The answer is there was no risk. This was fixed. If Watson wagered in the range of $16,000, it could not be beat. Watson wagered conveniently conservatively on the idiomatically phrased question. On the next show it wagers recklessly with the possibility of losing on a similar idiomatic stumper. It would not be allowed to happen in my opinion. Betting logarithms? I vote television dramatics.

The winner here won a million dollar charity donation, not their running dollar total. In reality a $1,000,000 donation for IBM? They could scrounge that from the glove box of their CEO's car. We all know that the money is meaningless here. This is all about the commercial. That's where the big money resides. If the computer did not show vulnerability on Day 2, Day 3 is an afterthought. Slightly vulnerable increases interest and can be quickly explained and overcome, losing however is unsellable and would be extremely difficult for IBM to recover from. There was no chance of that in my opinion.

Originally posted by _Del_

Originally posted by Hemisphere
Those "Torontos" should have been ignored as they don't fit with the other caveats. I have yet to see IBM state that Watson meant Toronto, Iowa as the correct answer. Mark Twain once referred to Berlin as "The Chicago of Europe". That doesn't make Berlin a viable answer for a super computer to come up with.

I don't think you understand how Watson "reads" or understands the questions. I'm not at all trying to be argumentative; it just seems like you're a bit confused on how the process works.

Me neither, I get it, it de-stresses the information in the category. The background information for O'hare and Midway Airports are contained on Wiki and Wiki is listed as one of the sources for base information fed to Watson. It totally clammed on the second airport due to the wording of the question. I'm not confused. It did not mean Toronto, Iowa when it answered "What is Toronto????????" It meant Toronto, Canada due to being confused by Billy Bishop Airport in Toronto, named after World War I flying ace: William Avery "Billy" Bishop. As opposed to O'hare in Chicago named after Lieutenant Commander Edward "Butch" O'Hare a World War II flying ace.

Explain to me _Del_ if you will, how did Watson miss "World War II hero" in the question?

Here is the actual wording of the Day 2 final once again:

Its largest airport is named for a World War II hero; its second largest, for a World War II battle

Everybody thinks the anti-Christ will be a human.

$10 says we start having AI play a public role in government in under 10 years.

Originally posted by alyoshablue
Everybody thinks the anti-Christ will be a human.

$10 says we start having AI play a public role in government in under 10 years.

That's an interesting comment. Watching these shows and knowing these were essentially long commercials, I wondered who or what could afford to buy such powerful computers? I could imagine only two possibilities. Only the very largest corporations, far larger than IBM and......... governments.

Originally posted by Hemisphere

The winner here won a million dollar charity donation, not their running dollar total. In reality a $1,000,000 donation for IBM?

A $1,000,000 donation to IBM's choice of charities. Not a charitable donation to IBM of course. Sorry if I worded that unclearly. I need to consult my programmers. Need idiomatic updates.

Originally posted by SaturnFX
There is no possible way a computer can fail at trivia..it simply can't unless programmed to fail.

What? That's a broad statement and just not true. I think you're assuming that it will 100% correctly interpret all sentences and know 100% of all answers. The difficult part is speech recognition and natural language processing. Most likey when Alex asked the question the S from US and the soft-c from city combined so the computer translated it: What you city... It probably threw out the extra word (you) and interpreted it as "What city..."

Originally posted by Gamma MO
Paraphrasing:

What US City's major airport is named after a WWII hero. Its second largest airport is named after a WWII battle.

First of all, Toronto is not in the US, last I checked.

Maybe Watson has some undisclosed news on NAFTA.

I chuckled at the televised verison of WATSON since it's old news...

WATSON and LC Chips

Either or its still a trip that a computer program can do that.

Thursday, February 16, 2011 on WABC, AM Radio NYC on the "Imus In The Morning" program.
The following in reference to the Watson demonstration on Jeopardy:

"This was a sleazy promotion for IBM." - Don Imus

Short and sweet. From the mouth of one of radio's all-time sleazy promoters.

Originally posted by goatfish

Originally posted by SaturnFX
There is no possible way a computer can fail at trivia..it simply can't unless programmed to fail.

What? That's a broad statement and just not true. I think you're assuming that it will 100% correctly interpret all sentences and know 100% of all answers. The difficult part is speech recognition and natural language processing. Most likey when Alex asked the question the S from US and the soft-c from city combined so the computer translated it: What you city... It probably threw out the extra word (you) and interpreted it as "What city..."

You've got that wrong. The computer does not hear. This is also why the computer gave the same wrong answer as Jennings on Day 1, if I am not mistaken. The question is typed into Watson as Alex is saying it for the human contestants. As some have told me over and over, the computer does not put much weight on the information provided within the category title.

Originally posted by Ferris.Bueller.II

Originally posted by Gamma MO
Paraphrasing:

What US City's major airport is named after a WWII hero. Its second largest airport is named after a WWII battle.

First of all, Toronto is not in the US, last I checked.

Maybe Watson has some undisclosed news on NAFTA.

Oh man that's funny! How the heck did that go by here on ATS for three pages.

I've only seen part one, am downloading the show from The Pirate Bay on Bit Torrent, (hint, hint, those who can't watch it on TV so missed it all) as i'm UK, part 2 tonight, 3 tomorrow, the Nova show about Watson is upped by TV Team as well for any outside USA if interested.

But did anybody notice, Watson got an easy question wrong in the first round, then as though flustered in human terms, went very wrong in 3 subsequent questions and didn't even buzz in, i mean way out in the possible permutes, when it should have got all 3 correct, like the 'puter lost confidence all of a sudden and had to regain it's composure, again in Human terms.

It's by no means the most powerful or even the smartest mainframe created, it's just a large server, there's super cooled 'puters with much more power, but, the ability to discern the context of questions which have mutable meanings as mentioned, like glass being a drinking vessel as much as a window, etc, as we Humans assign context to reference in 100th's of a second, most of us most of the time anyway, is an advance, indeed.

So if this large server mainframe, which is all it is after all, can do this now, what about quantum AI prototypes etc and military grade mainframes, ay.

Cyberdyne systems, not too far away, ay.
en.wikipedia.org...





Peace.

reply to post by DeltaPan


Thanks for the tips on the vids DP! I think that some categories, as hard as it is to believe, are/were blindspots for Watson. Precious few categories though. Whether an entire category was worded more idiomatically I can't say. I missed Day 1. There also might be holes in the data base. They will correct that off of the show performance I would think.

reply to post by Hemisphere


Indeed Hemisphere, no worries.
Bit Torrent is great, i only see wrong in downloading cam versions of new films still at the box office and brand new music or current version 'puter applications etc, that does take from primary profit uptake of creators and traditional outlets etc, but TV shows shouldn't be any concern to anti piracy, nor films which have been released to video for a while etc, it's mainly wanton greed that motivates so much dissent against downloading via bit Torrent, not our fault they are greedy oiks. IMHO.
[TV Team or EZTV on TPB, eztv.it or newtorrents.info, great for so much TV material.]

Won't say too much as aware many outside US won't have seen it, mind you, UK MSM already mentioned who won, in this case, worse than a twit spoiling PPV wrestling results before i've seen it [Thanks I.T.N.!], lucky i like analysing the bout mechanics etc so enjoy even if i know results, not just about who wins, but how.

But second show, Watson was back on top and the 2x Human contestants, were not happy bunnies by the end man, looked like they were seething in the closing frames lols.

Like you say, i think Watson ran a sort of diagnosis due to getting the question wrong and may have concluded the algorithms used when the question was answered wrongly were at fault so omitted them in the subsequent questions, then realised it was leaving it disadvantaged so reintroduced those algorithms it set aside, so began answering correctly again, in the first show.

That's part of how this mainframe works, it's self learning, and like a child, if something seems not to work and that child gets an answer wrong in class, the child will discount a thought process that led to error in an academic environment and may not want to think that way due to fear of failure, all sub mind processes, but that's how our little super computers tend to work but we've all those pesky psychological dynamics and emotional reactions thrown in and in children, the teacher and/or parent has to encourage with positive reinforcement to get thought processes reengaged after error or such may return much later of own accord after the mind essentially says, what was so then may not be so now so try that way again, we all do that to greater or lesser degree and a great many times during our formative and educational years, and beyond, being Humans prone to error.

That with a child obviously has emotions and psychology tied up with it, fear of failure because wanting to please teacher and/or parents, this is a machine, but take out emotion and psychology, the fact it's programmed to win, and has learning heuristics, like a child, it reacted by omitting various processes, then concluded that itself was an error because it led to greater disadvantage.

My opinion on it anyhows.