Constantinople isn't just a Train in Rome: Watson's error rate on Jeopardy

A lot of ink has been spilled about Watson's error in guessing "Toronto" for a U.S. city, but amusing as that goof was, it wasn't one of his worst errors at all. According to IBM scientist David Ferucci, Watson only had 14% confidence in that answer, hence the multiple question marks. Had it not been a Final Jeopardy question it was forced to answer, it never would have buzzed in.

The far more egregious errors were those that Watson had full confidence in, but was dead wrong about. In fact, given IBM's stated goal of using similar technology to aid doctors in diagnosing patients, those are the most dangerous outcomes possible. It's one thing for Watson to throw its hands up and admit to not having a clue (as it essentially did with the Toronto question), but what if Watson informed a doctor that it was 97% certain that a patient had cancer, when in fact they have no malignant cells at all? That would be far more troubling, and is analogous to Watson's 97% confidence that Latin finis is a also a word for where trains originate:

Final Frontiers (400): From the Latin for "end", this is where trains can also originate 

Correct Answer:

terminus/terminal 

Watson's Answers:

finis 97%

Constantinople 13%

Pig Latin 10%