The Sentences Computers Can't Understand, But Humans Can

Tom Scott
Views 1 744 149
99% 17 302 139

The Winograd schema is a language test for intelligent computers. So far, they're not doing well. MORE LANGUAGE FILES: ruvid.net/group/PL96C35uN7xGLDEnHuhD7CTZES3KXFnwm0

Written with Gretchen McCulloch and Molly Ruhl. Gretchen's podcast Lingthusiasm is at lingthusiasm.com/ - and Gretchen's new book, BECAUSE INTERNET, is available:
🇺🇸 US: amzn.to/30tLpjT
🇨🇦 CA: amzn.to/2JsTYWH
🇬🇧 UK: amzn.to/31K8eRD
(Those are affiliate links that give a commission to me or Gretchen, depending on country!)
Levesque, H.J., Davis, E., and Morgenstern, L. (2011). The winograd schema challenge. In
AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning.
Trask, R. (1993). A dictionary of grammatical terms in linguistics. London ; New York: Routledge. (page 233)
Winograd, T. (1972). Understanding natural language. Cognitive Psychology, 3(1), 1-191. (page 33)
Hunston, S. (2002). Corpora in Applied Linguistics. Cambridge: Cambridge University Press.
Jurafsky, D., & Martin, J. (2009). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition (2nd ed., Prentice Hall series in artificial intelligence). Upper Saddle River, N.J.: Pearson Prentice Hall.
Gray, M. & Suri, S. (2019) Ghost work. Boston, M.A.: HMH Books.
I'm at tomscott.com
on Twitter at twitter.com/tomscott
on Facebook at facebook.com/tomscott
and on Instagram as tomscottgo

Published on


Feb 17, 2020




Loading link...

Add to:

My playlist
Watch later
Comments 80
Tom Scott
Tom Scott 5 months ago
And that's the last in this run of the Language Files! There may well be more later in the year, but for now: thank you so much to co-authors Gretchen and Molly. Pull down the description for more about Gretchen's podcast and book!
Cody Nemo
Cody Nemo 2 months ago
I actually would like to offer a major counterpoint here: The interpretation has nothing to do with language experience and could actually be very easily solved with just bruteforce data dumping and a little bit of neural network trickery. A) Context. Languages without gendered pronouns - most languages - solve the issue arising from unclear referents with context. The same can be done in english in some cases. B) Messing around a bit with multiplexing and demultiplexing - something that neural networks desperately need advancement in anyway for all sorts of applications - would allow contextual clues to be picked up easier. C) Context is also important because the natural unit of a single sentence is optimized for our human way of processing and understanding. There is a reason we dont often see sentences like Victor Hugo's famous 810 word sentence. Telephone numbers are capped at 7 digits in part because that's the most that human short term memory can handle. D) Computers are incredibly strong at lexical knowldege and with access to the internet stronger at determining the meanings of words than we ourselves are. While writing this i just said "ok google define suitcase" and it came back with a much more clear and concise definition than i could have ever given. Cross checking is something neural networks absolutely excel at already, so determining that something associated with "carrying/containing" is often importantly associated with size qualifiers such as big or small should be trivial. Meanwhile the cup shape of trophies is the only thing associated with carrying something and thus less likely that it would be associated with language talking about that. Word-space distance has been successfully used to track the development of human consciousness and prediction of mental problems. Very interesting TED talk about this btw. E) Finally if the context doesn't help, if raw data dumping doesn't help, if looking at the sentence as a whole doesn't help, and if word-space distance and using definitions as added input doesnt help....try introducing basic logic. Google had a NOT operator in its search engine for almost two decades. As someone else in the comments stated, this is not yet implemented properly in the language processing software. It is also the crux of the statement in your example. "not fit" is negating and from that point on the only thing required is an understanding of what fit means and that it can only happen if the something that fits is smaller than the container, which is a simple definition that can either be programmed in or can be distilled from text trawling.
Craft CGame
Craft CGame 2 months ago
I'm making the smartest AI that can solve those sentences.
blackfalcon1324 2 months ago
This is one thing I like about the swedish language. They have a pronoun for the subject of a sentence aswell as the object.
Heysauce Mikehere
Heysauce Mikehere 2 months ago
Matrix Mirage So you use souvenir and satchel?
Eric Saccoman
Eric Saccoman 2 months ago
Could this be solve by some how teaching context to code because the cloth and table sentence still remains ambiguous because a table could be used to protect a cloth and the same goes for the other way around
Meme4U Videos
Meme4U Videos 8 hours ago
finally, an advantage point for humans
Dalton Rachal
Dalton Rachal 9 hours ago
Update on that AI. It now uses different models, but it's enhanced massively within the past few months. The details that AI Dungeon can now keep track of, and use effectively are quite incredible.
莉莉 11 hours ago
Samuel Chen
Samuel Chen 12 hours ago
overwatch reference
Knuckles Day ago
1:48 ( ͡° ͜ʖ ͡°)
Kazoo 2 days ago
Guys i have an easy solution... Just put the computers to school
Myotis Welwitschii
That is the reason for languages like Lojban. Lojban (and similar languages) not like that because the grammar is more linear, and the problem doesn't appear at all. If humans would agree on using one of those languages it would make things a lot easier.
Traugott Müller
Traugott Müller 4 days ago
Ahh, back in the 70's when people thought "teaching computers how to identify objects on a picture is a good exercise for the semester break" and "language is clearly defined and it is easy to teach computers about language". Sometimes I feel where are only half an inch closer to "intelligent" computers then we where 50 years ago.
Twitter Videos
Twitter Videos 4 days ago
The trophy, would not fit in the brown suitcase, because it, was too big
Kurt Wagner
Kurt Wagner 6 days ago
Thats one thing that i love about the German Language. You just have much more words to describe things.
Drago from Bakugan
Imagine being gay and breaking down a computer
Josh Combs
Josh Combs 9 days ago
I don't understand computers, so someone smarter than me please explain this. Why can't we teach a computer what a suitcase and trophy are without the computer needing experience with those items? It seems all you'd have to do is create a dictionary for all nouns, but give definitions that would be useless to humans, but adequate for a computer. For instance the definition of Suitcase can be Container. The definition of Table can be Surface. Then when trying to decide which noun the "It" is in the sentence, just look at the preposition before it, if it's the word "on' or "off" then look for the noun that means surface, if it's the word "in" or "out" then look for the word that means container. If the words "on" and "off" were intended as "connected or not connected", as in "the light is off" it would recognize that there's only one noun in the sentence and the definition of "off" would change.
epic dimples123
epic dimples123 9 days ago
I can not understand the sentence in the video, oh frick I just failed the I’m a robot test... AAAAAAA
Stickman on a stick
Ok, but why aren't there any Doctor Who references down here!!?? I expected at least one!
Suzanne Potts
Suzanne Potts 9 days ago
We saw his star in the east.
Just Not
Just Not 10 days ago
4:02 you monster
Grim Affiliations
Grim Affiliations 11 days ago
We’ve got experience of suitcases and trophies Tottenham: nervous laughter
Grim Affiliations
Grim Affiliations 11 days ago
Had to do it 😂
Komatik 11 days ago
Tom's OW player examples are clearly degenerates.
Jackson Zheng
Jackson Zheng 12 days ago
GPT -3 now has a commercial lease that you can use and the studies show its text generation is better and much closer to humans than gpt2 as it has more than 100 times more parameters. Although gpt2 does require a few more tries to get the context right in some cases.
dmdz 12 days ago
I feel like training an AI to properly recognise context and syntax would have to be a grueling, manual process. In fact, you could probably crowdsource it. I'd be totally down to help in crowdsourced AI language training.
Obi-Wan Kenobi
Obi-Wan Kenobi 14 days ago
04:02 Mei mains are incarnation of pure evil.
Nofxthepirate 15 days ago
That computer response was almost accurate until "you pull out your gun"..... How about not.
Vexel 16 days ago
0:41 - “And that a trophy usually doesn’t contain things.” Do y’all not drink out of your trophies?
Nathan Zotov
Nathan Zotov 16 days ago
the brown suitcase would not fit in the trophy because nothing was too small
Yuvvraj Koparde
Yuvvraj Koparde 17 days ago
Tom: you've got experience with trophies and suitcases Me: cries in failure
Random Awesome
Random Awesome 17 days ago
That moment you go, "hmm, i usually use visual or character traits to distinguish between same pronouned characters in my gay romance fanfics" and then Tom calls you out on it as weird and amateur.
Martin Cattell
Martin Cattell 18 days ago
I'm sorry Tom ... "Larger things can't fit inside smaller things ... Take gay-romance, for instance ..." EDIT: But seriously, interesting topic.
The Freemankiller
The Freemankiller 20 days ago
Kinda funny to see how AI Dungeon aged
David Smith
David Smith 20 days ago
Replace 'AI' and 'computers' with 'kids' and you get the life of a parent
Slaughter Round
Slaughter Round 22 days ago
3:57 didn't the old english usage of main have a similar usage, meaning "strong", "strength" or "most adapt"?
Noodle Tribunal
Noodle Tribunal 23 days ago
i main kirby actually, i co-main kirby and marth
FBI 24 days ago
So your telling me my trophy CANT fit in my brown suit case
shelbyvcobra 28 days ago
We don't all have experience with trophies... Why am I so bad
Arterexius 28 days ago
The solution seems rather simple, although incredibly long winded to do. "just" define what each and every word means and run the schema on it again. It should - logically speaking - now know that a trophy won't fit inside the suitcase, if it's too big. There's lots of holes in this of course and it needs a lot of defining everything, in order to actually understand what you're trying to tell it
MrFunzeige 28 days ago
Ok dude stop I can't understand it what does it mean what are you saying
MJY Animations
MJY Animations 29 days ago
We will defeat robots after tge tech apocalypse
Slylie Zeroseven
Slylie Zeroseven 29 days ago
Imagine maining Mei
Doritos lance fresno
Just thinking of what percent of lighthing would reveal the true color of the human skin (Completely unrelated to this video)
Rafael Villca Poggian
Now GPT-3 is out with a massive number of parameters, it would be interesting to test it
pax und peace
pax und peace Month ago
This video didn't aged well.
pax und peace
pax und peace Month ago
2020 a lot of violence a lot of revolution a lot of protests.
Alex C
Alex C Month ago
The trophy(1) didn't fit in the suitcase(2) because it(1) was too big. The trophy(1) didn't fit in the suitcase(2) because it(2) was too small.
Alan Blankman
Alan Blankman Month ago
Hey Tom - You're out of date with your update mentioning GPT-2! GPT-3 is out...
Alan Blankman
Alan Blankman Month ago
BTW - "GPT-3 has a whopping 175 billion parameters. By comparison, the largest version of GPT-2 was 1.5 billion parameters, and the largest Transformer-based language model in the world - introduced by Microsoft earlier this month - is 17 billion parameters."
”Time flies like -”
kram emher
kram emher Month ago
I want to win the loottery.
Javier Albinarrate
3:38 that one was definitely an American AI, a British one would have asked if you wanted a cuppa.
MangoSeal Month ago
The Paper lines are not straight. why.
Nedko066 Month ago
GPT-3 came out just recently and it seems to be a big improvement, maybe in this regard too. They have not released yet the trained models but they have a long list of generated examples. They claim humans are fooled without cherry picking. Of course we should take it with a grain of salt until we have actual models but it is worth noting.
Irtap Month ago
Pragmatics 😍
Michael Will
Michael Will Month ago
Maybe people shouldn't be writing gay romance novels. Problem solved.
Watrmelone Month ago
I wish I had experience with trophies ;(
Zack R.
Zack R. Month ago
So basically you play the pronoun game with computers to figure out if they are smart
E030E03 Month ago
winograd schema that nobody can solve: **Rrrrrrr agagaga(1) anino hlr lea(x) anoyepa(2)&**
Beatrice Kvammen
Is this the new captcha?
DodoGaming Month ago
You toxic Overwatch player! You main Mei and Reaper -.-
arsenicCatnip Month ago
Mei mains represent
Jack Dog
Jack Dog Month ago
1:12 Isn’t it also possible to interpret this as the permit fearing violence?
Cringeyness Expressway
White Fang
White Fang Month ago
Sorry mate, I can’t bring your trophy. Why you say? Ah the suitcase is too big. I can’t put the trophy in it. 😂
Han Lockhart
Han Lockhart Month ago
So basically we need an Ai with its own body that can grow and experience the _real world_ for some years, before it`ll be any good at understanding things like a Human?
Harry Giles
Harry Giles Month ago
The blonde played with the hem of her shirt is metonymy not synecdoche. You should do a video on the difference between the two, it’s fascinating!
Ben Bruland
Ben Bruland Month ago
I am fairly certain i could write a context free grammar that would understand that trophy sentence. That doesn't seem like a huge problem, though I could be wrong.
PKMEddie Month ago
TL;DR English is a dumb language
Glider Fan
Glider Fan Month ago
Most of the examples in the video would not work in Polish. I put the cloth on the table to protect it - in Polish "it" refers to the closes preceding noun, no exception, so you protect the table with zero doubt. To protect the cloth you should say something like "to protect the cloth, I spread it on the table". Same for "the trophy would not fit in the brown suitcase because it was too big" - but the mechanism is bit different. We have grammar genres that help. "Puchar nie wejdzie do walizki, bo jest za duży" - the form "duży" refers to the trophy, suitcase would be "duża". So we can say it other way round, "Puchar nie wejdzie do walizki, bo jest za duża" - bit silly, suitcase is too big for trophy to fit in. If you need perfect fit it may be the case. This system allows us to put words almost in any order in the sentence. "Do walizki nie wejdzie puchar, bo za duży jest" = "Za duży jest puchar, nie wejdzie do walizki" = "Puchar za duży jest, do walizki nie wejdzie" etc.
LRT_ Unimog
LRT_ Unimog Month ago
4:11 Water/gas mains?
LRT_ Unimog
LRT_ Unimog Month ago
1:47 My grandma!
Michael Ellis
Michael Ellis Month ago
I got redirected to this from a game grumps doki doki literature club video... ...Yall thinking what I'm thinking?
drew Month ago
This is my new favourite channel dont know how it's taken me so long to find it
Stefano Siclari
Stefano Siclari Month ago
This reminds me of a guy I met at a party who told me that people like me (at the time I worked as a translator/interpreter) should be fired and replaced by machines, so that more money is available to be spent for the poor. On a side note, the guy's a millionaire and his car alone costs just a bit more than what my family earns in four years
Sean Murphy
Sean Murphy Month ago
3 years ago..... Seattle, USA...... Jeff Bezos - “Alexes buy me something from WholeFoods....” Alexa - “ok, I just bought you WholeFoods” not my joke.. passing it along to the 4.78% who had not heard that one!
BoggyBolt Month ago
4:02 someone's been playing too much Overwatch
This should be what Captcha is. Easy for humans, impossible for computers.
Araknala Month ago
computer checking this video: “write this down; WRITE THIS DOWN”
T Trep4
T Trep4 Month ago
Instead of changing computer programming so it can understand human languages, perhaps we should change our languages to be simpler, consistent, and have fewer rules that are full of exceptions.
Serene Haze
Serene Haze Month ago
So Tom's a Mei main then.
Finlay Wales
Finlay Wales Month ago
Everyone else: meaningful comments Me: Ah, so he's a Mei main. Interesting.
Jordan Ferrazza
Jordan Ferrazza Month ago
I noticed Talk to Transformer changed the API it was using and stopped advertising the API (probably due to the other one being proprietary). I think it was changed to this one.
William W. Campbell-Shepherd IX
I don't think "sentient" computers are a threat of any kind, let alone possible. Any thoughts on this?
lordsm1ley Month ago
*You've been reading fanfiction, haven't you?*
Next videos
The Console Wars
Views 1 550 429