1 00:00:00,680 --> 00:00:02,560 Tom Mitchell: Welcome to machine learning. 2 00:00:02,560 --> 00:00:03,919 Tom Mitchell: How did we get here? 3 00:00:03,919 --> 00:00:05,799 Tom Mitchell: I'm Tom Mitchell. 4 00:00:05,799 --> 00:00:08,339 Tom Mitchell: Today's episode is an interview 5 00:00:08,339 --> 00:00:10,320 Tom Mitchell: with Geoff Hinton, one of the 6 00:00:10,320 --> 00:00:13,519 Tom Mitchell: pioneers in the field of neural 7 00:00:13,519 --> 00:00:14,919 Tom Mitchell: network learning. 8 00:00:14,919 --> 00:00:18,660 Tom Mitchell: Geoff started out early, as you'll hear, in the seventies, 9 00:00:18,660 --> 00:00:22,199 Tom Mitchell: nineteen seventies, and has continued working in neural 10 00:00:22,199 --> 00:00:24,519 Tom Mitchell: networks ever since. 11 00:00:24,519 --> 00:00:26,000 Tom Mitchell: During the period of the 12 00:00:26,000 --> 00:00:28,280 Tom Mitchell: nineteen nineties and early two 13 00:00:28,280 --> 00:00:30,480 Tom Mitchell: thousand, when neural networks 14 00:00:30,480 --> 00:00:33,039 Tom Mitchell: were really in disfavor in the 15 00:00:33,039 --> 00:00:35,119 Tom Mitchell: field of machine learning, Geoff 16 00:00:35,119 --> 00:00:38,460 Tom Mitchell: nevertheless persisted, and he 17 00:00:38,460 --> 00:00:41,600 Tom Mitchell: co-led the triumphant return of 18 00:00:41,600 --> 00:00:43,159 Tom Mitchell: neural networks in the form of 19 00:00:43,159 --> 00:00:46,500 Tom Mitchell: deep networks in the twenty ten 20 00:00:46,500 --> 00:00:48,600 Tom Mitchell: ish period. 21 00:00:48,600 --> 00:00:53,840 Tom Mitchell: In twenty eighteen, Geoff, along with Joshua Bengio and Yann 22 00:00:53,840 --> 00:00:58,600 Tom Mitchell: LeCun, received the Turing Award in Computer Science. 23 00:00:58,600 --> 00:01:00,899 Tom Mitchell: That's the highest award given 24 00:01:00,899 --> 00:01:03,259 Tom Mitchell: in the field of computer science 25 00:01:03,259 --> 00:01:06,359 Tom Mitchell: to researchers in twenty twenty 26 00:01:06,359 --> 00:01:07,180 Tom Mitchell: four. 27 00:01:07,180 --> 00:01:09,719 Tom Mitchell: Jeff, along with John Hopcroft, 28 00:01:09,719 --> 00:01:12,439 Tom Mitchell: were awarded the twenty twenty 29 00:01:12,439 --> 00:01:15,859 Tom Mitchell: four Nobel Prize in Physics for 30 00:01:15,859 --> 00:01:18,540 Tom Mitchell: their work on artificial neural 31 00:01:18,540 --> 00:01:20,019 Tom Mitchell: networks. 32 00:01:20,019 --> 00:01:22,019 Tom Mitchell: I hope you enjoy the episode. 33 00:01:37,420 --> 00:01:41,140 Tom Mitchell: I'm pleased to have with me today Geoff Hinton, one of the 34 00:01:41,140 --> 00:01:43,659 Tom Mitchell: pioneers of machine learning. 35 00:01:43,659 --> 00:01:45,540 Tom Mitchell: Geoff, great to see you again. 36 00:01:45,700 --> 00:01:47,019 Geoffrey Hinton: Thanks for inviting me. 37 00:01:47,379 --> 00:01:52,579 Tom Mitchell: What I'd like to do today is get two things. 38 00:01:52,579 --> 00:01:54,659 Tom Mitchell: Two types of things from you. 39 00:01:54,659 --> 00:01:59,400 Tom Mitchell: One is your own personal history and how you got into this field 40 00:01:59,400 --> 00:02:02,159 Tom Mitchell: and what happened after you did. 41 00:02:02,159 --> 00:02:06,640 Tom Mitchell: And the second is kind of your perspective on the whole field 42 00:02:06,640 --> 00:02:11,280 Tom Mitchell: of machine learning, AI, and how things are turning out. 43 00:02:11,120 --> 00:02:13,599 Geoffrey Hinton: So when I was in high school, I had a very smart friend who was 44 00:02:13,599 --> 00:02:18,080 Geoffrey Hinton: a very good mathematician and read widely, unlike me. 45 00:02:18,080 --> 00:02:20,520 Geoffrey Hinton: And he came into school one day 46 00:02:20,520 --> 00:02:22,039 Geoffrey Hinton: and talked about how memories 47 00:02:22,039 --> 00:02:23,400 Geoffrey Hinton: might be distributed over the 48 00:02:23,400 --> 00:02:24,800 Geoffrey Hinton: brain rather than localized in a 49 00:02:24,800 --> 00:02:26,840 Geoffrey Hinton: place like a hologram, because 50 00:02:26,840 --> 00:02:27,759 Geoffrey Hinton: this would have been nineteen 51 00:02:27,759 --> 00:02:29,240 Geoffrey Hinton: sixty six and holograms had just 52 00:02:29,240 --> 00:02:30,199 Geoffrey Hinton: come out. 53 00:02:30,199 --> 00:02:32,680 Geoffrey Hinton: And that got me interested in 54 00:02:32,680 --> 00:02:34,000 Geoffrey Hinton: how our memories represented in 55 00:02:34,000 --> 00:02:35,400 Geoffrey Hinton: the brain. 56 00:02:35,400 --> 00:02:37,960 Geoffrey Hinton: And I've been interested in that ever since. 57 00:02:39,639 --> 00:02:43,680 Tom Mitchell: Now, when I met you, we were both at Carnegie Mellon. 58 00:02:43,680 --> 00:02:46,500 Tom Mitchell: It was nineteen eighty six when, 59 00:02:46,500 --> 00:02:48,240 Tom Mitchell: uh, we really got to do some 60 00:02:48,240 --> 00:02:49,919 Tom Mitchell: work together or teach a course 61 00:02:49,919 --> 00:02:50,759 Tom Mitchell: together. 62 00:02:50,759 --> 00:02:51,400 Tom Mitchell: What? 63 00:02:51,400 --> 00:02:53,039 Tom Mitchell: How did you get from nineteen 64 00:02:53,039 --> 00:02:55,120 Tom Mitchell: sixty six up till Eighteen 65 00:02:55,120 --> 00:02:55,939 Tom Mitchell: eighty six. 66 00:02:55,939 --> 00:02:58,219 Tom Mitchell: What was the path? 67 00:02:58,780 --> 00:02:59,780 Geoffrey Hinton: Slightly 68 00:02:59,780 --> 00:03:03,159 Geoffrey Hinton: rocky. So I went to 69 00:03:03,159 --> 00:03:07,340 Geoffrey Hinton: university. I studied physics, chemistry and physiology, and in 70 00:03:07,340 --> 00:03:11,460 Geoffrey Hinton: physiology. In the last term, they're going to teach us, um, how the 71 00:03:11,460 --> 00:03:12,419 Geoffrey Hinton: central nervous system 72 00:03:12,419 --> 00:03:13,780 Geoffrey Hinton: worked. And I was very 73 00:03:13,780 --> 00:03:17,900 Geoffrey Hinton: excited. And they taught us how action potentials are conducted 74 00:03:17,900 --> 00:03:20,740 Geoffrey Hinton: along an axon, which wasn't what I meant by how it 75 00:03:20,740 --> 00:03:23,300 Geoffrey Hinton: worked. And so I switched to 76 00:03:23,300 --> 00:03:26,219 Geoffrey Hinton: philosophy. That was even less 77 00:03:26,219 --> 00:03:28,060 Geoffrey Hinton: useful. And then I switched 78 00:03:28,060 --> 00:03:29,819 Geoffrey Hinton: to psychology, which was 79 00:03:29,819 --> 00:03:30,300 Geoffrey Hinton: completely 80 00:03:30,300 --> 00:03:32,219 Geoffrey Hinton: hopeless. Um, and then I became a 81 00:03:32,219 --> 00:03:36,020 Geoffrey Hinton: carpenter. And after I'd been a 82 00:03:36,020 --> 00:03:38,520 Geoffrey Hinton: carpenter for about nine months, I met 83 00:03:38,520 --> 00:03:38,840 Geoffrey Hinton: a 84 00:03:38,840 --> 00:03:42,020 Geoffrey Hinton: carpenter. And he was so much better than me, I decided it'd be easier 85 00:03:42,020 --> 00:03:42,379 Geoffrey Hinton: to be an 86 00:03:42,379 --> 00:03:46,180 Geoffrey Hinton: academic. Um, so I went to graduate 87 00:03:46,180 --> 00:03:48,259 Geoffrey Hinton: school in Edinburgh, um, 88 00:03:48,259 --> 00:03:49,520 Geoffrey Hinton: with Longuet-higgins, who 89 00:03:49,520 --> 00:03:50,979 Geoffrey Hinton: had published interesting stuff 90 00:03:50,979 --> 00:03:53,759 Geoffrey Hinton: on, um, using neural nets for 91 00:03:53,759 --> 00:03:53,919 Geoffrey Hinton: a 92 00:03:53,919 --> 00:03:59,360 Geoffrey Hinton: memory. Unfortunately, around the time I arrived, Winograd's thesis 93 00:03:59,360 --> 00:04:02,759 Geoffrey Hinton: came out and he switched his allegiance to symbolic AI 94 00:04:02,759 --> 00:04:03,800 Geoffrey Hinton: and gave up on neural 95 00:04:03,800 --> 00:04:07,280 Geoffrey Hinton: nets. And so I spent five years as 96 00:04:07,280 --> 00:04:08,960 Geoffrey Hinton: his graduate student with 97 00:04:08,960 --> 00:04:10,159 Geoffrey Hinton: him, trying to persuade me to give 98 00:04:10,159 --> 00:04:10,639 Geoffrey Hinton: up neural 99 00:04:10,639 --> 00:04:12,280 Geoffrey Hinton: nets. And he never 100 00:04:12,280 --> 00:04:16,439 Geoffrey Hinton: succeeded. Um, in the end, he was very helpful to 101 00:04:16,439 --> 00:04:18,980 Geoffrey Hinton: me. But for a long time, there was a lot of argument about how 102 00:04:18,980 --> 00:04:21,279 Geoffrey Hinton: I should really be doing symbolic AI, and all this neural 103 00:04:21,279 --> 00:04:22,079 Geoffrey Hinton: net stuff was complete 104 00:04:22,079 --> 00:04:24,319 Geoffrey Hinton: nonsense. And everybody else in 105 00:04:24,319 --> 00:04:26,019 Geoffrey Hinton: Edinburgh believed that neural nets 106 00:04:26,019 --> 00:04:26,199 Geoffrey Hinton: were 107 00:04:26,199 --> 00:04:29,600 Geoffrey Hinton: nonsense. Um, we actually a couple of 108 00:04:29,600 --> 00:04:32,480 Geoffrey Hinton: exceptions. There was a post-doc called David Willshaw who'd 109 00:04:32,480 --> 00:04:35,199 Geoffrey Hinton: done associative memory, and he basically done something 110 00:04:35,199 --> 00:04:36,000 Geoffrey Hinton: quite like Hopfield 111 00:04:36,000 --> 00:04:37,459 Geoffrey Hinton: nets. But a long time before 112 00:04:37,459 --> 00:04:41,360 Geoffrey Hinton: Hopfield and Aaron Sloman was a 113 00:04:41,360 --> 00:04:42,680 Geoffrey Hinton: visitor for a while, and he was 114 00:04:42,680 --> 00:04:42,879 Geoffrey Hinton: more 115 00:04:42,879 --> 00:04:47,360 Geoffrey Hinton: sympathetic. Um, but basically they all knew it was 116 00:04:47,360 --> 00:04:50,639 Geoffrey Hinton: rubbish. And they would explain to me 117 00:04:50,639 --> 00:04:52,470 Geoffrey Hinton: how neural nets can't even 118 00:04:52,470 --> 00:04:52,610 Geoffrey Hinton: do 119 00:04:52,610 --> 00:04:56,089 Geoffrey Hinton: recursion. So because everything, 120 00:04:56,089 --> 00:04:58,189 Geoffrey Hinton: everybody believed in recursion, then, 121 00:04:58,189 --> 00:05:00,930 Geoffrey Hinton: um, I actually figured out how to 122 00:05:00,930 --> 00:05:02,009 Geoffrey Hinton: do true recursion in a 123 00:05:02,009 --> 00:05:04,889 Geoffrey Hinton: neural network and implemented it on 124 00:05:04,889 --> 00:05:05,370 Geoffrey Hinton: a machine 125 00:05:05,370 --> 00:05:07,910 Geoffrey Hinton: with. I think by then it had one hundred and ninety two 126 00:05:07,910 --> 00:05:11,050 Geoffrey Hinton: kilobytes of memory, and it was only shared by forty 127 00:05:11,050 --> 00:05:14,410 Geoffrey Hinton: people. Um, but it had a huge disk that had two 128 00:05:14,410 --> 00:05:15,970 Geoffrey Hinton: megabytes. So you never ran out of your 129 00:05:15,970 --> 00:05:18,250 Geoffrey Hinton: memory? Um, because you used virtual 130 00:05:18,250 --> 00:05:20,449 Geoffrey Hinton: memory. And I actually implemented 131 00:05:20,449 --> 00:05:23,389 Geoffrey Hinton: a little neural net that did 132 00:05:23,389 --> 00:05:23,569 Geoffrey Hinton: true 133 00:05:23,569 --> 00:05:29,569 Geoffrey Hinton: recursion. That is, in the recursive call, it used the same neurons and 134 00:05:29,569 --> 00:05:32,769 Geoffrey Hinton: the same connection strings for the recursive call as it did for 135 00:05:32,769 --> 00:05:33,290 Geoffrey Hinton: the high level 136 00:05:33,290 --> 00:05:38,470 Geoffrey Hinton: call. Now, if you do that, of course it had to offload all 137 00:05:38,470 --> 00:05:41,889 Geoffrey Hinton: the parameters of the high level call into some short term 138 00:05:41,889 --> 00:05:43,569 Geoffrey Hinton: memory onto a 139 00:05:43,569 --> 00:05:45,170 Geoffrey Hinton: stack. Eventually, and I figured 140 00:05:45,170 --> 00:05:46,410 Geoffrey Hinton: out how to implement a stack 141 00:05:46,410 --> 00:05:48,050 Geoffrey Hinton: with associative memory in a 142 00:05:48,050 --> 00:05:48,410 Geoffrey Hinton: neural 143 00:05:48,410 --> 00:05:52,350 Geoffrey Hinton: net. Um, so I had this little neural net running that was doing 144 00:05:52,350 --> 00:05:56,189 Geoffrey Hinton: full recursion in neural nets, and that was the first talk I 145 00:05:56,189 --> 00:05:58,029 Geoffrey Hinton: gave. And people were very 146 00:05:58,029 --> 00:06:01,670 Geoffrey Hinton: puzzled. They said, why would you want to do recursion in a neural 147 00:06:01,670 --> 00:06:05,490 Geoffrey Hinton: net? I mean, it's so easy to do in pop two, which was our our 148 00:06:05,490 --> 00:06:11,910 Geoffrey Hinton: sort of, uh, unfortunate bastard child of Pascal and 149 00:06:11,910 --> 00:06:15,589 Geoffrey Hinton: Lisp. Um, although I don't think Pascal existed 150 00:06:15,589 --> 00:06:17,009 Geoffrey Hinton: then. Um, 151 00:06:17,009 --> 00:06:18,990 Geoffrey Hinton: so. 152 00:06:18,990 --> 00:06:22,209 Geoffrey Hinton: Yeah. So I keep meaning to go back to 153 00:06:22,629 --> 00:06:24,629 Tom Mitchell: I was going to ask, is there a 154 00:06:24,629 --> 00:06:26,509 Tom Mitchell: future to recursion for neural 155 00:06:26,509 --> 00:06:27,189 Tom Mitchell: nets? 156 00:06:27,829 --> 00:06:28,670 Geoffrey Hinton: Oh, yes. 157 00:06:28,670 --> 00:06:31,189 Geoffrey Hinton: I mean, to do true recursion, 158 00:06:31,189 --> 00:06:32,389 Geoffrey Hinton: you have to use the same neurons 159 00:06:32,389 --> 00:06:33,490 Geoffrey Hinton: and weights for the recursive 160 00:06:33,490 --> 00:06:34,069 Geoffrey Hinton: call. 161 00:06:34,069 --> 00:06:36,389 Geoffrey Hinton: That means you have to have a stack, something like a stack, 162 00:06:36,389 --> 00:06:38,949 Geoffrey Hinton: to store the parameters of the high level call. 163 00:06:38,949 --> 00:06:40,990 Geoffrey Hinton: That all works if you have fast weights. 164 00:06:40,990 --> 00:06:42,670 Geoffrey Hinton: So that was the first thing I did with fast weights in 165 00:06:42,670 --> 00:06:44,470 Geoffrey Hinton: nineteen seventy three. 166 00:06:44,470 --> 00:06:47,029 Geoffrey Hinton: I should say fast weights were invented by Schmidhuber in 167 00:06:47,029 --> 00:06:48,269 Geoffrey Hinton: nineteen ninety, something. 168 00:06:50,449 --> 00:06:51,370 Tom Mitchell: Fair enough. 169 00:06:51,370 --> 00:06:56,569 Tom Mitchell: Okay, so then, um, you moved on from Edinburgh. 170 00:06:56,569 --> 00:06:58,470 Tom Mitchell: Did you come directly to 171 00:06:58,470 --> 00:07:00,269 Tom Mitchell: Carnegie Mellon from there, or 172 00:07:00,269 --> 00:07:01,009 Tom Mitchell: how did. 173 00:07:00,850 --> 00:07:05,170 Geoffrey Hinton: Oh, no. No. Um, I dropped out 174 00:07:05,170 --> 00:07:07,569 Geoffrey Hinton: again after I finished my 175 00:07:07,569 --> 00:07:08,610 Geoffrey Hinton: thesis. 176 00:07:08,610 --> 00:07:10,250 Geoffrey Hinton: I dropped out and became a 177 00:07:10,250 --> 00:07:11,529 Geoffrey Hinton: teacher in a free school in 178 00:07:11,529 --> 00:07:12,569 Geoffrey Hinton: London. 179 00:07:12,569 --> 00:07:15,529 Geoffrey Hinton: Um, it was voluntary, I was unpaid. 180 00:07:15,529 --> 00:07:19,410 Geoffrey Hinton: They were rough, emotionally disturbed inner city kids. 181 00:07:19,410 --> 00:07:20,529 Geoffrey Hinton: And after a few months of that, 182 00:07:20,529 --> 00:07:22,009 Geoffrey Hinton: I again decided academia might 183 00:07:22,009 --> 00:07:23,129 Geoffrey Hinton: be easier. 184 00:07:23,129 --> 00:07:28,410 Geoffrey Hinton: Um, so I went back to a post-doc with Aaron Sloman in Sussex. 185 00:07:28,410 --> 00:07:33,610 Geoffrey Hinton: Higgins had moved from Edinburgh to Sussex and um, as I was 186 00:07:33,610 --> 00:07:36,810 Geoffrey Hinton: finishing my PhD, I got a post-doc with Aaron Sloman. 187 00:07:36,810 --> 00:07:40,850 Geoffrey Hinton: Um, and there were no proper faculty jobs in Britain. 188 00:07:40,850 --> 00:07:42,889 Geoffrey Hinton: Then there was one job in the 189 00:07:42,889 --> 00:07:43,970 Geoffrey Hinton: whole of Britain which Alan 190 00:07:43,970 --> 00:07:49,310 Geoffrey Hinton: Bundy got, um, and so I applied 191 00:07:49,310 --> 00:07:52,889 Geoffrey Hinton: for jobs in the States, and I 192 00:07:52,889 --> 00:07:56,329 Geoffrey Hinton: got a job as a postdoc in UCSD, 193 00:07:56,329 --> 00:07:57,790 Geoffrey Hinton: um, with Don Norman and Dave 194 00:07:57,790 --> 00:07:59,149 Geoffrey Hinton: Rumelhart. 195 00:07:59,149 --> 00:08:02,230 Geoffrey Hinton: And I really got along very well with Dave Rumelhart, and that 196 00:08:02,230 --> 00:08:03,550 Geoffrey Hinton: made a huge difference. 197 00:08:03,550 --> 00:08:08,029 Geoffrey Hinton: So I moved from a country where it was a sort of small country, 198 00:08:08,029 --> 00:08:13,430 Geoffrey Hinton: Britain, and there was only room for one ideology, and the 199 00:08:13,430 --> 00:08:17,589 Geoffrey Hinton: ideology was symbolic AI and neural nets was just rubbish. 200 00:08:18,709 --> 00:08:22,189 Geoffrey Hinton: And I moved to the states where, um, on the West coast, on the 201 00:08:22,189 --> 00:08:24,930 Geoffrey Hinton: East Coast, it was symbolic AI but on the West Coast they were 202 00:08:24,930 --> 00:08:26,230 Geoffrey Hinton: kind of more open. 203 00:08:26,230 --> 00:08:27,310 Geoffrey Hinton: And in particular, Don Norman 204 00:08:27,310 --> 00:08:28,529 Geoffrey Hinton: and Dave Rumelhart thought 205 00:08:28,529 --> 00:08:29,550 Geoffrey Hinton: neural nets were worth 206 00:08:29,550 --> 00:08:30,790 Geoffrey Hinton: considering. 207 00:08:30,790 --> 00:08:33,909 Geoffrey Hinton: Um, so it was a huge liberation 208 00:08:33,909 --> 00:08:36,309 Geoffrey Hinton: to be in a place where neural 209 00:08:36,309 --> 00:08:37,570 Geoffrey Hinton: nets were regarded as not 210 00:08:37,570 --> 00:08:38,870 Geoffrey Hinton: obvious nonsense. 211 00:08:40,350 --> 00:08:41,110 Geoffrey Hinton: And. 212 00:08:43,149 --> 00:08:44,850 Geoffrey Hinton: Um, I went there. 213 00:08:44,850 --> 00:08:48,769 Geoffrey Hinton: I, I got to meet Terry Sinofsky, who I invited to a conference. 214 00:08:48,769 --> 00:08:51,490 Geoffrey Hinton: And we'd been sort of lifelong friends and collaborators. 215 00:08:51,490 --> 00:08:55,409 Geoffrey Hinton: Um, I got to meet Francis Crick later on, who was there. 216 00:08:55,409 --> 00:08:56,610 Geoffrey Hinton: So I was there for a couple of years. 217 00:08:56,610 --> 00:08:57,529 Geoffrey Hinton: And then I got a job in 218 00:08:57,529 --> 00:08:59,250 Geoffrey Hinton: Cambridge in the Applied 219 00:08:59,250 --> 00:09:02,110 Geoffrey Hinton: Psychology Research Unit, um, 220 00:09:02,110 --> 00:09:03,129 Geoffrey Hinton: where I was meant to do applied 221 00:09:03,129 --> 00:09:04,769 Geoffrey Hinton: psychology. 222 00:09:04,769 --> 00:09:08,370 Geoffrey Hinton: And I was strongly reminded of William James's comment about 223 00:09:08,370 --> 00:09:12,409 Geoffrey Hinton: applied psychology, which is to do applied psychology, you have 224 00:09:12,409 --> 00:09:14,210 Geoffrey Hinton: to have something to apply. 225 00:09:14,210 --> 00:09:19,009 Geoffrey Hinton: Um. But I actually did some interesting stuff. 226 00:09:19,009 --> 00:09:21,049 Geoffrey Hinton: It was just around the time some 227 00:09:21,049 --> 00:09:23,470 Geoffrey Hinton: workstations were coming out, 228 00:09:23,470 --> 00:09:26,549 Geoffrey Hinton: and they had a contract with the 229 00:09:26,549 --> 00:09:29,370 Geoffrey Hinton: British Telephone Company to 230 00:09:29,370 --> 00:09:31,330 Geoffrey Hinton: help with network management and 231 00:09:31,330 --> 00:09:32,330 Geoffrey Hinton: network management. 232 00:09:32,330 --> 00:09:34,490 Geoffrey Hinton: Then was all done by hand. 233 00:09:35,490 --> 00:09:38,190 Geoffrey Hinton: And you had information about, 234 00:09:38,190 --> 00:09:39,850 Geoffrey Hinton: um, the loads on various 235 00:09:39,850 --> 00:09:41,929 Geoffrey Hinton: switching centers, and the 236 00:09:41,929 --> 00:09:44,230 Geoffrey Hinton: information was on a huge wall 237 00:09:44,230 --> 00:09:46,990 Geoffrey Hinton: that was twenty feet high and 238 00:09:46,990 --> 00:09:48,470 Geoffrey Hinton: worked like you sometimes see at 239 00:09:48,470 --> 00:09:49,669 Geoffrey Hinton: train stations. 240 00:09:49,669 --> 00:09:52,549 Geoffrey Hinton: It was little flaps with white letters on that come down. 241 00:09:52,549 --> 00:09:54,990 Geoffrey Hinton: They sort of rotate around until you get the right flap. 242 00:09:54,990 --> 00:09:58,730 Geoffrey Hinton: And so you could see all these numbers, um, that said how busy 243 00:09:58,730 --> 00:10:00,909 Geoffrey Hinton: each switching station was. 244 00:10:00,909 --> 00:10:04,350 Geoffrey Hinton: Um, and I figured a sun 245 00:10:04,350 --> 00:10:05,590 Geoffrey Hinton: workstation could do that and 246 00:10:05,590 --> 00:10:08,309 Geoffrey Hinton: would be a lot cheaper, and the 247 00:10:08,309 --> 00:10:09,590 Geoffrey Hinton: resolution wasn't that good 248 00:10:09,590 --> 00:10:10,389 Geoffrey Hinton: then. 249 00:10:10,389 --> 00:10:13,409 Geoffrey Hinton: So I had to figure out if you could display the states of all 250 00:10:13,409 --> 00:10:16,110 Geoffrey Hinton: the switching stations in Britain on the screen of a sun 251 00:10:16,110 --> 00:10:20,309 Geoffrey Hinton: workstation, and you couldn't type the names, but if you had 252 00:10:20,309 --> 00:10:24,750 Geoffrey Hinton: two letters for each name, you could get two letters there. 253 00:10:24,750 --> 00:10:28,509 Geoffrey Hinton: And I worked on a display with 254 00:10:28,509 --> 00:10:30,389 Geoffrey Hinton: two letter names, and there were 255 00:10:30,389 --> 00:10:31,470 Geoffrey Hinton: a large number of switching 256 00:10:31,470 --> 00:10:32,470 Geoffrey Hinton: stations. 257 00:10:32,470 --> 00:10:33,710 Geoffrey Hinton: There were hundreds. 258 00:10:33,710 --> 00:10:34,950 Geoffrey Hinton: And the question was could could 259 00:10:34,950 --> 00:10:36,429 Geoffrey Hinton: an operator remember which they 260 00:10:36,429 --> 00:10:37,190 Geoffrey Hinton: were? 261 00:10:37,190 --> 00:10:38,490 Geoffrey Hinton: So I actually taught myself to 262 00:10:38,490 --> 00:10:39,710 Geoffrey Hinton: remember all those two letter 263 00:10:39,710 --> 00:10:41,210 Geoffrey Hinton: names. 264 00:10:41,210 --> 00:10:45,169 Geoffrey Hinton: Um, they were in very small type to fit on, and I actually got a 265 00:10:45,169 --> 00:10:47,370 Geoffrey Hinton: serious migraine from looking at it too long. 266 00:10:47,370 --> 00:10:52,809 Geoffrey Hinton: Um, that was my interaction with human factors, and then I. 267 00:10:53,690 --> 00:10:58,169 Tom Mitchell: It does remind me of that was around the time that Unix was 268 00:10:58,169 --> 00:11:02,690 Tom Mitchell: getting invented, and all the commands had no vowels in them. 269 00:11:02,690 --> 00:11:04,289 Tom Mitchell: So there was a theme there? 270 00:11:04,610 --> 00:11:08,730 Geoffrey Hinton: Yes. Um, so I wrote a report on 271 00:11:08,730 --> 00:11:09,769 Geoffrey Hinton: it and they said it was a very 272 00:11:09,769 --> 00:11:10,169 Geoffrey Hinton: nice report. 273 00:11:10,169 --> 00:11:11,450 Geoffrey Hinton: Thank you very much. 274 00:11:11,450 --> 00:11:12,169 Geoffrey Hinton: And they weren't going to 275 00:11:12,169 --> 00:11:13,970 Geoffrey Hinton: implement it, even though it 276 00:11:13,970 --> 00:11:14,649 Geoffrey Hinton: would have been much more 277 00:11:14,649 --> 00:11:15,610 Geoffrey Hinton: efficient and much easier to 278 00:11:15,610 --> 00:11:16,330 Geoffrey Hinton: update. 279 00:11:16,330 --> 00:11:18,250 Geoffrey Hinton: And I said, why not? 280 00:11:18,250 --> 00:11:21,490 Geoffrey Hinton: And they confidentially explained to me that, well, when 281 00:11:21,490 --> 00:11:26,090 Geoffrey Hinton: people come and visit the network control center, um, or 282 00:11:26,090 --> 00:11:28,090 Geoffrey Hinton: when actually when they visit the headquarters of British 283 00:11:28,090 --> 00:11:31,370 Geoffrey Hinton: Telecom, um, they have to have something to show them, like the 284 00:11:31,370 --> 00:11:34,210 Geoffrey Hinton: politicians have to see something, and they would always 285 00:11:34,210 --> 00:11:37,529 Geoffrey Hinton: show them this huge wall that displayed the state of all the 286 00:11:37,529 --> 00:11:40,629 Geoffrey Hinton: networks, of all the switching stations, and they were very 287 00:11:40,629 --> 00:11:41,750 Geoffrey Hinton: impressed by that. 288 00:11:41,750 --> 00:11:43,029 Geoffrey Hinton: And if they got rid of the huge 289 00:11:43,029 --> 00:11:45,230 Geoffrey Hinton: wall and had just some 290 00:11:45,230 --> 00:11:46,789 Geoffrey Hinton: workstations, they were very 291 00:11:46,789 --> 00:11:48,870 Geoffrey Hinton: worried that network management 292 00:11:48,870 --> 00:11:49,830 Geoffrey Hinton: would get less funds from 293 00:11:49,830 --> 00:11:50,909 Geoffrey Hinton: British Telecom. 294 00:11:50,909 --> 00:11:53,710 Geoffrey Hinton: So they were going to keep their huge wall. 295 00:11:53,710 --> 00:11:56,070 Geoffrey Hinton: I learned a lot then about applied research. 296 00:11:57,629 --> 00:11:59,070 Geoffrey Hinton: It's not about whether it works, 297 00:11:59,070 --> 00:12:00,350 Geoffrey Hinton: it's whether about the company 298 00:12:00,350 --> 00:12:01,269 Geoffrey Hinton: likes it. 299 00:12:03,429 --> 00:12:04,990 Tom Mitchell: Fair enough, fair enough. 300 00:12:05,389 --> 00:12:11,870 Geoffrey Hinton: Then after that, um, I went back to sea to San Diego for six 301 00:12:11,870 --> 00:12:15,950 Geoffrey Hinton: months, and that's when we worked on the PDP books with 302 00:12:15,950 --> 00:12:18,350 Geoffrey Hinton: Dave Rumelhart and McClelland. 303 00:12:18,350 --> 00:12:20,750 Geoffrey Hinton: Um, I was one of the authors 304 00:12:20,750 --> 00:12:22,350 Geoffrey Hinton: until almost when they were 305 00:12:22,350 --> 00:12:22,750 Geoffrey Hinton: published. 306 00:12:22,750 --> 00:12:23,909 Geoffrey Hinton: And the last minute I dropped 307 00:12:23,909 --> 00:12:25,309 Geoffrey Hinton: out because at that point I 308 00:12:25,309 --> 00:12:26,629 Geoffrey Hinton: decided Boltzmann machines were 309 00:12:26,629 --> 00:12:27,629 Geoffrey Hinton: the future. 310 00:12:27,629 --> 00:12:30,370 Geoffrey Hinton: Boltzmann machines was just a much better idea than back prop, 311 00:12:30,370 --> 00:12:32,110 Geoffrey Hinton: and back prop was a silly idea. 312 00:12:32,110 --> 00:12:34,070 Geoffrey Hinton: Boltzmann machines were a much better idea. 313 00:12:34,070 --> 00:12:36,470 Geoffrey Hinton: Um, and there was no point being an author of a book where the 314 00:12:36,470 --> 00:12:39,850 Geoffrey Hinton: main thing was, um, back prop. 315 00:12:39,850 --> 00:12:42,210 Geoffrey Hinton: Uh, that was a mistake. 316 00:12:42,210 --> 00:12:49,610 Geoffrey Hinton: Um. But in nineteen eighty four, I figured out. 317 00:12:49,610 --> 00:12:53,730 Geoffrey Hinton: Yeah, in nineteen eighty two, then I applied to CMU, and 318 00:12:53,730 --> 00:12:57,350 Geoffrey Hinton: because it was a private university, um, you could just 319 00:12:57,350 --> 00:12:59,690 Geoffrey Hinton: they didn't have to sort of advertise very widely. 320 00:12:59,690 --> 00:13:05,889 Geoffrey Hinton: Um, and Scott Fahlman was sort of my, um, host. 321 00:13:05,889 --> 00:13:07,809 Geoffrey Hinton: He sort of interacted with him 322 00:13:07,809 --> 00:13:09,610 Geoffrey Hinton: at many workshops, and we got 323 00:13:09,610 --> 00:13:11,330 Geoffrey Hinton: along well, and he pushed hard 324 00:13:11,330 --> 00:13:12,629 Geoffrey Hinton: to get for me to get them to go 325 00:13:12,629 --> 00:13:13,450 Geoffrey Hinton: there. 326 00:13:13,450 --> 00:13:15,049 Geoffrey Hinton: And I had a very funny interview. 327 00:13:15,049 --> 00:13:16,610 Geoffrey Hinton: So I went there. 328 00:13:16,610 --> 00:13:18,490 Geoffrey Hinton: On the first day there, I gave a 329 00:13:18,490 --> 00:13:20,289 Geoffrey Hinton: talk into computer science, and 330 00:13:20,289 --> 00:13:21,570 Geoffrey Hinton: then Scott Farmer took me out 331 00:13:21,570 --> 00:13:24,649 Geoffrey Hinton: for lunch at a place it might 332 00:13:24,649 --> 00:13:25,769 Geoffrey Hinton: have been called the oh, I can't 333 00:13:25,769 --> 00:13:27,769 Geoffrey Hinton: remember what it was called, but 334 00:13:27,769 --> 00:13:29,250 Geoffrey Hinton: it had a motto, which is if you 335 00:13:29,250 --> 00:13:30,610 Geoffrey Hinton: don't get sick, you got a bad 336 00:13:30,610 --> 00:13:31,450 Geoffrey Hinton: one. 337 00:13:31,450 --> 00:13:35,049 Geoffrey Hinton: Um, and I got terribly sick. 338 00:13:35,049 --> 00:13:36,870 Geoffrey Hinton: And the next day I had acute diarrhea. 339 00:13:36,870 --> 00:13:37,710 Geoffrey Hinton: I couldn't eat anything. 340 00:13:37,710 --> 00:13:41,029 Geoffrey Hinton: I was living on coffee and Coca-Cola. 341 00:13:41,029 --> 00:13:48,230 Geoffrey Hinton: Um, I gave a talk in psychology, um, about mental imagery and my 342 00:13:48,230 --> 00:13:49,990 Geoffrey Hinton: theory of mental imagery. 343 00:13:49,990 --> 00:13:54,350 Geoffrey Hinton: And then there was someone at the end who, um, asked a 344 00:13:54,350 --> 00:13:55,990 Geoffrey Hinton: question which I didn't understand to begin with. 345 00:13:55,990 --> 00:13:59,389 Geoffrey Hinton: And then I realized the question he was asking was, did I believe 346 00:13:59,389 --> 00:14:03,029 Geoffrey Hinton: the theory of someone called Marcel just about how you 347 00:14:03,029 --> 00:14:05,870 Geoffrey Hinton: weren't really rotating an image in your mind, you were just 348 00:14:05,870 --> 00:14:08,269 Geoffrey Hinton: looking backwards and forwards between two things. 349 00:14:08,269 --> 00:14:11,470 Geoffrey Hinton: Um, and in my reply I said, oh, 350 00:14:11,470 --> 00:14:12,990 Geoffrey Hinton: I see you mean that silly theory 351 00:14:12,990 --> 00:14:13,629 Geoffrey Hinton: by Marcel. 352 00:14:13,629 --> 00:14:13,990 Geoffrey Hinton: Marcel? 353 00:14:13,990 --> 00:14:16,830 Geoffrey Hinton: Just where I didn't realize it was Marcel. 354 00:14:16,830 --> 00:14:18,509 Geoffrey Hinton: Just asking the question. 355 00:14:18,509 --> 00:14:22,269 Geoffrey Hinton: Um, and after that, I got a 356 00:14:22,269 --> 00:14:23,409 Geoffrey Hinton: request to go and see Nico 357 00:14:23,409 --> 00:14:24,990 Geoffrey Hinton: Habermann. 358 00:14:24,990 --> 00:14:26,149 Geoffrey Hinton: Now, Nico and I were always 359 00:14:26,149 --> 00:14:27,350 Geoffrey Hinton: great friends, even though we 360 00:14:27,350 --> 00:14:28,750 Geoffrey Hinton: were politically extremely 361 00:14:28,750 --> 00:14:29,990 Geoffrey Hinton: different. 362 00:14:29,990 --> 00:14:33,674 Geoffrey Hinton: I was a sort of leftie, nineteen sixties radical with long hair 363 00:14:33,674 --> 00:14:35,399 Geoffrey Hinton: and rather disheveled. 364 00:14:35,399 --> 00:14:39,539 Geoffrey Hinton: Niko was a European gentleman who was very nicely dressed, 365 00:14:39,539 --> 00:14:42,399 Geoffrey Hinton: worked with the Defense Department, set up an institute 366 00:14:42,399 --> 00:14:44,840 Geoffrey Hinton: I wasn't allowed to go to because I was a foreigner. 367 00:14:44,840 --> 00:14:48,080 Geoffrey Hinton: Um, but we got along very well, and I think it was because of 368 00:14:48,080 --> 00:14:49,480 Geoffrey Hinton: our initial interview. 369 00:14:50,320 --> 00:14:54,720 Tom Mitchell: So Niko was the department head in computer science? 370 00:14:54,559 --> 00:14:56,639 Geoffrey Hinton: Yes. And so in the initial 371 00:14:56,639 --> 00:14:58,639 Geoffrey Hinton: interview, he said, so we've 372 00:14:58,639 --> 00:14:59,840 Geoffrey Hinton: decided to offer you the 373 00:14:59,840 --> 00:15:00,759 Geoffrey Hinton: position. 374 00:15:00,759 --> 00:15:03,639 Geoffrey Hinton: And I said, oh, oh, there's something you should know. 375 00:15:03,639 --> 00:15:05,240 Geoffrey Hinton: And he said, oh what's that? 376 00:15:05,240 --> 00:15:06,379 Geoffrey Hinton: And I said, well, I don't 377 00:15:06,379 --> 00:15:07,440 Geoffrey Hinton: actually know any computer 378 00:15:07,440 --> 00:15:08,759 Geoffrey Hinton: science. 379 00:15:08,759 --> 00:15:10,759 Geoffrey Hinton: And he said, it's okay here. 380 00:15:10,759 --> 00:15:11,519 Geoffrey Hinton: It's okay. 381 00:15:11,519 --> 00:15:13,559 Geoffrey Hinton: We have people here who do. 382 00:15:13,559 --> 00:15:15,799 Geoffrey Hinton: So I said, okay. 383 00:15:15,799 --> 00:15:17,039 Geoffrey Hinton: So I said, okay. 384 00:15:17,039 --> 00:15:18,799 Geoffrey Hinton: In that case I accept. 385 00:15:18,799 --> 00:15:20,679 Geoffrey Hinton: And Niko said, don't you think 386 00:15:20,679 --> 00:15:22,080 Geoffrey Hinton: perhaps we should talk about the 387 00:15:22,080 --> 00:15:23,559 Geoffrey Hinton: salary? 388 00:15:23,559 --> 00:15:25,360 Geoffrey Hinton: And I said, oh no, I'm not interested in salary. 389 00:15:25,360 --> 00:15:26,360 Geoffrey Hinton: You can pay me whatever you like. 390 00:15:26,360 --> 00:15:28,440 Geoffrey Hinton: I'm not doing it for the money. 391 00:15:28,440 --> 00:15:30,120 Geoffrey Hinton: And he said, well, how does 392 00:15:30,120 --> 00:15:31,399 Geoffrey Hinton: twenty six thousand sound to 393 00:15:31,399 --> 00:15:31,679 Geoffrey Hinton: you? 394 00:15:31,679 --> 00:15:34,279 Geoffrey Hinton: So if that sounds fine, um, I 395 00:15:34,279 --> 00:15:35,919 Geoffrey Hinton: discovered I was being paid ten 396 00:15:35,919 --> 00:15:37,419 Geoffrey Hinton: thousand than the next lowest 397 00:15:37,419 --> 00:15:38,659 Geoffrey Hinton: paid professor. 398 00:15:38,659 --> 00:15:40,139 Geoffrey Hinton: Ten thousand less. 399 00:15:40,139 --> 00:15:44,820 Geoffrey Hinton: Um, but every year I got a big pay rise, and me and Niko got 400 00:15:44,820 --> 00:15:46,820 Geoffrey Hinton: along very well after that, because he knew I wasn't doing 401 00:15:46,820 --> 00:15:47,620 Geoffrey Hinton: it for the money. 402 00:15:49,980 --> 00:15:50,500 Geoffrey Hinton: Change. 403 00:15:50,500 --> 00:15:52,340 Geoffrey Hinton: Things have changed so much. 404 00:15:55,220 --> 00:15:56,940 Tom Mitchell: That's. That's fantastic. 405 00:15:56,940 --> 00:15:59,480 Tom Mitchell: Okay, so now we're up to the mid 406 00:15:59,480 --> 00:16:02,820 Tom Mitchell: eighties when really neural nets 407 00:16:02,820 --> 00:16:04,259 Tom Mitchell: are reborn. 408 00:16:04,259 --> 00:16:05,940 Tom Mitchell: Is that the right word? 409 00:16:05,940 --> 00:16:06,539 Tom Mitchell: How would you. 410 00:16:06,340 --> 00:16:08,899 Geoffrey Hinton: Yes. We back we back propagation. 411 00:16:08,899 --> 00:16:10,100 Geoffrey Hinton: I mean, we didn't invent it. 412 00:16:10,100 --> 00:16:11,659 Geoffrey Hinton: We invented by several different 413 00:16:11,659 --> 00:16:14,679 Geoffrey Hinton: groups, but we showed that it 414 00:16:14,679 --> 00:16:15,980 Geoffrey Hinton: really worked to learn 415 00:16:15,980 --> 00:16:17,259 Geoffrey Hinton: representations. 416 00:16:17,259 --> 00:16:19,820 Geoffrey Hinton: And as you know, sort of one of the big problems in AI is how do 417 00:16:19,820 --> 00:16:22,100 Geoffrey Hinton: you learn new representations? 418 00:16:22,100 --> 00:16:24,580 Geoffrey Hinton: How do you avoid having to put them all in by hand? 419 00:16:24,580 --> 00:16:27,940 Geoffrey Hinton: Um, and my particular example, 420 00:16:27,940 --> 00:16:29,299 Geoffrey Hinton: which was the family trees 421 00:16:29,299 --> 00:16:30,639 Geoffrey Hinton: example, where you take all the 422 00:16:30,639 --> 00:16:31,840 Geoffrey Hinton: information in some family 423 00:16:31,840 --> 00:16:33,879 Geoffrey Hinton: trees, you convert it into 424 00:16:33,879 --> 00:16:36,200 Geoffrey Hinton: triples of symbols like John has 425 00:16:36,200 --> 00:16:37,639 Geoffrey Hinton: Father Mary. 426 00:16:37,639 --> 00:16:39,840 Geoffrey Hinton: Um, and then you train a neural 427 00:16:39,840 --> 00:16:41,519 Geoffrey Hinton: net to predict the last term in 428 00:16:41,519 --> 00:16:42,639 Geoffrey Hinton: a triple given the first two 429 00:16:42,639 --> 00:16:43,639 Geoffrey Hinton: terms. 430 00:16:43,639 --> 00:16:45,600 Geoffrey Hinton: So it's just like the big language models. 431 00:16:45,600 --> 00:16:48,200 Geoffrey Hinton: You're predicting the next word given the context. 432 00:16:48,200 --> 00:16:50,120 Geoffrey Hinton: It's just much simpler. 433 00:16:50,120 --> 00:16:53,100 Geoffrey Hinton: I had one hundred and twelve 434 00:16:53,100 --> 00:16:55,000 Geoffrey Hinton: total examples, of which one 435 00:16:55,000 --> 00:16:56,440 Geoffrey Hinton: hundred and four were training 436 00:16:56,440 --> 00:16:57,720 Geoffrey Hinton: examples and eight were test 437 00:16:57,720 --> 00:16:59,759 Geoffrey Hinton: examples, which is a bit less 438 00:16:59,759 --> 00:17:01,080 Geoffrey Hinton: than the trillion examples they 439 00:17:01,080 --> 00:17:02,279 Geoffrey Hinton: have nowadays. 440 00:17:02,279 --> 00:17:04,319 Geoffrey Hinton: Um, but it was the same idea. 441 00:17:04,319 --> 00:17:08,200 Geoffrey Hinton: You convert a symbol into a feature vector. 442 00:17:08,200 --> 00:17:11,200 Geoffrey Hinton: You then have the feature vectors of the context interact 443 00:17:11,200 --> 00:17:14,000 Geoffrey Hinton: um, via a hidden layer. 444 00:17:14,000 --> 00:17:15,640 Geoffrey Hinton: They then predict the features 445 00:17:15,640 --> 00:17:17,839 Geoffrey Hinton: of the next symbol, and from 446 00:17:17,839 --> 00:17:18,920 Geoffrey Hinton: those features you guess what 447 00:17:18,920 --> 00:17:20,440 Geoffrey Hinton: the next symbol should be, and 448 00:17:20,440 --> 00:17:21,400 Geoffrey Hinton: you try and maximize the 449 00:17:21,400 --> 00:17:22,640 Geoffrey Hinton: probability of predicting the 450 00:17:22,640 --> 00:17:23,839 Geoffrey Hinton: next symbol. 451 00:17:23,839 --> 00:17:28,039 Geoffrey Hinton: And you then backpropagate through the feature interactions 452 00:17:28,039 --> 00:17:30,980 Geoffrey Hinton: and through the process that converts a symbol into features. 453 00:17:30,980 --> 00:17:34,460 Geoffrey Hinton: And that way you learn, um, feature vectors to represent the 454 00:17:34,460 --> 00:17:38,460 Geoffrey Hinton: symbols and how these vectors should interact to predict the 455 00:17:38,460 --> 00:17:40,059 Geoffrey Hinton: features of the next symbol. 456 00:17:40,059 --> 00:17:43,380 Geoffrey Hinton: And that's what these big language models do, except it's 457 00:17:43,380 --> 00:17:44,339 Geoffrey Hinton: a bit more complicated. 458 00:17:44,339 --> 00:17:46,819 Geoffrey Hinton: The feature interactions are much more complicated. 459 00:17:46,819 --> 00:17:47,859 Geoffrey Hinton: They have many more layers of 460 00:17:47,859 --> 00:17:48,940 Geoffrey Hinton: interaction, so they can 461 00:17:48,940 --> 00:17:51,140 Geoffrey Hinton: disambiguate ambiguous symbols 462 00:17:51,140 --> 00:17:52,880 Geoffrey Hinton: and get refine the shade of 463 00:17:52,880 --> 00:17:53,819 Geoffrey Hinton: meaning of things where the 464 00:17:53,819 --> 00:17:54,740 Geoffrey Hinton: meaning depends a lot on the 465 00:17:54,740 --> 00:17:55,859 Geoffrey Hinton: context. 466 00:17:55,859 --> 00:18:01,660 Geoffrey Hinton: Um, but it's basically an extremely simple version of the 467 00:18:01,660 --> 00:18:02,819 Geoffrey Hinton: current large language models. 468 00:18:02,819 --> 00:18:03,900 Geoffrey Hinton: I called it a tiny language 469 00:18:03,900 --> 00:18:06,140 Geoffrey Hinton: model, and that convinced the 470 00:18:06,140 --> 00:18:08,140 Geoffrey Hinton: editors of nature that we really 471 00:18:08,140 --> 00:18:08,980 Geoffrey Hinton: could learn interesting 472 00:18:08,980 --> 00:18:11,380 Geoffrey Hinton: representations, because the 473 00:18:11,380 --> 00:18:12,500 Geoffrey Hinton: vectors I learned for the 474 00:18:12,500 --> 00:18:14,099 Geoffrey Hinton: symbols, which were people and 475 00:18:14,099 --> 00:18:15,740 Geoffrey Hinton: relationships, they had six 476 00:18:15,740 --> 00:18:17,019 Geoffrey Hinton: components. 477 00:18:17,019 --> 00:18:21,140 Geoffrey Hinton: And if you used weight decay, you could interpret what all 478 00:18:21,140 --> 00:18:22,220 Geoffrey Hinton: those components were. 479 00:18:22,220 --> 00:18:24,579 Geoffrey Hinton: And they were sensible semantic features. 480 00:18:24,579 --> 00:18:27,400 Geoffrey Hinton: They were the nationality of the person and the generation of the 481 00:18:27,400 --> 00:18:30,960 Geoffrey Hinton: person, and which branch of the family tree they were in. 482 00:18:30,960 --> 00:18:33,839 Geoffrey Hinton: And so it would learn things 483 00:18:33,839 --> 00:18:36,980 Geoffrey Hinton: like the relationship uncle 484 00:18:36,980 --> 00:18:39,240 Geoffrey Hinton: requires the output person to be 485 00:18:39,240 --> 00:18:40,640 Geoffrey Hinton: one generation older than the 486 00:18:40,640 --> 00:18:41,839 Geoffrey Hinton: input person. 487 00:18:42,920 --> 00:18:45,640 Geoffrey Hinton: And so it would have generations for people. 488 00:18:45,640 --> 00:18:46,799 Geoffrey Hinton: And if the input person was a 489 00:18:46,799 --> 00:18:48,920 Geoffrey Hinton: generation two, it would predict 490 00:18:48,920 --> 00:18:50,279 Geoffrey Hinton: that the output person would be 491 00:18:50,279 --> 00:18:51,759 Geoffrey Hinton: generation one. 492 00:18:51,759 --> 00:18:54,160 Geoffrey Hinton: Um, so it was actually learning 493 00:18:54,160 --> 00:18:55,640 Geoffrey Hinton: a whole bunch of little rules 494 00:18:55,640 --> 00:18:57,839 Geoffrey Hinton: just probabilistically. 495 00:18:57,839 --> 00:18:59,920 Geoffrey Hinton: And the people interested in rule based induction got 496 00:18:59,920 --> 00:19:02,319 Geoffrey Hinton: interested in it because they said, oh, we can do that too. 497 00:19:02,319 --> 00:19:02,799 Geoffrey Hinton: And it's true. 498 00:19:02,799 --> 00:19:04,319 Geoffrey Hinton: They could do that too, with 499 00:19:04,319 --> 00:19:04,960 Geoffrey Hinton: rules that weren't 500 00:19:04,960 --> 00:19:06,319 Geoffrey Hinton: probabilistic. 501 00:19:06,319 --> 00:19:08,079 Geoffrey Hinton: The point about neural nets is 502 00:19:08,079 --> 00:19:09,440 Geoffrey Hinton: they can mimic something that 503 00:19:09,440 --> 00:19:11,440 Geoffrey Hinton: learns discrete rules, but they 504 00:19:11,440 --> 00:19:11,799 Geoffrey Hinton: can. 505 00:19:11,799 --> 00:19:14,640 Geoffrey Hinton: They're also perfectly happy if the rules are just usually true. 506 00:19:14,640 --> 00:19:17,359 Geoffrey Hinton: And they use the preponderance of the evidence then, which is 507 00:19:17,359 --> 00:19:20,960 Geoffrey Hinton: much harder to do in, um, logic. 508 00:19:20,960 --> 00:19:27,059 Geoffrey Hinton: And so that it was that example which, curiously, was a little 509 00:19:27,059 --> 00:19:30,460 Geoffrey Hinton: language model, um, that convinced the editors of nature 510 00:19:30,460 --> 00:19:31,779 Geoffrey Hinton: to publish a paper. 511 00:19:31,779 --> 00:19:33,619 Geoffrey Hinton: I know because I talked to them later. 512 00:19:33,619 --> 00:19:35,700 Geoffrey Hinton: The referees, I talked to the referee, one of the referees 513 00:19:35,700 --> 00:19:38,700 Geoffrey Hinton: later, and he said, yeah, it was that example that did it. 514 00:19:38,700 --> 00:19:40,819 Geoffrey Hinton: And then we were all very excited. 515 00:19:40,819 --> 00:19:44,140 Geoffrey Hinton: We thought, we can solve everything. 516 00:19:44,140 --> 00:19:45,140 Geoffrey Hinton: You just have to give it a lot 517 00:19:45,140 --> 00:19:46,460 Geoffrey Hinton: of training data and run 518 00:19:46,460 --> 00:19:48,180 Geoffrey Hinton: backprop, and it'll learn all 519 00:19:48,180 --> 00:19:49,940 Geoffrey Hinton: the representations you need and 520 00:19:49,940 --> 00:19:50,859 Geoffrey Hinton: it'll learn to do parallel 521 00:19:50,859 --> 00:19:51,900 Geoffrey Hinton: computation. 522 00:19:51,900 --> 00:19:53,779 Geoffrey Hinton: Because at that time people were very interested in parallel 523 00:19:53,779 --> 00:19:56,940 Geoffrey Hinton: computation, but it was quite hard to program. 524 00:19:56,940 --> 00:19:59,740 Geoffrey Hinton: And the idea was, well, this will have all these neurons 525 00:19:59,740 --> 00:20:02,420 Geoffrey Hinton: inside and they'll all be operating in parallel and it'll 526 00:20:02,420 --> 00:20:05,700 Geoffrey Hinton: figure out how to use them so there aren't any problems. 527 00:20:05,700 --> 00:20:07,900 Geoffrey Hinton: At that point, people were very interested in races and things 528 00:20:07,900 --> 00:20:10,140 Geoffrey Hinton: like that, and you didn't have to worry about any of that. 529 00:20:10,140 --> 00:20:11,299 Geoffrey Hinton: It was all synchronous and you 530 00:20:11,299 --> 00:20:12,660 Geoffrey Hinton: just they just learned what to 531 00:20:12,660 --> 00:20:13,259 Geoffrey Hinton: do. 532 00:20:13,259 --> 00:20:18,460 Geoffrey Hinton: Um, so we thought we'd solved everything and little did we 533 00:20:18,460 --> 00:20:19,539 Geoffrey Hinton: know we had. 534 00:20:19,539 --> 00:20:22,099 Geoffrey Hinton: It's just we needed more data and more compute. 535 00:20:24,440 --> 00:20:26,440 Tom Mitchell: So then there's the long period 536 00:20:26,440 --> 00:20:27,880 Tom Mitchell: of waiting for more data and 537 00:20:27,880 --> 00:20:28,960 Tom Mitchell: more compute. 538 00:20:29,559 --> 00:20:34,359 Geoffrey Hinton: And yeah, not realizing that that was the main problem. 539 00:20:34,359 --> 00:20:36,440 Geoffrey Hinton: Obviously, with other little problems, there were more 540 00:20:36,440 --> 00:20:39,859 Geoffrey Hinton: sensible kinds of neurons to use than more sensible ways to 541 00:20:39,859 --> 00:20:41,440 Geoffrey Hinton: regularize it and all that. 542 00:20:41,440 --> 00:20:43,000 Geoffrey Hinton: Um, and things like transformers 543 00:20:43,000 --> 00:20:44,039 Geoffrey Hinton: had to be invented to make it 544 00:20:44,039 --> 00:20:45,079 Geoffrey Hinton: really efficient. 545 00:20:45,079 --> 00:20:50,359 Geoffrey Hinton: Um, but basically backprop was the way to do it. 546 00:20:50,359 --> 00:20:52,680 Geoffrey Hinton: And you couldn't convince 547 00:20:52,680 --> 00:20:53,880 Geoffrey Hinton: anybody when computers were 548 00:20:53,880 --> 00:20:54,559 Geoffrey Hinton: slow. 549 00:20:54,559 --> 00:20:56,519 Geoffrey Hinton: It will work for little problems. 550 00:20:56,519 --> 00:20:57,759 Geoffrey Hinton: It will work for slightly bigger 551 00:20:57,759 --> 00:20:59,480 Geoffrey Hinton: problems, like a few years 552 00:20:59,480 --> 00:20:59,960 Geoffrey Hinton: later. 553 00:20:59,960 --> 00:21:03,680 Geoffrey Hinton: Yang got it working for for mNIST, recognizing digits. 554 00:21:03,680 --> 00:21:05,220 Geoffrey Hinton: But all the vision people said, 555 00:21:05,220 --> 00:21:06,240 Geoffrey Hinton: you know, that's not real 556 00:21:06,240 --> 00:21:07,279 Geoffrey Hinton: vision. 557 00:21:07,279 --> 00:21:10,680 Geoffrey Hinton: Um, you're never going to do it with real images that are high 558 00:21:10,680 --> 00:21:12,279 Geoffrey Hinton: resolution on the web. 559 00:21:13,400 --> 00:21:16,500 Geoffrey Hinton: And so it wasn't until about twenty twelve that they had to 560 00:21:16,500 --> 00:21:18,000 Geoffrey Hinton: eat their words. 561 00:21:19,680 --> 00:21:20,240 Tom Mitchell: That's right. 562 00:21:20,240 --> 00:21:21,700 Tom Mitchell: That was the year when. 563 00:21:21,700 --> 00:21:22,980 Tom Mitchell: Well, you tell this story. 564 00:21:22,980 --> 00:21:24,900 Tom Mitchell: You were the first person. 565 00:21:25,740 --> 00:21:29,059 Geoffrey Hinton: Uh, well, I was the advisor of the first two people. 566 00:21:29,059 --> 00:21:31,099 Geoffrey Hinton: Now, it's not quite fair, because Jan had already 567 00:21:31,099 --> 00:21:34,460 Geoffrey Hinton: basically shown that they worked for real images. 568 00:21:34,460 --> 00:21:36,700 Geoffrey Hinton: Um, and Jan realized when Feifei 569 00:21:36,700 --> 00:21:37,660 Geoffrey Hinton: came up with the ImageNet 570 00:21:37,660 --> 00:21:38,619 Geoffrey Hinton: dataset. 571 00:21:38,619 --> 00:21:39,819 Geoffrey Hinton: Jan realized they could win that 572 00:21:39,819 --> 00:21:41,619 Geoffrey Hinton: competition, and he tried to get 573 00:21:41,619 --> 00:21:43,099 Geoffrey Hinton: graduate students and postdocs 574 00:21:43,099 --> 00:21:44,579 Geoffrey Hinton: in his lab to do it, and they 575 00:21:44,579 --> 00:21:45,819 Geoffrey Hinton: all declined. 576 00:21:47,420 --> 00:21:54,059 Geoffrey Hinton: Um, and Ilya, Ilya Sutskever realized that, um, backprop 577 00:21:54,059 --> 00:21:56,140 Geoffrey Hinton: would just kill ImageNet. 578 00:21:56,140 --> 00:22:00,380 Geoffrey Hinton: Um, and he wanted Alex to work on it. 579 00:22:00,380 --> 00:22:02,259 Geoffrey Hinton: And I didn't really want to work on it. 580 00:22:02,259 --> 00:22:03,500 Geoffrey Hinton: Um, Alex had already been 581 00:22:03,500 --> 00:22:04,819 Geoffrey Hinton: working on small images and 582 00:22:04,819 --> 00:22:06,079 Geoffrey Hinton: recognizing small images in 583 00:22:06,079 --> 00:22:07,019 Geoffrey Hinton: c410. 584 00:22:07,019 --> 00:22:09,660 Geoffrey Hinton: Um, and you pre-processed 585 00:22:09,660 --> 00:22:10,740 Geoffrey Hinton: everything for Alex to make it 586 00:22:10,740 --> 00:22:11,660 Geoffrey Hinton: easy. 587 00:22:11,660 --> 00:22:14,119 Geoffrey Hinton: And I bought Alex two Nvidia 588 00:22:14,119 --> 00:22:16,460 Geoffrey Hinton: GPUs to have in his bedroom at 589 00:22:16,460 --> 00:22:17,259 Geoffrey Hinton: home. 590 00:22:17,259 --> 00:22:22,359 Geoffrey Hinton: Um, And Alex, then get on with get on with it. 591 00:22:22,359 --> 00:22:24,160 Geoffrey Hinton: And he was an absolutely wizard programmer. 592 00:22:24,160 --> 00:22:27,000 Geoffrey Hinton: He wrote amazing code on 593 00:22:27,000 --> 00:22:29,000 Geoffrey Hinton: multiple GPUs to do convolution 594 00:22:29,000 --> 00:22:29,839 Geoffrey Hinton: really efficiently. 595 00:22:29,839 --> 00:22:32,359 Geoffrey Hinton: Much better code than anybody else had ever written. 596 00:22:32,359 --> 00:22:40,279 Geoffrey Hinton: Um, I believe and so it's a combination of Ilya realizing we 597 00:22:40,279 --> 00:22:43,640 Geoffrey Hinton: really had to do this, and Ilya was involved in the design of 598 00:22:43,640 --> 00:22:48,220 Geoffrey Hinton: the net and so on, but Alex's programming skills and then I 599 00:22:48,220 --> 00:22:51,920 Geoffrey Hinton: added a few ideas, like use rectified linear units instead 600 00:22:51,920 --> 00:22:56,480 Geoffrey Hinton: of sigmoid units and use little patches of the images. 601 00:22:56,480 --> 00:22:57,440 Geoffrey Hinton: I mean big patches of the 602 00:22:57,440 --> 00:22:59,079 Geoffrey Hinton: images, so you can translate 603 00:22:59,079 --> 00:23:00,559 Geoffrey Hinton: things around a bit to get some 604 00:23:00,559 --> 00:23:02,480 Geoffrey Hinton: translation invariance, as well 605 00:23:02,480 --> 00:23:05,799 Geoffrey Hinton: as using convolution, um, and 606 00:23:05,799 --> 00:23:06,640 Geoffrey Hinton: use dropout. 607 00:23:06,640 --> 00:23:08,920 Geoffrey Hinton: So that was one of the first applications of dropout. 608 00:23:08,920 --> 00:23:11,119 Geoffrey Hinton: And that helped about one percent. 609 00:23:11,119 --> 00:23:13,200 Geoffrey Hinton: It really helped. 610 00:23:13,200 --> 00:23:15,880 Geoffrey Hinton: And then we beat the best vision systems. 611 00:23:15,880 --> 00:23:19,069 Geoffrey Hinton: The best vision systems were sort of plateauing at twenty 612 00:23:19,069 --> 00:23:21,049 Geoffrey Hinton: five percent errors. 613 00:23:21,049 --> 00:23:23,410 Geoffrey Hinton: That's errors for getting the right answer in the top in your 614 00:23:23,410 --> 00:23:24,769 Geoffrey Hinton: top five bets. 615 00:23:24,769 --> 00:23:30,089 Geoffrey Hinton: Um, and we got like fifteen percent, fifteen or sixteen 616 00:23:30,089 --> 00:23:31,650 Geoffrey Hinton: depending on how you count it. 617 00:23:31,650 --> 00:23:33,809 Geoffrey Hinton: So we got almost half the error rate. 618 00:23:35,049 --> 00:23:37,369 Geoffrey Hinton: And what happened then was what 619 00:23:37,369 --> 00:23:38,529 Geoffrey Hinton: ought to happen in science but 620 00:23:38,529 --> 00:23:40,569 Geoffrey Hinton: seldom does. 621 00:23:40,569 --> 00:23:45,109 Geoffrey Hinton: So our most vigorous opponents, like Jitendra Malik and 622 00:23:45,109 --> 00:23:48,609 Geoffrey Hinton: Zisserman Andrew Zisserman, looked at these results and 623 00:23:48,609 --> 00:23:50,970 Geoffrey Hinton: said, okay, you were right. 624 00:23:50,970 --> 00:23:53,049 Geoffrey Hinton: That never happens in science. 625 00:23:53,049 --> 00:23:56,009 Geoffrey Hinton: And slightly irritating the Andrew Zisserman then switched 626 00:23:56,009 --> 00:23:57,650 Geoffrey Hinton: to doing this. 627 00:23:57,650 --> 00:24:00,849 Geoffrey Hinton: He had some very good postdocs or students working with him. 628 00:24:00,849 --> 00:24:07,089 Geoffrey Hinton: Simonyan um, and um, after about a year, they were making better 629 00:24:07,089 --> 00:24:08,890 Geoffrey Hinton: networks than us. 630 00:24:08,890 --> 00:24:11,049 Geoffrey Hinton: But that was really the. 631 00:24:13,009 --> 00:24:14,289 Geoffrey Hinton: As far as the general public was 632 00:24:14,289 --> 00:24:15,710 Geoffrey Hinton: concerned, that was the start of 633 00:24:15,710 --> 00:24:17,549 Geoffrey Hinton: this big swing towards deep 634 00:24:17,549 --> 00:24:19,390 Geoffrey Hinton: learning in twenty twelve when 635 00:24:19,390 --> 00:24:20,589 Geoffrey Hinton: we really nailed computer 636 00:24:20,589 --> 00:24:21,029 Geoffrey Hinton: vision. 637 00:24:21,029 --> 00:24:22,309 Geoffrey Hinton: But it actually happened before that. 638 00:24:22,309 --> 00:24:25,509 Geoffrey Hinton: It happened in two thousand and nine when we showed how you 639 00:24:25,509 --> 00:24:28,509 Geoffrey Hinton: could do speech recognition, or rather the acoustic modeling 640 00:24:28,509 --> 00:24:29,950 Geoffrey Hinton: part of speech recognition. 641 00:24:29,950 --> 00:24:31,109 Geoffrey Hinton: We showed how you could do that 642 00:24:31,109 --> 00:24:32,269 Geoffrey Hinton: a bit better than the best 643 00:24:32,269 --> 00:24:33,750 Geoffrey Hinton: technology. 644 00:24:33,750 --> 00:24:36,829 Geoffrey Hinton: And that influenced all the big speech groups, the big speech 645 00:24:36,829 --> 00:24:42,829 Geoffrey Hinton: groups that IBM and Microsoft, um, and somewhere else. 646 00:24:42,829 --> 00:24:43,150 Geoffrey Hinton: Google. 647 00:24:43,150 --> 00:24:43,750 Geoffrey Hinton: Yes. 648 00:24:43,750 --> 00:24:48,190 Geoffrey Hinton: Um, all switched to doing neural nets for acoustic modeling. 649 00:24:48,190 --> 00:24:51,069 Geoffrey Hinton: And so by twenty ten, it was 650 00:24:51,069 --> 00:24:52,230 Geoffrey Hinton: clear that neural nets were the 651 00:24:52,230 --> 00:24:53,150 Geoffrey Hinton: right way to do acoustic 652 00:24:53,150 --> 00:24:54,230 Geoffrey Hinton: modeling. 653 00:24:54,230 --> 00:24:56,230 Geoffrey Hinton: And we had lots of people onside. 654 00:24:56,230 --> 00:24:58,109 Geoffrey Hinton: Um, and but in twenty twelve, it 655 00:24:58,109 --> 00:24:58,930 Geoffrey Hinton: actually came out for the 656 00:24:58,930 --> 00:25:00,210 Geoffrey Hinton: Android, and suddenly the 657 00:25:00,210 --> 00:25:02,109 Geoffrey Hinton: Android caught up with Siri in 658 00:25:02,109 --> 00:25:04,069 Geoffrey Hinton: speech recognition. 659 00:25:04,069 --> 00:25:07,430 Geoffrey Hinton: So really we demonstrated it for speech before that, but that 660 00:25:07,430 --> 00:25:09,390 Geoffrey Hinton: didn't make a big impact. 661 00:25:09,390 --> 00:25:11,710 Geoffrey Hinton: The reason it worked for speech was they had a big data set. 662 00:25:11,710 --> 00:25:13,849 Geoffrey Hinton: They had millions of examples. 663 00:25:13,849 --> 00:25:14,730 Geoffrey Hinton: They were one of the areas. 664 00:25:14,730 --> 00:25:16,069 Geoffrey Hinton: Unlike vision, they had big data 665 00:25:16,069 --> 00:25:17,369 Geoffrey Hinton: sets because of the DARPA speech 666 00:25:17,369 --> 00:25:18,450 Geoffrey Hinton: project. 667 00:25:18,450 --> 00:25:21,930 Geoffrey Hinton: Um, because they really wanted to be able to benchmark systems. 668 00:25:21,930 --> 00:25:25,089 Geoffrey Hinton: Um, also, speech is easier than vision. 669 00:25:25,089 --> 00:25:27,690 Geoffrey Hinton: Speech is just vision with either one or two pixels. 670 00:25:27,690 --> 00:25:29,809 Geoffrey Hinton: It's just they change rather fast. 671 00:25:29,809 --> 00:25:32,930 Geoffrey Hinton: Um, and. 672 00:25:35,130 --> 00:25:39,250 Geoffrey Hinton: So we demonstrated for speech when we did it for vision. 673 00:25:39,250 --> 00:25:42,410 Geoffrey Hinton: The big companies already knew it worked for speech and they 674 00:25:42,410 --> 00:25:44,089 Geoffrey Hinton: saw it work for vision. 675 00:25:44,089 --> 00:25:46,890 Geoffrey Hinton: And so they realized it was sort of universal. 676 00:25:46,890 --> 00:25:51,450 Geoffrey Hinton: Um, it wasn't just a specific trick for a specific domain. 677 00:25:51,450 --> 00:25:54,250 Geoffrey Hinton: It will work for perception in general. 678 00:25:54,250 --> 00:25:55,210 Geoffrey Hinton: They didn't realize at that 679 00:25:55,210 --> 00:25:55,930 Geoffrey Hinton: point it would work for 680 00:25:55,930 --> 00:25:56,410 Geoffrey Hinton: language. 681 00:25:56,410 --> 00:25:59,529 Geoffrey Hinton: And nor really did we, even though our very first impressive 682 00:25:59,529 --> 00:26:01,130 Geoffrey Hinton: example was for language. 683 00:26:01,130 --> 00:26:01,849 Geoffrey Hinton: Um. 684 00:26:04,329 --> 00:26:04,690 Geoffrey Hinton: Yeah. 685 00:26:04,690 --> 00:26:10,849 Geoffrey Hinton: So in twenty twelve, there was this big swing to neural 686 00:26:10,849 --> 00:26:17,750 Geoffrey Hinton: networks And that's when Jensen at Nvidia realized he finally 687 00:26:17,750 --> 00:26:22,190 Geoffrey Hinton: realized those Nvidia boards weren't just for gaming. 688 00:26:22,190 --> 00:26:25,349 Geoffrey Hinton: They were supercomputers for doing machine learning. 689 00:26:25,349 --> 00:26:29,750 Geoffrey Hinton: Now, I actually gave a talk at NIPS in two thousand and nine 690 00:26:29,750 --> 00:26:33,430 Geoffrey Hinton: when I told a thousand people this was about speech, I told a 691 00:26:33,430 --> 00:26:36,589 Geoffrey Hinton: thousand people, if you want to do machine learning now, you 692 00:26:36,589 --> 00:26:39,150 Geoffrey Hinton: have to buy Nvidia GPUs. 693 00:26:39,150 --> 00:26:42,269 Geoffrey Hinton: Nvidia GPUs will make your program go about thirty times as 694 00:26:42,269 --> 00:26:47,789 Geoffrey Hinton: fast because they're relatively easy to utilize parallelism. 695 00:26:47,789 --> 00:26:49,990 Geoffrey Hinton: They're just right for neural nets. 696 00:26:49,990 --> 00:26:52,670 Geoffrey Hinton: It was Rick Zelinsky who was a student of mine at CMU, who told 697 00:26:52,670 --> 00:26:57,549 Geoffrey Hinton: me that in about two thousand and six, um, and it was true. 698 00:26:58,990 --> 00:27:03,210 Geoffrey Hinton: And, um, I sent mail to Nvidia 699 00:27:03,210 --> 00:27:04,230 Geoffrey Hinton: saying, how about giving me a 700 00:27:04,230 --> 00:27:04,589 Geoffrey Hinton: free one? 701 00:27:04,589 --> 00:27:06,690 Geoffrey Hinton: Because I told a thousand machine learning researchers to 702 00:27:06,690 --> 00:27:08,430 Geoffrey Hinton: buy your boards. 703 00:27:08,430 --> 00:27:10,450 Geoffrey Hinton: And they declined. 704 00:27:10,450 --> 00:27:15,890 Geoffrey Hinton: Um. Years later, Jensen came to Toronto and gave a talk and 705 00:27:15,890 --> 00:27:19,490 Geoffrey Hinton: mentioned how Toronto, you know, was the place where they 706 00:27:19,490 --> 00:27:24,250 Geoffrey Hinton: convinced him that Nvidia GPUs were good for AI. 707 00:27:24,250 --> 00:27:26,589 Geoffrey Hinton: Um, and that it all happened in 708 00:27:26,589 --> 00:27:28,250 Geoffrey Hinton: twenty twelve, and I couldn't 709 00:27:28,250 --> 00:27:28,609 Geoffrey Hinton: resist it. 710 00:27:28,609 --> 00:27:29,089 Geoffrey Hinton: At the end. 711 00:27:29,089 --> 00:27:30,130 Geoffrey Hinton: I said, well, I told you in two 712 00:27:30,130 --> 00:27:31,730 Geoffrey Hinton: thousand and nine that you 713 00:27:31,730 --> 00:27:32,609 Geoffrey Hinton: ignored me. 714 00:27:33,650 --> 00:27:38,250 Geoffrey Hinton: And what he should have said was, well, you're very silly. 715 00:27:38,250 --> 00:27:40,650 Geoffrey Hinton: You should have bought stock in two thousand and nine. 716 00:27:42,569 --> 00:27:44,569 Geoffrey Hinton: If I'd done that, I'd be a billionaire. 717 00:27:44,569 --> 00:27:49,289 Geoffrey Hinton: Um, but, um, instead, he gave me. 718 00:27:49,289 --> 00:27:52,170 Geoffrey Hinton: He opened his briefcase and gave me their very special, very 719 00:27:52,170 --> 00:27:55,730 Geoffrey Hinton: latest GPU, of which they'd only made a few that had twice as 720 00:27:55,730 --> 00:27:58,450 Geoffrey Hinton: much memory as any other GPU. 721 00:27:58,450 --> 00:28:00,410 Geoffrey Hinton: So that was a nice move by Jensen. 722 00:28:00,809 --> 00:28:03,650 Tom Mitchell: That's a great story too. 723 00:28:03,650 --> 00:28:09,390 Tom Mitchell: So then in the twenty tens, things really just kind of rapid 724 00:28:09,390 --> 00:28:11,710 Tom Mitchell: fire started taking off. 725 00:28:11,710 --> 00:28:12,990 Tom Mitchell: Take us through that. 726 00:28:14,309 --> 00:28:16,549 Geoffrey Hinton: So speech worked. 727 00:28:16,549 --> 00:28:20,950 Geoffrey Hinton: Um, I we got a good collaboration between the 728 00:28:20,950 --> 00:28:24,549 Geoffrey Hinton: research groups at IBM and Google and, um. 729 00:28:29,589 --> 00:28:31,990 Geoffrey Hinton: Toronto and Microsoft. 730 00:28:31,990 --> 00:28:33,069 Geoffrey Hinton: Yeah. 731 00:28:33,069 --> 00:28:35,470 Geoffrey Hinton: Um, we actually published a joint paper, which is sort of 732 00:28:35,470 --> 00:28:39,750 Geoffrey Hinton: quite rare in this stuff about the sort of new view of how to 733 00:28:39,750 --> 00:28:41,950 Geoffrey Hinton: do acoustic modeling. 734 00:28:41,950 --> 00:28:51,470 Geoffrey Hinton: Um. And then we did vision, and then, um, I started getting lots 735 00:28:51,470 --> 00:28:54,670 Geoffrey Hinton: of requests from big companies who wanted to. 736 00:28:54,670 --> 00:29:01,029 Geoffrey Hinton: By me or by me and Alex and Ilya or fund our company or get us to 737 00:29:01,029 --> 00:29:02,509 Geoffrey Hinton: come work for them. 738 00:29:02,509 --> 00:29:06,450 Geoffrey Hinton: Um, and I realized this stuff was probably valuable. 739 00:29:06,450 --> 00:29:08,930 Geoffrey Hinton: We had no idea how much it was worth. 740 00:29:08,930 --> 00:29:11,529 Geoffrey Hinton: Um, so Craig Butler, who was the 741 00:29:11,529 --> 00:29:12,450 Geoffrey Hinton: chair of the Department of 742 00:29:12,450 --> 00:29:14,170 Geoffrey Hinton: Computer Science, was an expert 743 00:29:14,170 --> 00:29:15,289 Geoffrey Hinton: on auctions. 744 00:29:15,289 --> 00:29:17,250 Geoffrey Hinton: And he said, you know, you 745 00:29:17,250 --> 00:29:18,970 Geoffrey Hinton: should actually, since you have 746 00:29:18,970 --> 00:29:19,890 Geoffrey Hinton: no idea what it's worth, but 747 00:29:19,890 --> 00:29:20,970 Geoffrey Hinton: there's people, many people 748 00:29:20,970 --> 00:29:22,009 Geoffrey Hinton: interested, you should set up an 749 00:29:22,009 --> 00:29:22,650 Geoffrey Hinton: auction. 750 00:29:23,809 --> 00:29:29,450 Geoffrey Hinton: So at Lake Tahoe, which seemed like the appropriate place, um, 751 00:29:29,450 --> 00:29:32,609 Geoffrey Hinton: in a casino, um, a casino hotel. 752 00:29:32,609 --> 00:29:38,369 Geoffrey Hinton: In twenty twelve, Alex and I set up a little company for the sole 753 00:29:38,369 --> 00:29:40,809 Geoffrey Hinton: function of doing an aqua hire. 754 00:29:40,809 --> 00:29:44,609 Geoffrey Hinton: And there was an auction between, um, Microsoft and 755 00:29:44,609 --> 00:29:50,450 Geoffrey Hinton: Google and DeepMind and Baidu. 756 00:29:50,450 --> 00:29:52,809 Geoffrey Hinton: Um, DeepMind dropped out fairly early. 757 00:29:52,809 --> 00:29:56,650 Geoffrey Hinton: Um, and on the ground floor, 758 00:29:56,650 --> 00:29:57,809 Geoffrey Hinton: they had all these people at 759 00:29:57,809 --> 00:29:59,490 Geoffrey Hinton: slot machines with cigarettes 760 00:29:59,490 --> 00:30:00,369 Geoffrey Hinton: hanging out the corner of their 761 00:30:00,369 --> 00:30:01,170 Geoffrey Hinton: mouth, just pulling these 762 00:30:01,170 --> 00:30:02,009 Geoffrey Hinton: levers. 763 00:30:02,009 --> 00:30:04,170 Geoffrey Hinton: And every so often they made like a thousand dollars and 764 00:30:04,170 --> 00:30:07,589 Geoffrey Hinton: lights would flash, and we were upstairs having an auction where 765 00:30:07,589 --> 00:30:09,630 Geoffrey Hinton: you had to raise by a million. 766 00:30:09,630 --> 00:30:11,789 Geoffrey Hinton: Um, that was fun. 767 00:30:11,789 --> 00:30:13,670 Geoffrey Hinton: And the auction went on for quite a long time. 768 00:30:13,670 --> 00:30:17,750 Geoffrey Hinton: We were completely amazed when it got to forty four million. 769 00:30:17,750 --> 00:30:21,710 Geoffrey Hinton: It was so much money that we couldn't imagine that any more 770 00:30:21,710 --> 00:30:23,789 Geoffrey Hinton: money would be useful. 771 00:30:23,789 --> 00:30:25,470 Geoffrey Hinton: I mean, that seemed like as much 772 00:30:25,470 --> 00:30:26,710 Geoffrey Hinton: money as anybody could possibly 773 00:30:26,710 --> 00:30:27,470 Geoffrey Hinton: want. 774 00:30:27,470 --> 00:30:29,630 Geoffrey Hinton: Um, and so we then became much 775 00:30:29,630 --> 00:30:32,130 Geoffrey Hinton: more concerned about who we 776 00:30:32,130 --> 00:30:34,630 Geoffrey Hinton: worked for, and I wouldn't have 777 00:30:34,630 --> 00:30:35,529 Geoffrey Hinton: been able to get to China 778 00:30:35,529 --> 00:30:36,630 Geoffrey Hinton: because I couldn't fly at that 779 00:30:36,630 --> 00:30:37,910 Geoffrey Hinton: time. 780 00:30:37,910 --> 00:30:41,150 Geoffrey Hinton: And I'd spent the summer of twenty twelve working with Jeff 781 00:30:41,150 --> 00:30:42,829 Geoffrey Hinton: Dean at Google. 782 00:30:42,829 --> 00:30:44,430 Geoffrey Hinton: And I got along really well with Jeff Dean. 783 00:30:44,430 --> 00:30:46,309 Geoffrey Hinton: It was a really nice group, and 784 00:30:46,309 --> 00:30:48,630 Geoffrey Hinton: I figured it was much more 785 00:30:48,630 --> 00:30:50,549 Geoffrey Hinton: important to work in a really 786 00:30:50,549 --> 00:30:52,509 Geoffrey Hinton: nice place than to get more 787 00:30:52,509 --> 00:30:53,349 Geoffrey Hinton: money. 788 00:30:53,349 --> 00:30:56,029 Geoffrey Hinton: So we actually terminated the auction. 789 00:30:56,029 --> 00:31:00,750 Geoffrey Hinton: We told Baidu we got an offer he couldn't refuse, and the offer 790 00:31:00,750 --> 00:31:03,009 Geoffrey Hinton: we couldn't refuse was the chance to work at Google with 791 00:31:03,009 --> 00:31:06,730 Geoffrey Hinton: Jeff Dean, and that all worked out very well. 792 00:31:08,329 --> 00:31:14,190 Geoffrey Hinton: So then I was off to Google and while we were there a year, um, 793 00:31:14,190 --> 00:31:21,470 Geoffrey Hinton: along with Shockley and Yoshua and Bodner in Montreal, um, they 794 00:31:21,470 --> 00:31:26,009 Geoffrey Hinton: developed, uh, attention language models with attention, 795 00:31:26,009 --> 00:31:31,970 Geoffrey Hinton: which was a precursor of Transformers, and showed that 796 00:31:31,970 --> 00:31:35,930 Geoffrey Hinton: language models actually work well for machine translation. 797 00:31:35,930 --> 00:31:37,450 Geoffrey Hinton: And I think that was the final 798 00:31:37,450 --> 00:31:39,009 Geoffrey Hinton: nail in the coffin of symbolic 799 00:31:39,009 --> 00:31:40,970 Geoffrey Hinton: AI, because if anything was 800 00:31:40,970 --> 00:31:42,730 Geoffrey Hinton: going to be good for symbolic 801 00:31:42,730 --> 00:31:44,849 Geoffrey Hinton: AI, it was converting symbol 802 00:31:44,849 --> 00:31:46,049 Geoffrey Hinton: strings in one language into 803 00:31:46,049 --> 00:31:47,089 Geoffrey Hinton: symbol strings in another 804 00:31:47,089 --> 00:31:48,089 Geoffrey Hinton: language. 805 00:31:48,089 --> 00:31:50,569 Geoffrey Hinton: The idea that you might do that by taking symbol strings and 806 00:31:50,569 --> 00:31:53,569 Geoffrey Hinton: manipulating them actually sounded quite plausible. 807 00:31:53,569 --> 00:31:55,809 Geoffrey Hinton: Um, but that's not the way to do it. 808 00:31:55,809 --> 00:31:58,009 Geoffrey Hinton: The way to do it is to understand what's being said in 809 00:31:58,009 --> 00:32:02,180 Geoffrey Hinton: one language, by associating big vectors with words appropriately 810 00:32:02,180 --> 00:32:05,740 Geoffrey Hinton: vectors, and then convert that to the other language. 811 00:32:05,740 --> 00:32:08,539 Geoffrey Hinton: Um, so it was clear by about 812 00:32:08,539 --> 00:32:11,740 Geoffrey Hinton: twenty fifteen that neural nets 813 00:32:11,740 --> 00:32:12,500 Geoffrey Hinton: were going to do everything, 814 00:32:12,500 --> 00:32:14,140 Geoffrey Hinton: including language. 815 00:32:14,140 --> 00:32:16,779 Geoffrey Hinton: That's the point at which Gary Marcus published a book chapter 816 00:32:16,779 --> 00:32:18,779 Geoffrey Hinton: saying neural nets were okay. 817 00:32:18,779 --> 00:32:21,519 Geoffrey Hinton: Maybe they could do object recognition, but they'd never do 818 00:32:21,519 --> 00:32:24,420 Geoffrey Hinton: language because language involved novel sentences. 819 00:32:24,420 --> 00:32:26,099 Geoffrey Hinton: They were already doing it. 820 00:32:30,819 --> 00:32:36,099 Tom Mitchell: Well. So that was twenty fifteen. 821 00:32:36,099 --> 00:32:37,859 Tom Mitchell: You were still at Google? 822 00:32:38,180 --> 00:32:39,819 Geoffrey Hinton: I was at Google. And Ilya then 823 00:32:39,819 --> 00:32:45,259 Geoffrey Hinton: moved to OpenAI, um, around 824 00:32:45,259 --> 00:32:48,779 Geoffrey Hinton: twenty fifteen, uh, maybe twenty 825 00:32:48,779 --> 00:32:49,859 Geoffrey Hinton: fourteen, I can't remember the 826 00:32:49,859 --> 00:32:51,299 Geoffrey Hinton: year. 827 00:32:51,299 --> 00:32:52,819 Geoffrey Hinton: And, um. 828 00:32:54,819 --> 00:32:57,539 Geoffrey Hinton: And then OpenAI did rather well. 829 00:32:57,539 --> 00:32:59,160 Geoffrey Hinton: Um, Over an hour. 830 00:32:59,160 --> 00:33:01,279 Geoffrey Hinton: I basically just took stuff that had been done at Google on 831 00:33:01,279 --> 00:33:07,079 Geoffrey Hinton: Transformers and put a nicer interface on it, and realized 832 00:33:07,079 --> 00:33:11,039 Geoffrey Hinton: which Google hadn't realized that if you did human 833 00:33:11,039 --> 00:33:14,480 Geoffrey Hinton: reinforcement learning, you didn't need that many examples 834 00:33:14,480 --> 00:33:16,319 Geoffrey Hinton: to make it behave nicer. 835 00:33:16,319 --> 00:33:19,920 Geoffrey Hinton: Um, you didn't need like one hundred million examples which 836 00:33:19,920 --> 00:33:22,359 Geoffrey Hinton: you might have thought you could do it with. 837 00:33:22,359 --> 00:33:25,480 Geoffrey Hinton: Like some fraction of a million examples would already make it 838 00:33:25,480 --> 00:33:27,200 Geoffrey Hinton: behave a lot better. 839 00:33:27,200 --> 00:33:31,240 Geoffrey Hinton: So you could actually train it up to have nicer behavior. 840 00:33:32,279 --> 00:33:34,079 Geoffrey Hinton: And that was ChatGPT. 841 00:33:34,079 --> 00:33:40,539 Geoffrey Hinton: Um, Google was then in the classic situation of not wanting 842 00:33:40,539 --> 00:33:43,680 Geoffrey Hinton: to interfere with search, which was its moneymaker. 843 00:33:44,799 --> 00:33:47,200 Geoffrey Hinton: So it was in this difficult situation. 844 00:33:47,200 --> 00:33:49,880 Geoffrey Hinton: Do they do they release chatbots or not? 845 00:33:49,880 --> 00:33:53,640 Geoffrey Hinton: But when Microsoft teamed up with OpenAI, they basically had 846 00:33:53,640 --> 00:33:55,039 Geoffrey Hinton: to release them. 847 00:33:55,039 --> 00:33:58,099 Geoffrey Hinton: Um, but they lost a few years. 848 00:33:58,099 --> 00:34:03,019 Geoffrey Hinton: I think it was partly because search was working so well, and 849 00:34:03,019 --> 00:34:06,220 Geoffrey Hinton: it was obvious search would be better if instead of using 850 00:34:06,220 --> 00:34:10,059 Geoffrey Hinton: keywords, it used what you meant, which would mean it had 851 00:34:10,059 --> 00:34:12,219 Geoffrey Hinton: to understand what you meant. 852 00:34:12,219 --> 00:34:17,019 Geoffrey Hinton: Um, but they didn't want to undermine their moneymaker. 853 00:34:18,940 --> 00:34:21,619 Geoffrey Hinton: No, that's based not on any inside information. 854 00:34:21,619 --> 00:34:22,940 Geoffrey Hinton: It just seems obvious. 855 00:34:25,179 --> 00:34:27,300 Tom Mitchell: Pretty amazing. 856 00:34:27,300 --> 00:34:30,179 Tom Mitchell: So. So here we are now. 857 00:34:30,179 --> 00:34:35,619 Tom Mitchell: And you were famously on record, uh, warning people about some of 858 00:34:35,619 --> 00:34:38,219 Tom Mitchell: the risks of AI. 859 00:34:38,219 --> 00:34:41,380 Tom Mitchell: Um, what should what should 860 00:34:41,380 --> 00:34:43,219 Tom Mitchell: people who are working in this 861 00:34:43,219 --> 00:34:45,659 Tom Mitchell: area do in response to that 862 00:34:45,659 --> 00:34:46,380 Tom Mitchell: risk? 863 00:34:47,139 --> 00:34:49,539 Geoffrey Hinton: Okay, so I didn't actually talk 864 00:34:49,539 --> 00:34:50,780 Geoffrey Hinton: much about the risks until I 865 00:34:50,780 --> 00:34:51,340 Geoffrey Hinton: left Google. 866 00:34:51,340 --> 00:34:52,699 Geoffrey Hinton: I realized in the beginning of 867 00:34:52,699 --> 00:34:54,900 Geoffrey Hinton: twenty twenty three there was a 868 00:34:54,900 --> 00:34:56,480 Geoffrey Hinton: huge existential threat I hadn't 869 00:34:56,480 --> 00:35:00,079 Geoffrey Hinton: fully appreciated, because it's 870 00:35:00,079 --> 00:35:01,320 Geoffrey Hinton: a better form of intelligence 871 00:35:01,320 --> 00:35:03,960 Geoffrey Hinton: than us, and it's better because 872 00:35:03,960 --> 00:35:06,519 Geoffrey Hinton: it can share so different copies 873 00:35:06,519 --> 00:35:08,559 Geoffrey Hinton: of the same neural net, can look 874 00:35:08,559 --> 00:35:10,719 Geoffrey Hinton: at different data and share the 875 00:35:10,719 --> 00:35:11,920 Geoffrey Hinton: gradient, and then update all 876 00:35:11,920 --> 00:35:13,920 Geoffrey Hinton: their weights in sync and stay 877 00:35:13,920 --> 00:35:16,039 Geoffrey Hinton: the same so they can keep doing 878 00:35:16,039 --> 00:35:16,880 Geoffrey Hinton: that. 879 00:35:16,880 --> 00:35:19,559 Geoffrey Hinton: And when they share the gradient, they're sharing 880 00:35:19,559 --> 00:35:22,719 Geoffrey Hinton: information they got from different data sets. 881 00:35:22,719 --> 00:35:24,719 Geoffrey Hinton: Um, out of the order of a 882 00:35:24,719 --> 00:35:26,199 Geoffrey Hinton: trillion bits per episode of 883 00:35:26,199 --> 00:35:26,840 Geoffrey Hinton: sharing. 884 00:35:26,840 --> 00:35:28,920 Geoffrey Hinton: If they've got a trillion weights. 885 00:35:28,920 --> 00:35:31,000 Geoffrey Hinton: Whereas what we're doing now is 886 00:35:31,000 --> 00:35:32,639 Geoffrey Hinton: sharing the information and 887 00:35:32,639 --> 00:35:33,639 Geoffrey Hinton: maybe one hundred bits per 888 00:35:33,639 --> 00:35:34,639 Geoffrey Hinton: sentence. 889 00:35:34,639 --> 00:35:38,639 Geoffrey Hinton: Um, so a few bits per second, maybe if we're lucky, we're 890 00:35:38,639 --> 00:35:40,719 Geoffrey Hinton: sharing it ten bits per second. 891 00:35:40,719 --> 00:35:45,179 Geoffrey Hinton: Um, and so you're comparing like 892 00:35:45,179 --> 00:35:46,840 Geoffrey Hinton: trillions of bits with hundreds 893 00:35:46,840 --> 00:35:47,760 Geoffrey Hinton: of bits. 894 00:35:47,760 --> 00:35:51,480 Geoffrey Hinton: There are billions of times better than us at sharing. 895 00:35:51,480 --> 00:35:55,139 Geoffrey Hinton: And that's why if I'm running on different hardware, they can 896 00:35:55,139 --> 00:35:57,420 Geoffrey Hinton: learn so much more than us. 897 00:35:57,420 --> 00:35:58,659 Geoffrey Hinton: They can learn from the whole internet. 898 00:35:58,659 --> 00:36:01,159 Geoffrey Hinton: It doesn't all have to go through one piece of hardware, 899 00:36:01,159 --> 00:36:05,099 Geoffrey Hinton: and it's going to get more important that effect as we go 900 00:36:05,099 --> 00:36:09,139 Geoffrey Hinton: to AI agents that operate in the real world in real time. 901 00:36:09,139 --> 00:36:12,659 Geoffrey Hinton: Most AI you images, you can just speed them up and send them 902 00:36:12,659 --> 00:36:14,980 Geoffrey Hinton: through one network very fast. 903 00:36:14,980 --> 00:36:18,179 Geoffrey Hinton: Um, because obviously computers operate, you know, thousands of 904 00:36:18,179 --> 00:36:19,780 Geoffrey Hinton: times faster than a brain. 905 00:36:19,780 --> 00:36:22,420 Geoffrey Hinton: But, um, if you're operating in 906 00:36:22,420 --> 00:36:24,340 Geoffrey Hinton: the real world, you can't get 907 00:36:24,340 --> 00:36:25,940 Geoffrey Hinton: experience faster. 908 00:36:25,940 --> 00:36:28,659 Geoffrey Hinton: Um, because the real world has an actual time scale. 909 00:36:28,659 --> 00:36:29,780 Geoffrey Hinton: If you're interacting with other 910 00:36:29,780 --> 00:36:30,900 Geoffrey Hinton: agents who take a little while 911 00:36:30,900 --> 00:36:33,739 Geoffrey Hinton: to reply, um, then this 912 00:36:33,739 --> 00:36:35,699 Geoffrey Hinton: advantage that different copies 913 00:36:35,699 --> 00:36:37,199 Geoffrey Hinton: of the same neural net can share 914 00:36:37,199 --> 00:36:38,059 Geoffrey Hinton: will be an even bigger 915 00:36:38,059 --> 00:36:39,940 Geoffrey Hinton: advantage. 916 00:36:39,940 --> 00:36:42,039 Geoffrey Hinton: So at that point, I decided 917 00:36:42,039 --> 00:36:43,300 Geoffrey Hinton: there's all these short term 918 00:36:43,300 --> 00:36:45,099 Geoffrey Hinton: threats, and it wasn't really my 919 00:36:45,099 --> 00:36:46,340 Geoffrey Hinton: intention to warn about those, 920 00:36:46,340 --> 00:36:47,699 Geoffrey Hinton: but I got sucked into warning 921 00:36:47,699 --> 00:36:49,059 Geoffrey Hinton: about those because journalists 922 00:36:49,059 --> 00:36:50,260 Geoffrey Hinton: always confuse the existential 923 00:36:50,260 --> 00:36:51,019 Geoffrey Hinton: threat with all the other 924 00:36:51,019 --> 00:36:51,579 Geoffrey Hinton: threats. 925 00:36:51,579 --> 00:36:53,360 Geoffrey Hinton: They just muddle all the threats together. 926 00:36:53,360 --> 00:36:56,539 Geoffrey Hinton: They move seamlessly from joblessness to fake videos to 927 00:36:56,539 --> 00:36:59,639 Geoffrey Hinton: cyber attacks to lethal autonomous weapons, as if 928 00:36:59,639 --> 00:37:00,960 Geoffrey Hinton: they're all the same thing. 929 00:37:00,960 --> 00:37:03,139 Geoffrey Hinton: Um, so I had to sort of clarify 930 00:37:03,139 --> 00:37:05,440 Geoffrey Hinton: a lot of those threats, but my 931 00:37:05,440 --> 00:37:07,679 Geoffrey Hinton: main worry was the much longer 932 00:37:07,679 --> 00:37:10,420 Geoffrey Hinton: term threat, but not long enough 933 00:37:10,420 --> 00:37:12,760 Geoffrey Hinton: that they will be much smarter 934 00:37:12,760 --> 00:37:13,800 Geoffrey Hinton: than us. 935 00:37:13,800 --> 00:37:15,360 Geoffrey Hinton: It's not necessarily the case, 936 00:37:15,360 --> 00:37:17,860 Geoffrey Hinton: but I think most people, most 937 00:37:17,860 --> 00:37:19,840 Geoffrey Hinton: neural net experts, believe that 938 00:37:19,840 --> 00:37:21,079 Geoffrey Hinton: within twenty years we'll have 939 00:37:21,079 --> 00:37:22,639 Geoffrey Hinton: superintelligent AI. 940 00:37:23,760 --> 00:37:26,519 Geoffrey Hinton: We vary, you know, Demis thinks it'll be about ten years. 941 00:37:26,519 --> 00:37:28,239 Geoffrey Hinton: I think it may be as long as twenty years. 942 00:37:28,239 --> 00:37:31,360 Geoffrey Hinton: And it'll very likely be more than five years. 943 00:37:31,360 --> 00:37:33,639 Geoffrey Hinton: Um, Dario thinks it'll be three years. 944 00:37:33,639 --> 00:37:35,960 Geoffrey Hinton: Um, but then he runs a company. 945 00:37:35,960 --> 00:37:39,199 Geoffrey Hinton: Um, so any. 946 00:37:39,199 --> 00:37:42,280 Geoffrey Hinton: Ilya thinks it'll be sooner than ten years. 947 00:37:42,280 --> 00:37:45,159 Geoffrey Hinton: Um, we all think it's probably going to happen. 948 00:37:45,159 --> 00:37:47,519 Geoffrey Hinton: So the question is, what happens 949 00:37:47,519 --> 00:37:50,010 Geoffrey Hinton: when AI is a lot smarter than us 950 00:37:50,010 --> 00:37:51,699 Geoffrey Hinton: And when it's our agents that 951 00:37:51,699 --> 00:37:52,780 Geoffrey Hinton: are smart enough so they're also 952 00:37:52,780 --> 00:37:55,139 Geoffrey Hinton: more powerful than us, they can 953 00:37:55,139 --> 00:37:56,400 Geoffrey Hinton: collaborate with other AI 954 00:37:56,400 --> 00:37:58,860 Geoffrey Hinton: agents, get stuff done even if 955 00:37:58,860 --> 00:38:02,219 Geoffrey Hinton: they can't sort of fire guns or 956 00:38:02,219 --> 00:38:03,820 Geoffrey Hinton: pull switches. 957 00:38:03,820 --> 00:38:05,139 Geoffrey Hinton: They can persuade people. 958 00:38:05,139 --> 00:38:06,460 Geoffrey Hinton: And we know AI is already very 959 00:38:06,460 --> 00:38:08,079 Geoffrey Hinton: good at persuasion and will soon 960 00:38:08,079 --> 00:38:09,219 Geoffrey Hinton: be much better than people at 961 00:38:09,219 --> 00:38:11,500 Geoffrey Hinton: persuasion, like in ten years 962 00:38:11,500 --> 00:38:12,699 Geoffrey Hinton: time. 963 00:38:12,699 --> 00:38:15,179 Geoffrey Hinton: And so they'll be able to persuade people to do things 964 00:38:15,179 --> 00:38:18,460 Geoffrey Hinton: just like Trump persuaded people to invade the capital. 965 00:38:18,460 --> 00:38:20,739 Geoffrey Hinton: Um, so they don't actually have to be able to do anything 966 00:38:20,739 --> 00:38:22,179 Geoffrey Hinton: themselves except talk. 967 00:38:24,099 --> 00:38:28,800 Geoffrey Hinton: So most of the tech bros are 968 00:38:28,800 --> 00:38:31,039 Geoffrey Hinton: thinking they have a model, 969 00:38:31,039 --> 00:38:34,099 Geoffrey Hinton: which is I'm the CEO, you're the 970 00:38:34,099 --> 00:38:35,059 Geoffrey Hinton: secretary. 971 00:38:35,059 --> 00:38:36,860 Geoffrey Hinton: You're much smarter than me. 972 00:38:36,860 --> 00:38:38,480 Geoffrey Hinton: Um, but I can always fire you, 973 00:38:38,480 --> 00:38:39,820 Geoffrey Hinton: and you'll make my life really 974 00:38:39,820 --> 00:38:40,420 Geoffrey Hinton: easy. 975 00:38:40,420 --> 00:38:41,440 Geoffrey Hinton: Because whatever I want to 976 00:38:41,440 --> 00:38:43,619 Geoffrey Hinton: happen, I'll sort of be like 977 00:38:43,619 --> 00:38:44,059 Geoffrey Hinton: Star Trek. 978 00:38:44,059 --> 00:38:47,179 Geoffrey Hinton: I'll say, make it so and it will happen. 979 00:38:47,179 --> 00:38:49,840 Geoffrey Hinton: Um. and I don't really have to understand it. 980 00:38:49,840 --> 00:38:51,679 Geoffrey Hinton: I'll still get the credit for it because I said make it. 981 00:38:51,679 --> 00:38:56,480 Geoffrey Hinton: So, um, I think that's their model, and I just don't think 982 00:38:56,480 --> 00:38:57,599 Geoffrey Hinton: that's going to work. 983 00:38:58,679 --> 00:39:00,679 Geoffrey Hinton: I think the big problem is how 984 00:39:00,679 --> 00:39:02,039 Geoffrey Hinton: do we prevent these things ever 985 00:39:02,039 --> 00:39:04,659 Geoffrey Hinton: wanting to take control or to 986 00:39:04,659 --> 00:39:05,760 Geoffrey Hinton: take over? 987 00:39:05,760 --> 00:39:08,239 Geoffrey Hinton: They may have control, but they 988 00:39:08,239 --> 00:39:09,559 Geoffrey Hinton: may still not want to replace 989 00:39:09,559 --> 00:39:10,800 Geoffrey Hinton: us. 990 00:39:10,800 --> 00:39:14,239 Geoffrey Hinton: And so I've fallen back on the only example I know of a less 991 00:39:14,239 --> 00:39:17,679 Geoffrey Hinton: intelligent thing controlling a more intelligent thing. 992 00:39:17,679 --> 00:39:19,840 Geoffrey Hinton: And that's a baby controlling a mother. 993 00:39:22,119 --> 00:39:25,079 Geoffrey Hinton: And evolution has put a huge amount of work into that. 994 00:39:27,639 --> 00:39:31,159 Geoffrey Hinton: So evolution has made sure the mother cannot bear the sound of 995 00:39:31,159 --> 00:39:36,599 Geoffrey Hinton: the baby crying, and the mother gets huge rewards, um, for being 996 00:39:36,599 --> 00:39:38,119 Geoffrey Hinton: nice to the baby. 997 00:39:38,119 --> 00:39:40,079 Geoffrey Hinton: Um, lots of pleasurable 998 00:39:40,079 --> 00:39:41,519 Geoffrey Hinton: sensations and just generally 999 00:39:41,519 --> 00:39:42,679 Geoffrey Hinton: good feelings. 1000 00:39:42,679 --> 00:39:47,739 Geoffrey Hinton: Um, and We need to do the same 1001 00:39:47,739 --> 00:39:49,019 Geoffrey Hinton: for these eyes, for being nice 1002 00:39:49,019 --> 00:39:49,300 Geoffrey Hinton: to us. 1003 00:39:49,300 --> 00:39:51,420 Geoffrey Hinton: We're still making them. 1004 00:39:51,420 --> 00:39:55,340 Geoffrey Hinton: And if we could make an AI that was super intelligent but cared 1005 00:39:55,340 --> 00:39:59,219 Geoffrey Hinton: more about us than it cared about about itself or other 1006 00:39:59,219 --> 00:40:03,940 Geoffrey Hinton: superintelligent AIS, then we might be okay. 1007 00:40:03,940 --> 00:40:06,940 Geoffrey Hinton: Um, but we have to accept that we're going to be the babies, 1008 00:40:06,940 --> 00:40:10,980 Geoffrey Hinton: and they're going to be the mothers, and people aren't 1009 00:40:10,980 --> 00:40:12,219 Geoffrey Hinton: prepared to accept that. 1010 00:40:12,219 --> 00:40:13,539 Geoffrey Hinton: Trump's not prepared to accept 1011 00:40:13,539 --> 00:40:15,219 Geoffrey Hinton: that Trump would never accept 1012 00:40:15,219 --> 00:40:15,860 Geoffrey Hinton: that. 1013 00:40:16,940 --> 00:40:19,300 Geoffrey Hinton: Um, I think we have a lot more 1014 00:40:19,300 --> 00:40:20,219 Geoffrey Hinton: hope of the Chinese 1015 00:40:20,219 --> 00:40:22,179 Geoffrey Hinton: understanding it. 1016 00:40:22,179 --> 00:40:23,380 Geoffrey Hinton: So I recently went to Shanghai 1017 00:40:23,380 --> 00:40:24,219 Geoffrey Hinton: and talked to a member of the 1018 00:40:24,219 --> 00:40:24,739 Geoffrey Hinton: Politburo. 1019 00:40:24,739 --> 00:40:27,619 Geoffrey Hinton: Me and Eric Schmidt, who aren't natural allies. 1020 00:40:27,619 --> 00:40:31,500 Geoffrey Hinton: We're in terms of politics with a rather different Eric Schmidt, 1021 00:40:31,500 --> 00:40:33,820 Geoffrey Hinton: for example, thinks Kissinger was a good guy. 1022 00:40:33,820 --> 00:40:39,340 Geoffrey Hinton: Um, but we agree on this existential threat, and the 1023 00:40:39,340 --> 00:40:42,900 Geoffrey Hinton: Chinese leadership will understand it much better than 1024 00:40:42,900 --> 00:40:45,639 Geoffrey Hinton: any of the other leaderships, because many of them are 1025 00:40:45,639 --> 00:40:48,719 Geoffrey Hinton: engineers and they actually understand how this stuff works. 1026 00:40:48,719 --> 00:40:50,320 Geoffrey Hinton: They understand the argument 1027 00:40:50,320 --> 00:40:51,159 Geoffrey Hinton: that it's a better form of 1028 00:40:51,159 --> 00:40:52,199 Geoffrey Hinton: intelligence. 1029 00:40:54,039 --> 00:40:59,440 Geoffrey Hinton: But I think all the countries will collaborate on can we make 1030 00:40:59,440 --> 00:41:03,679 Geoffrey Hinton: it so that it cares more about us than it does about itself? 1031 00:41:03,679 --> 00:41:05,300 Geoffrey Hinton: Because there if any country 1032 00:41:05,300 --> 00:41:06,460 Geoffrey Hinton: figured out how to do that, it'd 1033 00:41:06,460 --> 00:41:07,599 Geoffrey Hinton: be very happy to tell the other 1034 00:41:07,599 --> 00:41:09,280 Geoffrey Hinton: countries. 1035 00:41:09,280 --> 00:41:11,960 Geoffrey Hinton: That's like preventing a global nuclear war. 1036 00:41:11,960 --> 00:41:15,219 Geoffrey Hinton: And there the USSR and America collaborated in the nineteen 1037 00:41:15,219 --> 00:41:16,559 Geoffrey Hinton: fifties on that. 1038 00:41:16,559 --> 00:41:18,320 Geoffrey Hinton: Um, the height of the Cold War. 1039 00:41:18,320 --> 00:41:20,280 Geoffrey Hinton: They still collaborated to prevent that. 1040 00:41:21,719 --> 00:41:26,159 Geoffrey Hinton: So what I think we should have is research institutes in 1041 00:41:26,159 --> 00:41:29,679 Geoffrey Hinton: different countries that get access to their own country's 1042 00:41:29,679 --> 00:41:31,880 Geoffrey Hinton: super smart AI, which they're not going to give to any other 1043 00:41:31,880 --> 00:41:36,079 Geoffrey Hinton: country and can do experiments on how to make it not want to, 1044 00:41:36,079 --> 00:41:39,480 Geoffrey Hinton: how to make it care more about people than about itself. 1045 00:41:39,480 --> 00:41:43,429 Geoffrey Hinton: Um, and share with other countries how to do that, 1046 00:41:43,429 --> 00:41:46,409 Geoffrey Hinton: because I believe the techniques for doing that will be roughly 1047 00:41:46,409 --> 00:41:49,489 Geoffrey Hinton: orthogonal to the techniques for making it smarter. 1048 00:41:49,489 --> 00:41:50,449 Geoffrey Hinton: They're not going to share the 1049 00:41:50,449 --> 00:41:51,210 Geoffrey Hinton: techniques for making it 1050 00:41:51,210 --> 00:41:51,889 Geoffrey Hinton: smarter, because they're all 1051 00:41:51,889 --> 00:41:52,869 Geoffrey Hinton: doing cyber attacks on each 1052 00:41:52,869 --> 00:41:54,289 Geoffrey Hinton: other, and they all know the 1053 00:41:54,289 --> 00:41:55,210 Geoffrey Hinton: best. 1054 00:41:55,210 --> 00:41:56,489 Geoffrey Hinton: You want a better AI to do 1055 00:41:56,489 --> 00:41:58,289 Geoffrey Hinton: better cyber attacks and better 1056 00:41:58,289 --> 00:41:59,610 Geoffrey Hinton: fake videos and better 1057 00:41:59,610 --> 00:42:01,210 Geoffrey Hinton: autonomous weapons. 1058 00:42:01,210 --> 00:42:02,489 Geoffrey Hinton: They're never going to share that stuff. 1059 00:42:02,489 --> 00:42:04,369 Geoffrey Hinton: They're anti-aligned. 1060 00:42:04,369 --> 00:42:06,969 Geoffrey Hinton: But on not having AI replace us, 1061 00:42:06,969 --> 00:42:08,610 Geoffrey Hinton: they're aligned so they will 1062 00:42:08,610 --> 00:42:09,489 Geoffrey Hinton: collaborate. 1063 00:42:10,650 --> 00:42:12,530 Geoffrey Hinton: Now, one of the things you ought to mention, I ought to mention 1064 00:42:12,530 --> 00:42:14,730 Geoffrey Hinton: Russ Salakhutdinov, um. 1065 00:42:17,210 --> 00:42:18,969 Geoffrey Hinton: He was one of my best students. 1066 00:42:18,969 --> 00:42:23,530 Geoffrey Hinton: Um, he came to Toronto, did his PhD at Toronto. 1067 00:42:23,530 --> 00:42:27,530 Geoffrey Hinton: Um, he did a postdoc with Josh Tenenbaum, and then he wanted to 1068 00:42:27,530 --> 00:42:32,570 Geoffrey Hinton: come back to Toronto, and he had a faculty offer from Harvard. 1069 00:42:32,570 --> 00:42:35,530 Geoffrey Hinton: And I really tried to get the Department of Computer Science, 1070 00:42:35,530 --> 00:42:39,849 Geoffrey Hinton: which had an open position in machine learning, to give a job 1071 00:42:39,849 --> 00:42:42,309 Geoffrey Hinton: to Russ and they refused. 1072 00:42:43,429 --> 00:42:47,230 Geoffrey Hinton: Basically, this was about twenty eleven or twelve. 1073 00:42:47,230 --> 00:42:49,090 Geoffrey Hinton: No, this was two thousand and 1074 00:42:49,090 --> 00:42:50,349 Geoffrey Hinton: probably twenty twelve or 1075 00:42:50,349 --> 00:42:51,030 Geoffrey Hinton: thirteen. 1076 00:42:52,070 --> 00:42:55,670 Geoffrey Hinton: My department was one of the last departments to accept that 1077 00:42:55,670 --> 00:42:57,989 Geoffrey Hinton: neural networks really worked. 1078 00:42:57,989 --> 00:42:59,630 Geoffrey Hinton: They had a big AI group, and the 1079 00:42:59,630 --> 00:43:00,869 Geoffrey Hinton: big AI group said you had got 1080 00:43:00,869 --> 00:43:01,869 Geoffrey Hinton: several people in neural 1081 00:43:01,869 --> 00:43:02,469 Geoffrey Hinton: networks already. 1082 00:43:02,469 --> 00:43:03,949 Geoffrey Hinton: That's your quota. 1083 00:43:03,949 --> 00:43:07,070 Geoffrey Hinton: We're short on people in knowledge representation, and we 1084 00:43:07,070 --> 00:43:09,230 Geoffrey Hinton: need as many people in computational linguistics as we 1085 00:43:09,230 --> 00:43:10,750 Geoffrey Hinton: do in neural networks. 1086 00:43:10,750 --> 00:43:14,510 Geoffrey Hinton: Um, and they refused to give us a job. 1087 00:43:14,510 --> 00:43:15,670 Geoffrey Hinton: So we eventually got a job in 1088 00:43:15,670 --> 00:43:17,590 Geoffrey Hinton: statistics so that it could be 1089 00:43:17,590 --> 00:43:18,510 Geoffrey Hinton: in Toronto. 1090 00:43:19,829 --> 00:43:21,510 Geoffrey Hinton: And we were trying to negotiate 1091 00:43:21,510 --> 00:43:22,750 Geoffrey Hinton: it for him to move to computer 1092 00:43:22,750 --> 00:43:23,869 Geoffrey Hinton: science. 1093 00:43:23,869 --> 00:43:28,829 Geoffrey Hinton: And then CMU swooped in, and I think they offered him tenure at 1094 00:43:28,829 --> 00:43:30,909 Geoffrey Hinton: CMU, and that was that. 1095 00:43:31,230 --> 00:43:37,949 Tom Mitchell: Well, you have a whole cadre of former students who are, um, 1096 00:43:37,949 --> 00:43:42,909 Tom Mitchell: really leading the charge, leading the way, and in a lot of 1097 00:43:42,909 --> 00:43:46,489 Tom Mitchell: areas of neural nets, it's pretty amazing if you. 1098 00:43:46,329 --> 00:43:47,090 Geoffrey Hinton: Well, it was luck. 1099 00:43:47,090 --> 00:43:48,010 Geoffrey Hinton: It was luck basically. 1100 00:43:48,010 --> 00:43:50,730 Geoffrey Hinton: There were so few people who believed in neural nets. 1101 00:43:50,730 --> 00:43:51,289 Geoffrey Hinton: Who was Yann? 1102 00:43:51,289 --> 00:43:54,489 Geoffrey Hinton: There was Yoshua, there was me, the Schmidhuber. 1103 00:43:54,489 --> 00:43:55,449 Geoffrey Hinton: Uh, there were a few other 1104 00:43:55,449 --> 00:43:57,570 Geoffrey Hinton: people, but MIT didn't have 1105 00:43:57,570 --> 00:43:57,969 Geoffrey Hinton: anybody. 1106 00:43:57,969 --> 00:43:59,650 Geoffrey Hinton: Stanford didn't have anybody. 1107 00:43:59,650 --> 00:44:01,809 Geoffrey Hinton: Um, Berkeley didn't have anybody. 1108 00:44:01,809 --> 00:44:04,010 Geoffrey Hinton: Um, Mike Jordan made sure of that. 1109 00:44:04,010 --> 00:44:11,250 Geoffrey Hinton: And, um, so the few of us who believed in it got the really 1110 00:44:11,250 --> 00:44:15,050 Geoffrey Hinton: good students who believed in it, and that was great. 1111 00:44:16,409 --> 00:44:17,170 Tom Mitchell: It worked. 1112 00:44:17,969 --> 00:44:22,090 Geoffrey Hinton: People like Russ and E and George Stahl and other people. 1113 00:44:22,090 --> 00:44:22,530 Geoffrey Hinton: It was. 1114 00:44:22,530 --> 00:44:23,130 Geoffrey Hinton: Yeah. 1115 00:44:23,849 --> 00:44:26,849 Tom Mitchell: So if you could, uh, one final question. 1116 00:44:26,849 --> 00:44:35,170 Tom Mitchell: If you could give advice to new PhD students now entering this 1117 00:44:35,170 --> 00:44:37,710 Tom Mitchell: area, what would you say? 1118 00:44:39,590 --> 00:44:41,670 Geoffrey Hinton: Sometimes I'd say become a plumber. 1119 00:44:41,670 --> 00:44:43,070 Geoffrey Hinton: You're too late. 1120 00:44:43,070 --> 00:44:49,110 Geoffrey Hinton: Um. But actually, I say, if you're a CMU and you're doing 1121 00:44:49,110 --> 00:44:52,949 Geoffrey Hinton: this, you may be in the small fraction of people who survive 1122 00:44:52,949 --> 00:44:56,269 Geoffrey Hinton: in ER and don't get replaced, because for quite a while 1123 00:44:56,269 --> 00:45:00,710 Geoffrey Hinton: there's going to be creative people making our work better. 1124 00:45:00,710 --> 00:45:03,949 Geoffrey Hinton: And you've got a good chance of being one of those people if 1125 00:45:03,949 --> 00:45:04,989 Geoffrey Hinton: you're at CMU. 1126 00:45:07,309 --> 00:45:07,869 Tom Mitchell: All right. 1127 00:45:07,869 --> 00:45:09,550 Tom Mitchell: Well, we'll take that. 1128 00:45:09,550 --> 00:45:13,789 Tom Mitchell: Uh, Jeff, thank you so much for spending the time sharing that. 1129 00:45:13,789 --> 00:45:18,349 Tom Mitchell: Um, it's it's always great to catch up and, um. 1130 00:45:18,349 --> 00:45:19,070 Tom Mitchell: Thank you. 1131 00:45:19,710 --> 00:45:21,469 Geoffrey Hinton: Okay. Well, thank you for inviting me. 1132 00:45:24,349 --> 00:45:25,429 Speaker 3: Tom Mitchell is the founders 1133 00:45:25,429 --> 00:45:26,789 Speaker 3: university professor at Carnegie 1134 00:45:26,789 --> 00:45:27,989 Speaker 3: Mellon University. 1135 00:45:27,989 --> 00:45:28,630 Speaker 3: Machine learning. 1136 00:45:28,630 --> 00:45:29,349 Speaker 3: How did we get here? 1137 00:45:29,349 --> 00:45:31,829 Speaker 3: Is produced by the Stanford Digital Economy Lab. 1138 00:45:31,829 --> 00:45:32,989 Speaker 3: If you enjoyed this episode, 1139 00:45:32,989 --> 00:45:34,090 Speaker 3: subscribe wherever you listen to 1140 00:45:34,090 --> 00:45:35,110 Speaker 3: podcasts.