Generally AI - Season 2 - Episode 6: The Godfathers of Programming and AI

Programmers' Day

• Russia observes the Day of the Programmer every year, which is held on the 256th day of the year, falling on September 13th in common years and September 12th in leap years (10s).
• In China, Programmers' Day is held on October 24th, as the date can be written as 1024 (35s).

The Godfather of AI: Jeffrey Hinton

• The podcast discusses famous programmers, starting from those who are currently relevant and going back to the foundation of computer science (1m20s).
• Jeffrey Hinton is commonly referred to as the Godfather of AI due to his perseverance in AI research and his significant contributions to the field over multiple decades (1m40s).
• Hinton's real interest is not computer science, but rather understanding how the brain works, which led him to study experimental psychology in Cambridge in 1970 (3m1s).
• Hinton received a PhD in artificial intelligence from Edinburgh in 1978, a time when AI PhD programs were already being offered (4m7s).
• The study of AI has been recognized as a distinct field, with some universities renaming degrees to include AI, such as cognitive psychology being renamed to AI (4m35s).
• The field of AI has come a long way, and one of the biggest contributions to this progress is Jeffrey Hinton's 1986 paper on learning representations by back-propagating errors, which introduced the famous back-propagation algorithm used to train large neural networks today (5m8s).
• Hinton is the second author of this paper, which was written in collaboration with two other researchers, and his work on this algorithm has made him a staple of being a Godfather of AI (5m26s).
• The back-propagation algorithm allows for learning from a large number of examples and generalizing to unseen sets of examples, which is a crucial aspect of AI (5m41s).
• In the past, there were two schools of thought in AI: one focused on representing things and learning from grammars, while the other focused on learning automatically from examples without representation (5m55s).
• Hinton's work has contributed to the latter approach becoming more prominent, and his persistence in this area has been instrumental in the development of AI (5m52s).
• Hinton also worked on stochastic neighbor embeddings, a probabilistic approach to placing objects in high-dimensional vectors, which is a technique for visualizing high-dimensional data in two or three dimensions (6m53s).
• This work was followed up by a paper on t-SNE (t-distributed Stochastic Neighbor Embedding), a technique for visualizing high-dimensional data in a lower-dimensional space (7m2s).
• Another notable paper by Hinton is his work on a mixture of experts, which explores how to train multiple neural networks to be experts on different topics and how to combine their answers (7m37s).
• This work has led to the development of mixture of expert networks, which are now being used by some big tech companies (8m1s).
• Hinton's work on neural networks was not widely recognized until 2012, when his PhD student, Alex Krizhevsky, and Alex's colleague, Ilya Sutskever, wrote a neural network that won the ImageNet competition with a large margin, leading to a surge in interest in neural networks (8m48s).
• The ImageNet competition had a thousand categories and measured the top five error rate, which dropped from 28% to 25% in two years before AlexNet, and then further dropped to 16% with AlexNet, demonstrating significant progress in object recognition (9m33s).
• Since AlexNet, the top one accuracy has improved from 58% to 92% with the best network, showing remarkable progress in object recognition (9m54s).
• Object recognition is considered "solved enough" to kickstart any problem and solve it temporarily, although there are still caveats, and it's definitely solved for most practical purposes (10m17s).
• The object recognition built into iPhones, for example, is pretty good and can do search on images for specific objects like dogs or mountains (10m40s).
• Computer vision is fairly easy nowadays, especially with access to the JBT API, and has come a long way from the days of finding edges using convolutions and doing Huff transforms (10m59s).
• AlexNet had 60 million parameters, 650,000 neurons, five convolutional layers, and three fully connected layers, demonstrating the power of deep learning in computer vision (11m37s).
• The success of AlexNet marked the start of larger datasets, more compute power, and larger networks that can achieve a generic understanding of the world without overfitting (11m55s).
• The AlexNet paper noted that their network took five to six days to train on two 3GB GPUs and suggested that results could be improved by waiting for faster GPUs and bigger datasets (12m28s).
• The idea of scaling AI by increasing compute power and data is not new, and Geoffrey Hinton had also anticipated the importance of more data and compute in improving AI performance (13m5s).
• The concept of "The Bitter Lesson" emphasizes the importance of more data and compute over trying to be clever, and is a lesson that has been learned in the development of AI (13m28s).
• The AlexNet paper also introduced the use of ReLU activation (Rectified Linear Units) and showed that their network learned faster than previous networks using the tanh activation function (13m42s).
• The paper also demonstrated the use of dropouts to prevent overfitting, another technique developed by Hinton (14m2s).
• Geoffrey Hinton's lab was making significant breakthroughs in computer vision and speech recognition, with large margins over classical methods, leading other labs to pivot towards his approach (14m8s).
• Hinton's research group in Toronto made massive breakthroughs in speech recognition, and he was making progress in multiple fields at once (14m49s).
• Big tech companies were already looking at his work, and Microsoft was sending him unsolicited gifts of $10,000 to $20,000 (15m9s).
• Hinton joined Google part-time as a researcher in 2013, but quit last year at the age of 76 to freely express his concerns about the existential threats of AI (15m30s).
• Hinton's concerns about AI include the potential for job replacement, the difficulty of knowing what is true or not due to the ease of generating content, and the possibility of lethal weapons (16m7s).
• He also worries about discrimination and bias in AI, but believes that machine learning solutions can be designed to have less bias than the systems they replace (17m8s).
• Hinton thinks that AI will be immensely helpful in areas like healthcare, but still sees a long-term existential threat from superintelligence being used by bad actors (17m39s).
• He believes that a superintelligent AI may take over if it is able to create sub-goals, one of which is to gain more control, and is able to communicate with people (17m57s).
• Jeff Hinton has expressed his existential fear of AI, believing that humans are too weak to resist its influence, and has chosen not to work with big tech companies to freely discuss his concerns about AI and its potential threats (18m15s).
• Despite uncertainty about the likelihood of an existential threat from AI, Hinton's expertise and knowledge in the field make his opinions worth listening to, and his lectures are recommended (18m49s).
• After leaving Google, Hinton is assumed to still be working at his research lab in Toronto, where he previously developed capture networks, a hierarchical approach to knowledge representation that did not gain significant attention (19m13s).
• Hinton is considered the "Godfather of AI" due to his significant contributions to the field (20m0s).

Cuon San Francisco Conference and John Von Neumann

• The Cuon San Francisco conference will feature software leaders discussing emerging trends, including generative AI in production, and will provide a platform for learning from senior software developers (20m23s).
• John Von Neumann, a lesser-known but fascinating intellectual figure of the 20th century, made several important contributions to computer science and will be the subject of discussion (22m25s).
• Von Neumann was born in Budapest, Hungary in 1903, just a few days after the Wright brothers' historic first flight (22m29s).
• John von Neumann was recognized as a prodigy at a young age, able to divide eight-digit numbers in his head, speak ancient Greek, and memorize entire books by the time he was a teenager (22m50s).
• Despite his father's disapproval, von Neumann pursued mathematics, enrolling in a PhD program while also studying chemical engineering, and finished both programs simultaneously (23m26s).
• Von Neumann did a postdoc year at Göttingen under David Hilbert and became a professor at the University of Berlin before moving to America in 1933 to join Princeton's Institute for Advanced Studies (23m41s).
• The Institute for Advanced Studies (IAS) is a familiar setting from movies such as Oppenheimer and A Beautiful Mind, although von Neumann is not featured in either film (24m2s).
• Von Neumann worked on computers at the IAS, and according to Freeman Dyson, he was considered to be in a class of his own, with a unique blend of mathematics, physics, and other disciplines (25m38s).
• Von Neumann made significant contributions to various fields, including mathematics, physics, economics, computer science, and defense work, with over 70 entries in his Wikipedia page listing things named after him (26m5s).
• In physics and defense work, von Neumann was the leading authority on shaped charges used in the Fat Man bomb and worked on the Manhattan Project (26m31s).
• Von Neumann's work on the Manhattan Project and his contributions to various fields have had a lasting impact, and his legacy serves as a reminder of the importance of controlling existential threats, including AI (26m53s).
• Von Neumann was a member of the Atomic Energy Commission and worked on the intercontinental ballistic missile program, and his contribution to economics ties him in with John Nash through the invention of game theory in collaboration with Oscar Morganstern (26m55s).
• Von Neumann and Morganstern wrote a book called "The Theory of Games and Economic Behavior" that provided a mathematical model of two-player Zero Sum games with perfect information, and also included work on games with imperfect information and games with more than two players (27m28s).
• John Nash famously extended game theory to include Cooperative games, as depicted in the movie "A Beautiful Mind" (28m2s).
• Von Neumann worked with computing machinery during the war, not only on the Manhattan Project but also on ballistics and other military projects, and is credited with inventing the merge sort algorithm (28m30s).
• Von Neumann is also known as a famous computer hardware designer, leading the project at the IAS to build a computer, and his design, known as the Von Neumann architecture, is still the high-level design of most modern computers (28m57s).
• The Von Neumann architecture includes a processing unit with both an ALU and registers, a control unit, memory for storing both data and instructions, and external storage and IO, which was a significant improvement over earlier computer devices that did not have stored programs (29m15s).
• Earlier computer devices, such as the ENIAC, had program code that was not stored in the machine's memory, but was instead hardcoded physically with wires or patch cables, making them one-purpose machines that could be reprogrammed but required physical reprogramming (29m46s).
• The Von Neumann architecture's use of stored programs in memory revolutionized computer design, allowing for more flexibility and ease of use (30m37s).
• There is an alternative architecture called the Harvard architecture, which has separate address and data buses for instructions versus data (30m42s).
• The Von Neumann architecture, where data and instructions are stored in memory and are interchangeable, is widely used and considered a fundamental concept in computer science (31m14s).
• This architecture allows for self-modifying programs and is considered a universal Turing machine, with most people believing that Von Neumann was aware of Turing's work and that it influenced his design (31m34s).
• Von Neumann described his architecture in a paper in 1945, and the team began working on it at The Institute in 1946, with the first operational version being completed in 1951 (31m45s).
• Von Neumann was a unique and interesting person, known for his great memory, love of music and parties, and fascination with the Byzantine Empire (32m12s).
• He was part of a group of Hungarian-American mathematicians and physicists known as "The Martians," which included notable figures such as Edward Teller and Leo Szilard (32m49s).
• The term "Martians" was coined due to a joke by physicist Enrico Fermi, who suggested that the lack of evidence of extraterrestrial intelligence could be explained by the fact that they were already on Earth, disguising themselves as Hungarians (33m16s).

Computer Architecture and The Martians

• The concept of computers as "alien technology" is a theme that has been explored in science fiction, but it is also reflected in the fact that many of the pioneers of computer science, including Von Neumann and Turing, were influenced by ideas from physics and mathematics (33m51s).
• Modern neural networks are based on a different architecture than the Von Neumann architecture, but there have been attempts to create hardware that is more similar to the structure of neural networks (34m49s).
• Richard Feynman, a physicist and computer scientist, worked on connection machines, which were an early type of hardware neural network, and wrote a book about his experiences (34m16s).

Book Recommendation and Miscellaneous Facts

• For those interested in learning more about computer science, a recommended book is "But How Do It Know? The Basic Principles of Computers for Everyone" by J. Clark Scott (35m2s).
• The Chinese celebrate Programmer's Day on October 24th, which is represented by the binary number 1024 (35m44s).
• Isaac Asimov, a science fiction author, also wrote mysteries and included a joke in one of his stories that Halloween (October 31) is equal to Christmas (December 25) because 31 in decimal is equal to 255 in hexadecimal (35m46s).
• A capybara, the largest rodent, was classified as a fish by the Vatican in 1784 so that Christians could eat it during Lent (36m43s).
• This classification was made because capybaras live in the water, have webbed feet, and taste like fish (36m47s).
• There is a story that the US government, possibly during the Obama Administration, declared pizza a vegetable for school lunches due to the nutritional value of the tomato sauce (37m50s).
• The tomato sauce on pizza contains a significant amount of vitamins, making it a healthier option than some other foods classified as vegetables (38m28s).
• The classification of pizza as a vegetable is an example of how logic-based systems can be flawed or inconsistent (38m46s).

Connecting with Listeners and Show Notes

• The hosts invite listeners in San Francisco to come and say hello if they recognize their voices, encouraging them not to be shy about approaching them (39m34s).
• The hosts express their desire to meet people who listen to their content, as it is usually difficult to know who their audience is when recording from different time zones (39m47s).
• The hosts mention that Cucon is an opportunity to meet the people they create content for, as normally they have no idea who is reading or listening to their work (40m5s).
• The show notes can be found on info.com, and the hosts thank their listeners for tuning in (40m14s).
• The hosts briefly discuss the recording process, noting that they are sitting alone in their rooms in different time zones (39m54s).