NVIDIA CEO Jensen Huang
05 Aug 2024 (1 month ago)
Teaser
- The speaker expresses a desire to have NVIDIA's full production team for every episode of the show, highlighting the benefits of having professional support for filming.
- The speaker mentions the use of RED cameras for the home studio, starting from the next episode.
- The speaker introduces the episode of "Acquired" with a playful and engaging question about who has the truth, setting the stage for an intriguing story.
Intro
- This episode of the podcast focuses on NVIDIA, a technology company at the forefront of the AI revolution. The hosts, Ben Gilbert and David Rosenthal, spent hundreds of hours researching NVIDIA and had the opportunity to interview CEO Jensen Huang at the company's headquarters.
- NVIDIA, currently valued at $1.1 trillion, is the sixth most valuable company globally. The company is facing a pivotal moment, with high expectations for its future success in the AI market. The hosts explore whether NVIDIA can maintain its dominance in this rapidly evolving field.
- The episode features Jensen Huang sharing his insights on NVIDIA's journey from graphics to data centers and AI, including stories of overcoming challenges and near-death experiences. He also offers advice for founders and reveals a more personal side to his leadership journey.
Riva 128
- Nvidia's risky bet on the Riva 128: In 1997, Nvidia was on the verge of bankruptcy. They were developing the Riva 128, a groundbreaking 3D graphics chip, but faced significant challenges. They had limited resources, a tight deadline, and a new technology (DirectX) that was incompatible with their existing architecture. Despite these obstacles, Nvidia made a bold decision to bet the company on the Riva 128. They decided to fully embrace DirectX, build the most powerful chip possible, and use an emulator to test their software before the chip was even manufactured. This risky strategy paid off, and the Riva 128 became a huge success, saving Nvidia from financial ruin.
- The importance of pre-simulation and conviction: Jensen Huang emphasizes the importance of pre-simulation and conviction in making bold decisions. He argues that by simulating the entire chip and software stack before production, Nvidia was able to ensure the Riva 128's success. He believes that when you have a strong conviction in your product, you should be willing to "bet the farm" and invest everything you have. This strategy has worked for Nvidia repeatedly, from the Riva 128 to the development of CUDA.
- The emergence of CUDA and its potential for machine learning: Nvidia's development of CUDA, a platform for general-purpose computing on GPUs, was also a bold move. While the potential for machine learning was not fully realized at the time, Nvidia saw the potential of programmable shaders and their ability to perform parallel processing. The success of AlexNet, a deep learning model, further solidified Nvidia's belief in the power of machine learning and the potential of CUDA to revolutionize the field.
- The transformative power of machine learning: Huang argues that machine learning, with its ability to learn from data and make predictions, has the potential to transform nearly every industry. He believes that machine learning can be applied to a wide range of problems, from predicting consumer behavior to understanding complex scientific phenomena. This realization led Nvidia to invest heavily in developing hardware and software specifically designed for machine learning, further solidifying their position as a leader in the field.
Post-AlexNet
- Nvidia CEO Jensen Huang reflects on the period following the breakthrough of AlexNet in 2012, a time when deep learning was still considered a "science project" by most. He describes how Silicon Valley companies like Google, Facebook, and Netflix were starting to realize the immense economic potential of this technology, leading to the creation of OpenAI a few years later.
- Huang emphasizes the importance of Nvidia's early focus on research and development, particularly through its CUDA platform. CUDA, initially designed for supercomputing, had already gained traction among researchers in various fields, including molecular dynamics, imaging, and quantum chemistry. This existing network of researchers proved crucial when deep learning emerged as a promising field.
- Nvidia actively engaged with leading AI researchers like Yann LeCun, Andrew Ng, and Geoffrey Hinton, providing them with the tools and resources to advance their work. Huang highlights the early days of Generative Adversarial Networks (GANs), where even rudimentary models with low resolution were seen as a potential for significant progress. Nvidia's belief in the scalability of deep learning, both in terms of data and model size, fueled their commitment to supporting research and development in this area.
OpenAI
- Jensen Huang acknowledges the significance of OpenAI's founding in 2015, recognizing its impact on the tech industry despite initial skepticism.
- While not directly involved in OpenAI's founding, Huang had connections with key figures like Elon Musk, Peter Thiel, and Ilya Sutskever, and understood the need for powerful computing resources.
- NVIDIA played a crucial role in supporting OpenAI's research by providing the first version of the DGX supercomputer, which was instrumental in accelerating their progress. Huang believed in the potential of deep learning and witnessed its rapid development through the increasing frequency of research papers published.
Language Models
- Jensen Huang was initially impressed by the cleverness of language models, particularly BERT, which uses self-supervised learning to predict the next word in a sequence. He was surprised by how much more accurate and lifelike language models became when they crossed the 10 billion parameter mark, and even more so when they reached 100 billion parameters.
- Huang believes that language models are essentially encoding and compressing information from the world's languages and text. This includes reasoning, which can be learned through reading. He suggests that the emergent capabilities of language models, such as reasoning, are a result of this encoded information.
- Huang compares the amazement of seeing a large language model's capabilities to the feeling of understanding how a computer works from a fundamental level. He finds it remarkable that these complex systems can function so effectively, even though the underlying mechanisms are still somewhat mysterious.
Statsig
- StatSig is a company founded by former Facebook engineers who recognized the importance of experimentation in deploying machine learning models. They built a platform that allows companies to conduct A/B testing, feature flagging, and product analytics, similar to the tools used by Google and Facebook.
- StatSig's platform is designed to help companies improve and deploy their AI models more effectively. It is used by a wide range of companies, including OpenAI and Anthropic.
- StatSig offers a variety of pricing options, including a free tier, a special program for venture-backed companies, and enterprise pricing. They also offer a special offer for the "Acquired" community, providing 5 million free events per month and white-glove onboarding support.
Direct Reports
- Nvidia's organizational structure is not traditional. Jensen Huang believes that a company's structure should reflect the architecture of the product it builds. He compares Nvidia's structure to a computing stack, with different layers of people responsible for different modules. This structure allows for a more agile and efficient organization, where the best person for the job, regardless of title, is in charge.
- Nvidia's organizational structure is designed to facilitate collaboration. Huang describes the company's structure as a "neural network" where "mission is the boss." This means that teams are assembled based on the specific mission at hand, regardless of traditional departmental boundaries. This approach allows for faster decision-making and information sharing.
- Nvidia's structure places a high emphasis on individual responsibility and leadership. Because information is disseminated quickly and widely, there is no hierarchy of power based on proximity to the source of information. Leaders earn their positions through their ability to reason through problems and help others succeed. This creates a culture of shared responsibility and accountability.
Product Shipping Cycle
- Jensen Huang emphasizes the importance of a rapid product shipping cycle, highlighting Nvidia's impressive ability to release new products frequently, especially considering the complexity of the technology they work with.
- He uses the example of Apple, suggesting that it would be unusual for them to release two flagship iPhones per year, to illustrate the rarity of such a rapid product release cycle in the tech industry.
- Huang emphasizes that while he draws inspiration from various sources, including business books and other companies, he doesn't simply imitate their strategies. Instead, he uses these sources to inform his own unique approach, considering the specific context of Nvidia's capabilities and goals.
Journey to the Data Center
- Nvidia's journey to the data center began almost 17 years ago, driven by the realization that the company's technology was limited by the physical constraints of desktop PCs. Nvidia's CEO, Jensen Huang, envisioned a future where computing could be done remotely, freeing it from the limitations of physical proximity to a viewing device. This vision led to the development of GeForce Now, Nvidia's first data center product.
- Nvidia's data center strategy was built upon a series of products that gradually expanded the company's capabilities. GeForce Now was followed by remote graphics, which placed Nvidia's GPUs in enterprise data centers. This led to the development of a supercomputer that combined CUDA and Nvidia's GPUs, further expanding the company's reach into the data center market.
- Nvidia's data center strategy was not a sudden shift, but rather a gradual evolution based on a long-term vision. Huang emphasizes the importance of anticipating future opportunities and positioning the company to capitalize on them. He highlights that Nvidia's success in AI and the data center was built upon a foundation of previous products that allowed the company to learn and adapt. This foresight and strategic planning allowed Nvidia to be well-positioned to power the rise of AI and cloud computing.
Mellanox Acquisition
- Nvidia's acquisition of Mellanox was driven by the company's vision of becoming a data center company. Jensen Huang recognized that the future of computing lay in data centers, and that networking infrastructure was crucial for this shift. He saw that traditional Ethernet networking was not suitable for the distributed computing required by AI, particularly for training large language models.
- Mellanox's expertise in high-performance networking made them the ideal acquisition target. Nvidia had already collaborated with Mellanox in high-performance computing, and Huang was impressed by their team and technology. He believed that Mellanox's expertise in Infiniband technology would be essential for the future of AI and data centers.
- The acquisition was a strategic move that positioned Nvidia at the forefront of the AI revolution. Huang recognized the potential of large language models (LLMs) and understood that the ability to train them efficiently would be a key differentiator. By acquiring Mellanox, Nvidia gained a significant advantage in this emerging field.
Crusoe
- Crusoe is a cloud provider specifically designed for AI workloads, powered by clean energy. They are a major partner of NVIDIA, utilizing their A100 and H100 GPUs in their data centers.
- Crusoe addresses the growing demand for high-performance GPUs by offering NVIDIA's H100s at scale, making them one of the first cloud providers to do so. They prioritize providing the best AI cloud solution by using top-tier GPU hardware and investing in an optimized cloud software stack.
- Crusoe leverages wasted or stranded energy sources to power their cloud, resulting in significantly better performance and cost efficiency compared to traditional cloud providers. This approach also reduces environmental harm by utilizing energy that would otherwise be wasted. In 2022, Crusoe captured over 4 billion cubic feet of gas, preventing the release of approximately 500,000 metric tons of CO2 emissions.
Advice For Company Building
- Nvidia's approach to competition: Nvidia focuses on creating new markets rather than competing directly with existing players. They identify "Z billion dollar markets" - markets that don't yet exist but have the potential to be huge. By focusing on these emerging markets, Nvidia often has less competition and can establish a strong foothold.
- Building a platform: Nvidia believes in creating platforms that enable an ecosystem of developers and customers. This approach creates a network effect, making it difficult for competitors to enter the market. Nvidia's early focus on developer relations and the creation of CUDA, a platform for developers to build on, are key examples of this strategy.
- The importance of a consistent architecture: Nvidia emphasizes the importance of a consistent architecture across all its products. This allows for compatibility and scalability, making it easier for developers to build and deploy applications. Nvidia's commitment to CUDA and its backward compatibility across generations of GPUs is a testament to this principle.
Luck & Skill
- Nvidia's early success was a combination of skill and luck. The company made smart decisions, like designing chips in a way that was efficient and cost-effective, but they also benefited from the timing of their technology.
- The rise of 3D graphics in gaming was a key factor in Nvidia's success. While Nvidia believed in the power of accelerated graphics, it was the adoption of 3D graphics by games like Doom and Quake that truly propelled the company forward.
- Nvidia's success was also influenced by the decisions of game developers like John Carmack and Tim Sweeney. If they had chosen to use software rendering instead of accelerated graphics, Nvidia's path to success might have been very different.
Job Displacement
- AI's Impact on Jobs: Jensen Huang acknowledges that AI will inevitably lead to job displacement in the short term as it automates tasks and increases productivity. However, he believes that AI will ultimately create more jobs in the long run due to increased prosperity and expansion into new areas.
- AI Safety and Human-in-the-Loop: Huang emphasizes the importance of AI safety, particularly in areas like robotics and self-driving cars. He advocates for a "human-in-the-loop" approach, where AI systems are trained and validated before being released, ensuring human oversight and control.
- AI and Productivity: Huang draws a parallel between the impact of Moore's Law and AI, arguing that increased productivity driven by AI will lead to greater economic activity and job creation. He believes that human ambition will drive companies to expand and hire more people, even with increased automation.
Blinkist
- The speaker was searching for Jensen Huang's favorite business books and found it difficult to locate them. They eventually used an AI chatbot, Bard, to find a list of books that Jensen had mentioned in public forums.
- Blinkist has created "blinks" of these books, providing summaries of key insights. They also offer a curated collection of books related to the themes of the episode, including tech innovation, leadership, and acquisitions.
- Blinkist is offering Acquired listeners a 50% discount on all premium content, allowing access to thousands of books condensed into easy-to-digest summaries. They also offer curated reading lists, progress tracking features, and dedicated customer success managers for teams.
Favorite Sci-Fi
- Jensen Huang, CEO of NVIDIA, was asked about his favorite science fiction book, to which he responded that he has never read a science fiction book before.
- He was then asked about his favorite TV series, and he stated that Star Trek is his favorite.
- He also mentioned that he saw a conference room named "Verer" on his way into the building and thought it was a good name.
Daily Driver
- Jensen Huang's current daily driver is a Mercedes EQS, which he drives for security and other reasons.
- He previously owned a Toyota Supra, which holds a special place in his heart as it was the car he and his wife Lori drove back in after getting engaged.
- The Supra was unfortunately totaled in an accident, but Huang emphasizes that it wasn't the car's fault.
Favorite Business Book
- Jensen Huang, CEO of NVIDIA, mentions that he enjoys reading business books, particularly those by Clayton Christensen.
- He believes Christensen's series is the best because it is intuitive, sensible, and approachable.
- Huang also enjoys books by Andy Grove, finding them to be of high quality.
Don Valentine
- Jensen Huang describes Don Valentine as a grumpy but endearing investor who made a memorable statement when investing in NVIDIA: "If you lose my money, I'll kill you."
- Valentine's support for NVIDIA was evident through his enthusiastic responses to positive news about the company. He would often write "good job" in crayon on newspaper articles about NVIDIA and mail them to Huang.
- Huang emphasizes Valentine's genuine care for the companies he invests in, highlighting his unique and special character.
40 Year-Old Jensen
- Jensen Huang believes that there is plenty of time to achieve things in life, even if you are 40 years old, as long as you prioritize yourself and your time effectively.
- He emphasizes the importance of not letting Outlook or other external factors control your schedule and instead focusing on what is truly important.
- He suggests making sacrifices and prioritizing your life to ensure that you have enough time to achieve your goals.
What are You Afraid of?
- Jensen Huang, CEO of NVIDIA, expresses his greatest fear as letting down his employees.
- He acknowledges that many employees joined the company because they believe in its vision and have adopted its goals as their own.
- Huang emphasizes his desire to see his employees succeed, build fulfilling careers, and enjoy the benefits of the company's success.
Final Job
- Jensen Huang, CEO of NVIDIA, shares that he would still be at LSI Logic today if not for Chris and Curtis convincing him to join NVIDIA. He believes LSI Logic was a revolutionary company that significantly impacted chip design and the computer industry.
- Huang emphasizes the importance of high-level design, which allows for expressing chip design in high-level languages, enabling optimization and increased productivity. He sees this concept as revolutionary and believes it has influenced other industries like machine learning and software programming.
- Huang acknowledges that he didn't know how to write a business plan when he started NVIDIA, but he was fortunate to have the support of Wol Coryan, who recommended him to Don Valentine, a venture capitalist. This support helped NVIDIA get off the ground and has been instrumental in its success.
Starting a Company in 2023
- Jensen Huang, CEO of NVIDIA, reflects on the challenges of starting a company, particularly in light of NVIDIA's 30th anniversary. He acknowledges that knowing the immense difficulty and emotional toll of building a company would likely deter anyone from starting one.
- He attributes his own success to a "superpower" of entrepreneurs: the ability to underestimate the difficulty of their endeavors. This mindset, while necessary for motivation, can be a double-edged sword.
- Huang emphasizes the importance of a strong support system, including family, friends, and colleagues, as crucial for navigating the emotional challenges of building a company. He highlights the unwavering support he received from long-time NVIDIA employees, who never gave up on him or the company.
Market Drawdowns
- Jensen Huang, CEO of NVIDIA, discusses the company's experience with significant market drawdowns, including multiple 80% drops in market value. He emphasizes the importance of a strong belief system and unwavering commitment to the company's vision, especially during such challenging times.
- Huang acknowledges the difficulty of navigating these periods, particularly the public scrutiny and internal questioning that accompany stock price declines. He highlights the human element involved in both leadership and company building, emphasizing the challenges of enduring such setbacks.
- Huang emphasizes the importance of understanding the market opportunity when facing these challenges. He uses the example of NVIDIA's transition from a chip company to an AI company, demonstrating how the market size can expand exponentially with technological advancements. He suggests that technology companies, particularly those involved in AI and generative work, have the potential to become significantly larger in the future due to the vast market opportunities they address.
Outro
- The hosts encourage listeners to sign up for their email list, which includes notifications about new episodes, listener corrections, and teasers for upcoming episodes.
- They express gratitude to their sponsors, Blinkist, StatSig, and Crusoe, and encourage listeners to check out their links in the show notes for exclusive offers.
- They highlight their new podcast, "Acq," which features more frequent releases and deeper dives into topics discussed on the main "Acquired" podcast. They also mention their "LP Acquired" program, which offers exclusive benefits like Zoom calls with the hosts and the opportunity to influence future episode topics.
- The hosts invite listeners to join their Slack community and purchase merchandise from their online store.
- They conclude by thanking listeners and teasing the next episode.