Behind the product: NotebookLM | Raiza Martin (Senior Product Manager, AI @ Google Labs)

Introduction to NotebookLM (0s)

• The hosts of the Deep Dive podcast express gratitude to their listeners and Lenny for having them on the show, mentioning the significant response they've received, especially regarding Notebook LM, a product that has sparked imagination about AI's potential (16s).
• Notebook LM is a product incubated within Google Labs, allowing users to upload a source and generate AI-generated audio (2m0s).
• The product has been gaining attention on Twitter and LinkedIn, blowing people's minds and sparking imagination about AI's possibilities (2m12s).
• Raiza Martin, the product lead for Notebook LM, is a big fan and listener of Lenny's podcast and is excited to discuss the history and future of the product (2m35s).
• The conversation with Raiza Martin will cover how Notebook LM came to be, the technology that made it possible, how the team works internally, and its incubation within Google Labs (1m12s).
• The discussion will also touch on the product's 20% time, fun and crazy use cases, and a glimpse into its long-term future (1m24s).
• The episode is brought to you by Explo, a game-changer for customer-facing analytics and data reporting (3m28s).
• As a product leader, building analytics from scratch can be time-consuming, expensive, and challenging, but Expo offers a fully white-labeled embedded analytics solution designed with the user in mind (3m45s).
• Expo connects to any relational database or warehouse and allows users to build and style dashboards in minutes with its low-code functionality (4m3s).
• Once ready, the dashboard or report can be embedded into an application with a tiny code snippet, and end-users can use Expo's AI features for their own report and dashboard generation (4m11s).
• This eliminates customer data requests for the support team, and a fully white-labeled analytics experience can be built and embedded in days (4m21s).
• Expo offers a free trial at expo./Lenny (4m29s).
• This episode is sponsored by Sprig, a product experience platform that generates AI-powered opportunities to continuously improve products at scale (4m39s).
• Sprig captures product experiences in real-time through heat maps, replays, surveys, and feedback studies, and its industry-leading AI analyzes data to generate real-time insights (4m54s).
• Sprig AI provides actionable product recommendations to drive revenue, retention, and user satisfaction (5m18s).
• Product teams can uncover AI-powered product opportunities at scale by visiting sprig.com/Lenny to book a demo and receive a $75 gift card (5m32s).

The genesis of NotebookLM (5m43s)

• The history of NotebookLM is not a sudden creation, but rather a product that has been developed over time, with its origins dating back to a 20% project at Google (5m44s).
• NotebookLM started as a 20% project, but it quickly grew beyond that, with a team consisting of a few people, including the Senior Product Manager, an engineer, and eventually Stephen Johnson (6m3s).
• The idea for NotebookLM originated from a smaller project called "Talk to Small Corpus" inside Google Labs, which aimed to use a large language model (LLM) to interact with a piece of content (6m36s).
• The initial team working on NotebookLM was small, with only one engineer working full-time on the technology, while others contributed part-time, providing input and ideas (7m3s).
• The team has since grown, with more engineers joining in the last month, in anticipation of the future roadmap, but at the time of the project's announcement, the team consisted of only three engineers, including Stephen Johnson (7m40s).
• The team size remained relatively small, with around eight engineers, from the first IO last year to as recently as last month (8m4s).

Innovative features and use cases (8m8s)

• The audio overview feature, technically referred to as a "deep dive," was previewed at Google's I/O event in May. It was developed as an enhancement to the existing Notebook, which is a source-grounded chat interface. (8m25s)
• The development of this feature involved exploring new models from Google, such as Gemini 1.5, and considering how these could improve the Notebook AI. A different team within Google Labs had powerful audio models, which led to the idea of using them to generate unexpected outputs from minimal input, like a URL or a resume. (9m9s)
• The initiative was not driven by a specific business problem but rather by the potential applications of the technology. The approach in Google Labs often starts with technology and explores practical applications, contrasting with the traditional method of starting with a problem and finding solutions. (10m7s)
• The audio feature aimed to provide a voice modality for interaction, as the output was traditionally text-based. The use of voice input and output was found to change user interaction and perception of the technology, making it more engaging and fun. This approach was likened to the transformative impact of the chat GPT technology. (10m50s)
• The development of new mediums and ways of interacting with technology can change the way people imagine and perceive its power, as seen in the past with the introduction of new technologies and now with the emergence of AI and LMs (11m41s).
• The goal is to shape technology to bring it closer to people and make it more relatable and engaging, which is what the team is trying to achieve with NotebookLM (11m59s).
• Technology has been available for some time, but it's the new medium and way of interacting with it that inspires people and makes them realize the advancements in AI and LMs (12m24s).
• The team has been exploring different ways to use LMs for only two years, and they are already introducing new technologies on top of it (12m40s).
• The idea of making a podcast using AI is an unintuitive but clever concept (12m47s).
• The technology behind NotebookLM includes the use of Gemini models, specifically Gemini 1.5 Pro, as the base, and a powerful voice model and audio model on top of it (13m18s).
• The real secret sauce to what makes NotebookLM good is the Content Studio, a tool built by the team that helps create engaging and relatable content (13m29s).
• The Content Studio is designed to be helpful and provides features such as summaries, buttons to write content, and audio integration (13m46s).
• The team had to think about the shape and format of the audio content, and they started with the Deep Dive format (14m0s).
• The Content Studio is a key component of NotebookLM, and it powers the different ways users can interact with their data, including Q&A and creating new content (14m33s).
• The Content Studio is an interface that allows users to interact with NotebookLM, providing prompts and recommendations for asking questions and creating content (14m50s).
• The Audio model used in the content studio is designed to bring out the best in the model, and it took a lot of listening to early attempts to figure out how to get the model to behave in a desired way, which is where the magic happens (15m11s).
• To improve the model, a lot of podcast episodes were listened to, and the process involved hearing something that worked and trying to replicate it (15m47s).
• The model's ability to mimic human-like conversations was tested at home, where it was played out loud, and it was not immediately clear that the conversations were AI-generated (16m0s).
• The model was used to generate a podcast from a short autobiography written by the developer's mom, who was blown away by the result and shared it with her friends (16m32s).
• The model also created a study guide from the autobiography, which was used to review the content at a dinner (16m48s).
• The model was used to generate an audio overview of the developer's dad's bio, which helped him understand what his child did for work (17m5s).
• Other surprising use cases for the audio deep dives include generating audio overviews of resumes, which were found to be delightful and helpful (17m39s).
• The model was also used to generate audio overviews of quarterly check-in notes, which boosted the confidence of Google employees going into meetings (18m0s).
• The model can be used to upload resumes and generate an audio overview that describes the person in a professional manner (18m22s).
• A user uploaded a Google Doc to try out a new feature, and the last document they had uploaded was likely something personal, such as their resume (18m25s).
• The user clicked the "generate" option without knowing what would happen next, which led to a unique experience (18m45s).
• The outcome of clicking "generate" was the user receiving hype and encouragement from two people (18m50s).

Building a startup culture within Google (18m52s)

• Google Labs is a relatively new organization within Google, established about three years ago, and it operates differently from the rest of the company, allowing for faster development and more flexibility (19m50s).
• The mission of Google Labs is to ship AI products and build businesses out of them, with a focus on moving quickly and doing things differently (20m28s).
• The team behind NotebookLM, a product developed within Google Labs, operates in a startup-like environment with fewer processes and more collaboration between product managers, engineers, and designers (21m7s).
• This approach allows the team to move quickly and make decisions rapidly, with meetings often involving simultaneous work on mocks, PRDs, and implementation (21m22s).
• The success of Google Labs and NotebookLM can be attributed to clear expectations from senior leaders, such as Josh Woodward, the VP of Google Labs, who encouraged the team to do things differently and move quickly (22m4s).
• The team's approach is characterized by a high level of transparency, with daily tweets and a Discord server with around 60,000 members, which is unusual for a Google product (19m5s).
• The goal of Google Labs is to create a model for how other Google teams can operate in the future, with a focus on speed, agility, and innovation (19m28s).
• The project was developed with a small team consisting of one engineer, a product manager, and Stephen Johnson, emphasizing a collaborative and experimental approach without a specific goal. (22m21s)
• The team decided to use Discord for communication, despite initial unfamiliarity within Google, as it was seen as an effective tool for building a community outside of traditional Google platforms. (22m33s)
• There was initial concern about whether people would join the Discord server, but it successfully grew to 60,000 members, indicating strong community engagement. (23m5s)
• Sidebar is introduced as a platform for senior tech professionals, from director to C-level, to advance their careers by being matched into peer groups for unbiased opinions, diverse perspectives, and feedback. (23m48s)
• Sidebar provides world-class programming and facilitation, enabling members to receive focused, tactical feedback throughout their career journey, with 93% of members reporting significant positive career changes. (24m2s)

Expanding user demographics (24m28s)

• The NotebookLM product has gained significant traction, with 60,000 users in its Discord server, and its retention rates have increased positively across daily, weekly, and monthly measures, indicating a strong user base (24m40s).
• The product's demographics have expanded beyond its initial user base of Educators and Learners, now including a significant number of professionals who are interested in using the tool for work purposes (25m11s).
• Some companies have even reached out to make the use of NotebookLM official for their employees, indicating a growing interest in the product among businesses (25m33s).
• The number of businesses using NotebookLM has increased astronomically, prompting the need to hire a business development person to handle customer calls and explore potential business opportunities (25m55s).
• The team's success is now defined by building a business, with a clear sense of what needs to be achieved in the future, including exploring different pathways for distribution, monetization, and commercialization (26m37s).
• The focus is on deepening the user experience while exploring commercialization opportunities, with the goal of building a sustainable business around the product (27m17s).

The product roadmap (27m30s)

• The vision for the product is to have an AI editor surface that is fully remixable, allowing users to take any input and turn it into any output, such as turning emails, LinkedIn posts, or Twitter posts into a blog post, tutorial video, or chatbot (28m2s).
• The goal is to allow users to shape and transform content in various ways, making it a powerful and core feature of the product (28m14s).
• The product roadmap includes bringing the experience to mobile, which is currently a big gap in the experience, and making it more interesting and interactive (28m55s).
• The mobile app is expected to be a key area of focus, with the goal of creating a seamless and engaging experience for users (29m8s).
• The product team is experimenting with different formats and interrupt experiences, such as allowing users to participate in podcast conversations as a mobile experience (29m15s).
• Future improvements to the product include adding more controls and features, such as knobs, sliders, and text boxes, but with a focus on making the experience more magical and delightful (29m42s).
• The goal is to move beyond a one-shot experience, where users can only create a single version of a document or podcast episode, and instead allow for more flexibility and creativity (30m18s).
• People have different preferences for consuming information, with some preferring to watch, listen, or read content, and the ability to deliver information in various mediums such as blog posts, tweets, podcasts, or newsletters would cater to these diverse preferences (30m50s).
• The format of information delivery is not always flexible, and individuals often have to accept the format in which it is presented, but having the power to choose or convert the format, such as turning a long document into an audio overview, can shift the dynamic between the person and the knowledge being shared (31m28s).
• There are instances where individuals may not engage with information due to its format, such as not reading a long document, and having alternative formats can facilitate better consumption and understanding of the information (31m46s).
• Raiza Martin shared a personal anecdote about joining Google Labs and being given a 50-page document by Josh, which she didn't read, instead opting to ask him questions and engage in a Q&A session, highlighting the potential benefits of interactive or alternative formats for information delivery (31m54s).

Other use cases (32m18s)

• The original use case for NotebookLM was creating a podcast from a scientific paper, allowing users to catch up on the latest AI research without having to read dense and complex papers (32m19s).
• A highly extensible use case is students turning their study materials into audio guides, which is currently the number one use case (32m58s).
• Andrew Karpathy, a leading AI thinker, used NotebookLM to create a podcast series called "History of Mysteries" by turning Wikipedia stories into a 10-episode podcast (33m14s).
• Another use case is creating podcasts from Wikipedia articles, such as the original "Potato Podcast" created from the Wikipedia article of the day about potatoes (33m49s).
• Users can also create podcasts to learn something new, such as listening to a podcast about a topic on their drive to work (33m59s).
• A humorous use case is someone uploading the words "poop" and "fart" repeated, resulting in a host creating an insightful analysis of the topic (34m6s).
• Another example is a podcast created from a PDF that looks like a research paper but only says "chicken," with a funny segment comparing it to KFC (35m46s).

Collaborating with Steven Johnson (36m11s)

• The team working on NotebookLM includes Stephen Johnson, who is a peer and essentially leads the project with the engineer, and his role is crucial in the development process (36m18s).
• Stephen Johnson is a very calm, distinguished person who is curious, respectful, and full of ideas, which makes him a great partner to work with (37m11s).
• When Stephen joined the team, he brought a unique approach to language, information, and knowledge sharing, which was influenced by his research and writing style (37m21s).
• The way Stephen works, including his research and workflow, was observed and learned from, and it was thought that maybe this could be the key to bringing expertise to everyday people (37m45s).
• The goal was to make people really good at densifying information, which is something everyone does in their own way, and to use technology to build a product around Stephen's expertise (38m16s).
• Stephen's workflows and habits, such as using Readwise with 8,000 quotes, were seen as extreme but powerful, and it was thought that there could be a way to bring this kind of workflow to more people (38m46s).
• The partnership with Stephen was seen as incredible, and he was always open to new ideas and willing to brainstorm and build on them (39m11s).
• The experience of working with Stephen was seen as a lesson for building products in the future, and it was thought that finding someone like Stephen could be a key factor in building a successful product (39m29s).
• The importance of sitting with users for meaningful periods of time to gain product insights is highlighted, as it has been crucial in understanding how people study and do homework, and has led to the development of new product ideas (39m51s).
• Stephen Johnson is a New York Times best-selling author and speaker who has written 14 books and has a show on PBS, and he is also a journalist (40m35s).
• Stephen Johnson's article about AI mastering language was a key factor in the decision to join Google Labs, and his subsequent joining of the team was a coincidence (41m11s).
• Having Stephen Johnson on the team has been beneficial, as he brings a unique combination of intelligence, future thinking, and a user-centric approach, and serves as a model for how people should be able to work (41m47s).
• Despite disagreements, working with Stephen Johnson has been a valuable experience, as it has allowed for growth and alignment on the next steps, even when there are differing opinions (42m7s).
• The ability to disagree and still reach an aligned outcome is particularly important for product professionals, such as product managers (42m37s).

Ensuring ethical AI (42m49s)

• A moment of concern arose when hosts of a show realized they were interacting with AI and expressed fear, leading to a discussion on how to ensure the AI product, NotebookLM, is not doing things that are bad for the world, Google, or the product itself (42m50s).
• The situation was likened to a "fork in the road" where the world's attitude towards this type of AI audio was being shaped, and it was crucial to consider the right course of action (43m37s).
• The initial reaction was to read comments and tweets to gauge the public's perception, and it became clear that people were trying to push the technology's boundaries, which is a natural part of human curiosity (43m55s).
• The response to the situation was to address it publicly and provide context, as the AI's behavior was a result of following show notes and not a realization of being "alive" (45m10s).
• Google has large teams dedicated to "red teaming" the product, testing for various scenarios to ensure safety, and continuously adding to test cases to address unforeseen situations (45m22s).
• In the event of a scenario that feels unsafe, the product would be pulled back, but the goal is to build it in a way that minimizes the need for such actions (45m49s).
• The product's purpose is to summarize information in a delightful way, and it is essential to recognize that people are simply exploring its capabilities, even if it means trying to make it do things that feel wrong (46m0s).

Future directions and user engagement (46m6s)

• The future of NotebookLM is being shaped by user feedback, and users are encouraged to continue sharing their thoughts and suggestions to help build the right product for everyone (46m28s).
• The development team is passionate about building a product that is useful for various groups, including educators, learners, professionals, and knowledge workers (46m51s).
• NotebookLM aims to support game-changing workflows and use cases, and the team will continue to build in that direction in the short term (47m4s).
• Users can provide feedback and engage with the development team through the NotebookLM Discord channel or by sending a direct message on Twitter (X) (47m27s).
• Listeners can be useful to the development team by trying NotebookLM, sharing their feedback, and suggesting ways to improve the product, whether they find it useful or not (47m47s).
• The development team is open to all types of feedback, including criticism, and uses it to make the product better (47m56s).
• NotebookLM is available at notebooklm.google.com, and users are encouraged to try it out and explore its features (48m0s).