The 10 Trillion Parameter AI Model With 300 IQ

Coming Up (0s)

• The concept of "o1" being magical raises questions about its implications for Founders and Builders, with one argument suggesting it could be detrimental to Builders as Open AI might capture all the value with its powerful technology (3s).
• This scenario implies that Open AI would capture a "light cone" of all present, past, and future value, potentially dominating the market (15s).
• A more optimistic alternative scenario suggests that with o1 becoming more deterministic and accurate, Founders can focus on higher-level tasks, such as building the best user experience, rather than spending time on getting prompts to work correctly and outputs to be accurate (30s).
• In this optimistic scenario, the winners would be those who build the best software, focusing on details that create a seamless user experience (38s).

What models get unlocked with the biggest venture round ever? (54s)

• OpenAI has raised the largest venture round ever, $6.6 billion, which will be used for compute, talent, and normal operating expenses, with a focus on scaling up their models (55s).
• The next model is expected to be an order of magnitude bigger, with 10 trillion parameters, two orders of magnitude out from the current state-of-the-art (1m39s).
• The current frontier models have roughly 500 billion parameters, such as LLaMA, Anthropic, and GPT-4, and getting to 10 trillion parameters would be a significant leap (2m7s).
• This leap could be similar to the one seen from GPT-2 to GPT-3.5, which was a two-order magnitude increase and led to a new flourishing era of AI companies (2m30s).
• If this scaling law continues, it could lead to a similar feeling of transition and wealth creation, as seen in 2023 when companies started building on top of GPT-3.5 (3m24s).
• The current state-of-the-art models already rival normal intelligence and can perform tasks with 90-98% accuracy, similar to a human knowledge worker with 120 IQ (3m50s).
• At 10 trillion parameters, the models could reach 200-300 IQ, unlocking new capabilities beyond what a normal human being can do, and potentially leading to breakthroughs in various fields (4m35s).
• There are examples of this happening in human history, such as the development of nuclear power, where theoretical modeling unlocked new capabilities (5m14s).
• The article in The Atlantic featuring Terence Tao, a Taiwanese mathematician with an IQ north of 200, shows how ChatGPT is already unlocking new capabilities for him (4m52s).

Some discoveries take a long time to actually be felt by regular people (5m35s)

• The impact of AI is not yet evenly distributed, and many people do not feel its effects in their daily lives, but some discoveries take time to pan out and have a significant impact (5m36s).
• The example of the Fourier transform, a mathematical representation discovered by Joseph Fourier in the 1800s, is given as a counterexample, which took 150 years to be fully utilized and have a significant impact on the world (6m0s).
• The Fourier transform was initially a seminal thesis that represented series of functions that were repeating in periods, and it was later found to be super good at representing signals, which is crucial for representing everything in the analog world in a digital format (6m5s).
• The Fourier transform has numerous applications, including radio waves, telecommunication, image representation, encoding, and information theory, which have unlocked much of the modern world, including the internet and cell towers (7m7s).
• The average person could feel the impact of the Fourier transform 150 years after its discovery, and it is interesting to consider how long it will take for the average person to feel the impact of current AI developments (7m35s).
• The development of color TV in the 1950s is also attributed to the Fourier transform, and it is possible that current AI developments will have a similar impact in the future (7m51s).
• The start of the clock for current AI developments is unclear, but it is possible that we are already decades into this development and are just starting to hit the inflection moment (7m57s).
• The math underlying current AI developments, such as linear algebra, is over 100 years old, but the availability of GPUs has enabled the computation of complex models, which could potentially alter the face of what humans are capable of (8m27s).
• The impact of 10 trillion parameter models could be significant, potentially unlocking something about the nature of reality and our ability to model it, but it is unclear what the outcome will be (8m39s).
• The fact that current AI developments are in software, as opposed to physical devices, means that the technology can be adopted more quickly, and companies like Facebook and Google already have a significant percentage of the world using their software (9m13s).
• The development of consumer devices, such as Meta's Ray-Ban smart glasses, could be a significant moment for the widespread adoption of AI, enabling people to interact with AI in a more visual and conversational way (9m28s).

Distillation may be how most of us benefit (9m53s)

• There is a bifurcation in the expectations of what can be achieved with the capability of 10 trillion parameter models, with some people pushing the edge of understanding and others benefiting from distillation, which is taking a large model and making it smaller and cheaper to run (9m54s).
• There is evidence that large models like Meta's 405B are mostly used to make smaller models better, and this process of distillation is being used by companies to make their models more efficient (10m23s).
• OpenAI has enabled distillation internally to its own API, allowing users to distill large models like GPT-4 into smaller, cheaper models like GPT-4 Mini (10m51s).
• This process of distillation is becoming more popular, with companies choosing to use smaller models over larger ones, and startups are building their products on top of these smaller models (11m50s).
• The trend of using smaller models is continuing, with Claude going from 5% to 25% developer market share in just six months, and Llama going from 0% to 8% (13m11s).
• The use of smaller models is a good predictor of what successful companies will use, and therefore what products will be most successful (13m27s).
• OpenAI was losing market share to other models, but may be coming back with their latest model, with 15% of the batch already using it despite it being only two weeks old (14m13s).

o1 making previously impossible things possible (14m26s)

• A hackathon is being hosted to give YC companies early access to 01, with Sam himself kicking off the event, and already some teams have built things that were not possible before with any other model (14m28s).
• One of the companies, Freestyle, is building a cloud solution fully built with TypeScript and has already seen a demo of a version of Repet Agent working with their product after just a couple of hours of work (15m0s).
• The demo was able to reason and inference with the documentation and built a web app that writes a to-do list after being prompted with developer documentation and code (15m38s).
• There are two arguments about the impact of 01 on founders and builders: one is that it's bad for builders because Open AI will capture all the value, and the other is that it will make it easier for founders to build things by making the process more deterministic and accurate (15m57s).
• The more optimistic scenario is that founders will spend less time on tooling and getting prompts to work correctly and more time on building better UI, customer experience, sales, and relationships (16m25s).
• This could make it a better time to start building now, as the knowledge learned around getting prompts accurate and working may become less relevant as these models get more powerful (17m0s).
• A conversation with Jake Hel from Casex highlighted the challenges of getting the legal copilot to work to 100% accuracy, but with 01, it's possible that this process could become much easier (17m9s).
• If 01 can guarantee 100% accuracy from day one, the barrier to entry to build these things will go way down, leading to more competition and a more traditional winner-takes-all software market (17m28s).
• An example of a company, Dry Merch, that went from 80% accuracy to almost 100% using 01 and unlocked a bunch of new things, is cited as evidence of the potential of 01 (17m42s).
• There may be an even more bullish version of this scenario, which is that there are use cases right now that people are not able to build because of the limitations of current models (18m7s).
• Large language models (LLMs) are not yet accurate enough to be used in mission-critical jobs where the consequences of stakes are dire, but as they improve, they will become more suitable for such applications (18m13s).
• A company in the YC portfolio, which was not profitable and growing 50% a year, automated about 60% of their customer support tickets using LLMs and became cash flow break-even while still growing 50% year on year (18m40s).
• This scenario is considered a dream for building enterprise value, as the company is compounding its growth with no additional capital coming in, and it has the potential to become a billion-dollar revenue company driving hundreds of millions of dollars in free cash flow (19m12s).
• The current overhang moment in 2024, where companies raised too much money at high multiples, is actually good news for them as they can go from not profitable to break-even to potentially very profitable (19m50s).
• Cler, a company, got attention for pitching that they are replacing their internal systems of records for HR and sales with home-built LLM-created apps, including getting rid of Workday (20m29s).
• Treating OpenAI as the Google of the next 20 years, it is desirable to invest in OpenAI and the things it enables, similar to how Google enabled companies like Airbnb (20m52s).
• However, it is unlikely that OpenAI will create its own Airbnb-like company due to the inefficiency and difficulty of doing so, requiring too much domain expertise (21m7s).

The new Googles (21m17s)

• New companies, referred to as the "new Googles," are being built, utilizing vertical agents, with examples including Tax GPT, a company that started off providing tax advice and has now expanded to building an Enterprise business on document upload, offering a low-cost or free service initially and then securing high-value contracts (21m18s).
• Tax GPT began as a rapper and leveraged its ability to perform research and analysis on existing case law and policy documents from the IRS or internationally, gaining traction among tens of thousands of accountants and accounting firms (21m29s).
• The company's business model involves offering a low-cost or free service initially, then securing high-value contracts, with an average contract value (ACV) of $10,000 to $100,000 per year, which can significantly reduce the workload of accountants (22m3s).
• The development of new AI models, such as the 01 model, is a significant step forward, offering a major breakthrough for any program or engineer building AI applications, and it is unclear whether this will give Open AI a temporary lead in market share or a more permanent advantage (22m44s).
• The AI landscape is rapidly evolving, with multiple models, including Llama, Claude, and Gemini, continually improving, and it is uncertain whether Open AI's breakthrough will be replicable by others or if it will maintain its lead (23m11s).
• Open AI has consistently been at the forefront of innovation, making major breakthroughs, but has yet to maintain its lead in the market, and it remains to be seen whether its latest development will be a true breakthrough, defensible and non-replicable by others (23m36s).

o1 makes the GPU needs even bigger (23m47s)

• AAN makes GPU needs even bigger due to increased computation needs for inference, taking more time and changing the dynamics for companies building AI infrastructure (23m47s).
• There are two types of use cases: one where AAN can be used for difficult tasks and then distilled into A40 or A40 mini for relatively rote and repeating use cases, and another where the full AAN experience is needed for specific and detailed tasks (24m10s).
• The cost of using AAN can be passed on to customers if the company is an enterprise software and the customers can tolerate higher latency, but this may not be feasible for consumer apps (24m52s).
• Open AI's Lace releases, such as the real-time voice API, are remarkable and have ongoing usage-based pricing of $9 per hour, which could be a threat to countries that rely heavily on call centers (25m16s).
• The $9 per hour pricing for the real-time voice API is comparable to the cost of a call center, making it a potentially powerful tool that could disrupt the industry (25m35s).

Voice apps are fast growing (25m44s)

• A company that specializes in AI voice for deck collection has shown phenomenal growth and success, with its track record being incredibly well. (25m54s)
• Many voice apps in S24 have experienced explosive growth, marking a clear trend for that batch. (26m3s)
• In previous batches, companies working on voice apps faced challenges such as high latency, confusion with interruptions, and other issues, but it seems that these problems have been overcome. (26m18s)
• A company called Happy Robot has developed a voice agent for coordinating phone calls in logistics, helping to streamline the process for truck drivers and others in the industry. (26m28s)
• The voice agent has gained significant usage and is solving menial problems over the phone, demonstrating the capabilities of AI in passing Turing tests. (26m51s)

Incumbents aren’t taking these innovations seriously (27m5s)

• Many engineering teams in incumbent industries do not take innovations in AI seriously, with most companies having no initiative on this front, and this lack of initiative may be generational, with managers and VPs being less aware of the rapid improvements in AI (27m5s).
• The rate of improvement in AI is not well understood by those not closely involved in the field, with many people being cynical about its potential impact, similar to how they viewed the cloud, which took around a decade to change enterprise software (27m49s).
• Even people within the tech industry are surprised by how quickly AI is moving, with its improvement being the fastest of any tech, surpassing processors and the cloud (28m41s).
• Technical founders are using the latest coding tools, such as coding assistants, with many using Cursor, a tool that is more advanced than GitHub Co-Pilot, and half of the summer 24 Founders surveyed are using Cursor, compared to 12% using GitHub Co-Pilot (29m39s).
• The use of these tools is allowing founders to build more polished demos and products, with some using the latest agented coding agents, and this is evident in the hackathons, where founders are building impressive projects (30m5s).
• The advantage of using these tools is giving founders an edge, with Cursor being a notable example, having come out of nowhere and gained significant traction, with five times the size of GitHub Co-Pilot within the batch (30m33s).
• The adoption of these tools by startup founders is a good indicator of their potential success, with companies like Stripe and AWS having successfully targeted YC batches as early customers (30m52s).
• The competition between these tools is ultimately benefiting developers, with the winners being those who build the best user experience and get the details correct, and this is why Cursor can compete with GitHub Co-Pilot despite the latter's advantages (31m29s).

Ten trillion parameters (31m52s)

• A world with a 10 trillion parameter AI model could potentially lead to significant advancements in scientific and technological progress (31m53s).
• The current rate of progress is arguably limited by the number of smart people who can analyze the vast amount of existing information, including millions of scientific papers and data (32m16s).
• If AI models become smart enough to perform original thinking, deep analysis, and correct logic, they could potentially unlock a near-infinite amount of intelligence to process the vast amount of data and knowledge about the world (32m37s).
• This could lead to groundbreaking scientific discoveries, such as room temperature fusion, room temperature superconductors, time travel, and flying cars, which humans have not been able to invent yet (32m58s).
• With enough intelligence, it is possible that these inventions and discoveries could finally be made, leading to a potentially transformative future (33m9s).

Outro (33m15s)

• The concept being discussed is not just a simple tool, but something more complex and powerful, likened to a self-driving car or even a rocket to Mars, indicating its potential to greatly exceed human capabilities (33m15s).
• The idea is not just a basic aid for the mind, but rather a highly advanced technology that can significantly surpass human intelligence (33m18s).
• The possibilities of this technology are vast and potentially revolutionary, with the comparison to a rocket to Mars suggesting its ability to greatly accelerate progress and achieve groundbreaking advancements (33m24s).
• The discussion will be continued in a future video, with the current one coming to a close (33m27s).