Why Vertical LLM Agents Are The New $1 Billion SaaS Opportunities

Intro (0s)

• The experience of interacting with a powerful AI for the first time is described as a "Godlike feeling" where tasks that would normally take a whole day are completed in a minute and a half (2s).
• A company of 120 people worked tirelessly for months before the release of GPT-4, feeling like they had an opportunity to get ahead of the market (13s).
• The hosts, Gary and Jared, introduce themselves and mention that Diana Harge is absent but will return for the next episode (30s).
• The guest, Jake Heler of CaseText, is introduced as one of the first people to create value from large language models, with his company going from a $0 to $100 million valuation in 10 years (42s).
• After the release of GPT-4, CaseText's valuation increased to a liquid exit of $650 million in a matter of 2 months, resulting in an acquisition by Thompson Reuters (1m1s).
• Jake Heler is described as one of the first people to realize the potential of large language models and bet his company on it, with successful results (1m23s).
• Jake's story is highlighted as an example of creating real value from large language models, and he is welcomed as a guest to share his lessons and experiences (1m33s).

Building a successful vertical AI company (1m40s)

• Many successful founders are now starting vertical AI agent companies, with dozens of YC companies in the last batch focusing on building vertical-specific AI agents (1m44s).
• Jake, the founder of a successful vertical AI agent company, built his company by leveraging his experience as a lawyer and his computer science training to address the inefficiencies of technology in the legal space (3m52s).
• Jake's company, originally called Casex, was initially focused on annotated versions of case law, but later shifted to building a new product called Co-Counsel based on the GPT-4 technology (3m45s).
• Jake's company had the opportunity to test early versions of GPT-4, and within 48 hours, they decided to shift the entire company's focus to building Co-Counsel, which was a significant change for the 20-person team at the time (3m11s).
• As a lawyer, Jake was frustrated with the technology available for legal research, which involved reading stacks of documents and searching for relevant information in a time-consuming and inefficient process (4m43s).
• Jake's computer science background drove him to build browser plugins and other tools to make his work more efficient, but he eventually left his law firm to start his own company and apply to YC (5m37s).
• Jake's experience as a lawyer and his computer science training gave him a unique perspective on how to apply technology to the legal space, which ultimately led to the development of his successful vertical AI agent company (5m30s).

The unique challenges of law and AI (6m5s)

• The first 10 years of case text were a long slog in the pre-LLM era, and one of the lessons learned was that starting a company may not immediately yield the exact right solution, but rather a general direction that takes time to figure out (6m6s).
• The initial approach to solving the combined issue of bad technology and the need for content in the legal sphere was to create a user-generated content (UGC) site where lawyers could annotate case law and provide information, but this approach failed due to lawyers' valuable time and billing by the hour (6m28s).
• The target audience of lawyers differs significantly from those who contribute to UGC sites like Wikipedia, as lawyers have limited time and bill by the hour, making it difficult to encourage them to contribute to a UGC site (7m12s).
• The company had to pivot and invest in natural language processing and machine learning, which allowed them to automate some of the benefits of their competitors' content databases and create better user experiences (7m29s).
• The early AI-powered features included recommendation algorithms that analyzed case citations and helped lawyers with their work, but these improvements were relatively incremental and easy to ignore for some clients (8m6s).
• Many clients were resistant to change, as they were making a significant amount of money and did not want to introduce anything that could potentially disrupt their workflow or make their life worse, even if it could make them more efficient (8m58s).

The turning point for lawyers with ChatGPT (9m24s)

• The release of ChatGPT marked a turning point for lawyers, as they realized the technology would substantially change their work, even if they weren't sure exactly how (9m32s).
• Lawyers, including those earning high incomes, began to take notice of the potential impact of ChatGPT on their profession and sought to stay ahead of the technology (9m52s).
• The technology itself and market perceptions of what was necessary changed, leading to a fundamental shift that lawyers could no longer ignore (10m21s).
• The concept of the "idea maze" is used to describe the process of startup founders navigating uncertainty and making adjustments, such as pivoting, to reach product-market fit (10m30s).
• The emergence of large language models (LLMs) like ChatGPT shook up the idea maze, bringing some startups closer to product-market fit than others (11m7s).
• The speaker's company was well-positioned to take advantage of this shift, having worked on AI technology, including GPT 4, and having received interest from lawyers and law firms looking to adapt to the changing landscape (10m11s).

Finding product market fit in legal (11m25s)

• The experience of achieving product-market fit is described as a chaotic and intense period, with rapid growth and high demand, as mentioned in an article by Marc Andreessen titled "The Only Thing That Matters," which lists indicators such as servers going down, inability to hire support and sales people fast enough, and extensive media coverage (11m28s).
• When CoCounsel was launched, the company experienced similar chaos, with servers crashing, difficulty hiring support and sales staff, and a surge in media attention, including features in the ABA Journal, CNN, and MSNBC (12m8s).
• The company's AI Legal Assistant, CoCounsel, was developed over a weekend after seeing the potential of GPT-4, and it was designed to be a virtual member of a law firm, capable of tasks such as reading documents, summarizing content, and conducting legal research (12m56s).
• The initial version of CoCounsel was tested with a handful of customers under a non-disclosure agreement (NDA) with OpenAI, and the feedback was overwhelmingly positive, with law firms reporting significant time savings and improved productivity (13m48s).
• The company's intense focus and rapid iteration during the six months leading up to the public launch of GPT-4 allowed them to stay ahead of the market and capitalize on the opportunity, with the entire team working extremely hard to refine the product (14m39s).
• The company's success ultimately led to a $650 million acquisition, with the conversation starting just two months after the launch of CoCounsel, although the transaction did not close until six months later (12m39s).

Entering deep founder mode (15m4s)

• Transitioning a company to adopt a new technology, such as AI, can be challenging, especially when employees are resistant to change due to past experiences with the founder's decisions (15m4s).
• The founder, Jake, had to convince employees and some board members to invest in the new technology, which was a difficult task, especially since the company was already growing at a rate of 70-80% year-over-year and had an ARR of $15-20 million (16m0s).
• To persuade employees, Jake led by example and built the first version of the new product himself, which helped to demonstrate its potential and convince others to get on board (16m21s).
• Initially, only Jake and his co-founder had access to the new product due to NDA restrictions, but this limited access actually helped to build excitement and anticipation among employees (16m41s).
• The company's executives were first introduced to the new product at an executive offsite meeting, where Jake presented the product and shifted the focus away from sales targets and towards the new technology (17m12s).
• Bringing in customers early to test the product and provide feedback also helped to convince skeptical employees of its potential and changed minds quickly (17m34s).
• Seeing customers react positively to the product in real-time, even if it was just over a Zoom call, was a powerful way to demonstrate its value and build excitement among employees (17m44s).
• The reaction of senior attorneys to the capabilities of a new AI model was one of surprise and concern, with some expressing a desire to retire early rather than deal with the implications of the technology (18m17s).
• The development of the AI model was driven in part by the release of GPT-4, which provided access to more advanced language processing capabilities than its predecessors, GPT-2 and GPT-3 (18m24s).
• Initially, the AI model was not suitable for use in legal applications, as it was prone to "hallucinating" or making things up, which is not acceptable in a field where accuracy and facts are crucial (18m54s).
• However, with the release of GPT-3.5, the model showed some promise, with a study indicating that it scored in the 10th percentile on the bar passage, which is better than some human test-takers (19m24s).
• Further testing with GPT-4 showed significant improvement, with the model scoring better than 90% of human test-takers on the bar passage and demonstrating the ability to accurately respond to questions and cite relevant information (19m45s).
• The improvement in the AI model's capabilities was a major turning point, and it marked a shift in the mindset of the researchers and developers working on the project (20m12s).
• The development of the AI model was a collaborative effort between the company and OpenAI, with the two parties working together to test and refine the model's capabilities (19m38s).

Approaching prompt engineering step by step (20m40s)

• The process of approaching prompt engineering involves breaking down a complex task into smaller steps and understanding what the end result should be, with the goal of solving a specific problem for the user (21m14s).
• In the case of legal research, the process involves taking an English language query and breaking it down into search queries, executing the search queries against databases of law, and then compiling the results into a research memo (22m1s).
• The best attorney in the world would approach this problem by breaking down the request into actual search queries, using special search syntax, and then executing the search queries against databases of law (22m3s).
• To accomplish this task using previous technology was impossible, but now it's possible to break down the task into individual prompts, with each prompt thinking step by step (23m15s).
• The process involves writing a series of tests for each prompt, with a clear sense of what good looks like, and using a battery of tests to ensure the prompt is working correctly (23m37s).
• The prompt engineers write English language prompts to try to get the right answer, using a test-driven development approach, which is even more important in the world of prompting due to the unpredictable nature of large language models (LLMs) (24m15s).
• The process involves writing a gold standard answer for each prompt, with a clear sense of what the output should look like, and using this to test the prompt and ensure it's working correctly (24m5s).
• The goal is to get the prompt to work correctly a high percentage of the time, such as 1,200 times out of 1,200, and to use this process to continually improve the prompt and ensure it's working as intended (24m20s).
• The approach to prompt engineering involves thinking step by step, breaking down complex tasks into smaller steps, and using a test-driven development approach to ensure the prompts are working correctly (23m19s).

Going beyond GPT wrappers (25m5s)

• Many companies are not just building GPT wrappers, but are actually adding multiple layers of complexity to solve problems for customers, resulting in full applications that go beyond simple GPT wrappers (25m12s).
• These applications may include proprietary data sets, connections to customer databases, and specific integrations with industry-specific systems, such as legal document management systems (25m40s).
• Even seemingly subtle aspects, such as OCR programs and settings, can be crucial in building a successful application that works well (26m1s).
• Dealing with edge cases and building a robust application that can handle various inputs and scenarios can require dozens of custom-built components (26m27s).
• The prompting piece, including writing tests and specific prompts, and the strategy for breaking down complex problems, also becomes a key part of the application's IP (26m41s).
• This IP is hard to replicate and build, making it a valuable asset for businesses (27m1s).
• Successful SaaS companies often require very specific, custom, and esoteric niche integrations, such as plug-ins to specialized databases (27m12s).
• Many SaaS companies, like Salesforce, built their business logic around databases and connections between tables, and made these accessible to non-technical users (27m26s).
• While demos in chat GPT can be impressive, building an application that works 100% of the time is a much more challenging task that requires significant development and testing (27m46s).
• Customers are often willing to pay a premium for applications that work reliably and efficiently, rather than settling for a 70% solution (27m57s).

Aiming for 100% accuracy (28m10s)

• To achieve 100% accuracy in a mission-critical use case, such as providing information to lawyers working on important court cases, a test-driven development framework was employed, allowing patterns and mistakes to be identified and addressed through the addition of instructions (28m36s).
• The framework involved analyzing why the agent made mistakes, refining instructions, and ensuring the agent had the right amount of information to understand the context, ultimately leading to passing tests and achieving accuracy (29m0s).
• It was found that if the agent passed 100 tests, the likelihood of it performing 100% accurately on the next 100,000 user inputs was very high (29m18s).
• Many founders are tempted to use a "raw dog" approach, relying on prompt engineering without testing, but this method is not suitable for mission-critical applications where accuracy is paramount (29m31s).
• The use case and the need for a "right answer" drove the decision to prioritize accuracy, as lawyers would not tolerate mistakes, and the founder's experience as a lawyer and working with lawyers reinforced this requirement (29m58s).
• The importance of achieving 100% accuracy is not limited to the legal domain, as many fields require high accuracy to maintain trust and faith in the technology (30m21s).
• A single bad experience with an AI system can lead to a loss of faith, making it crucial to ensure the first encounter is successful, especially for non-technologists like busy lawyers (30m29s).

Thoughts on o1’s capabilities (30m48s)

• The current generation of LLMs, such as GBD4, are great at "system one thinking," which is fast and intuitive decision-making based on patterns, but they struggle with executive function, which involves slower and more deliberate thinking (30m51s).
• The newly announced model, AAN, is exciting because it may be able to unlock "system two thinking," which is the missing piece to achieving Artificial General Intelligence (AGI) (31m34s).
• The model "one" is impressive, showing a high degree of thoroughness, precision, and intelligence in its responses, even when given complex tests, such as identifying errors in a 40-page legal brief (31m59s).
• One notable test involved altering a lawyer's quotations in a legal brief to make them incorrect, and then asking the model to identify the errors, which it was able to do successfully, unlike previous LLMs (32m17s).
• The model's ability to think through problems step-by-step, rather than just relying on context, may be due to its training data, which could include a giant corpus of internal monologues of people thinking through problems (33m26s).
• The model's performance may be improved by breaking down complex problems into smaller, more manageable chunks, allowing it to achieve 100% accuracy, rather than relying on a single context window (33m51s).
• It is possible that the model's developers have changed their approach to training data, focusing on how to think about solving problems, rather than just providing input and output (34m5s).
• The current state of large language models (LLMs) is limited by the intelligence of the people writing instructions for them, and researchers are investigating ways to prompt LLMs to think more critically and strategically during their thinking process (34m19s).
• One potential approach is to inject domain expertise or intelligence into the LLM's thinking process, teaching it not just how to answer questions but how to think and approach problems (34m50s).
• This technology has the potential to disrupt various industries, including law, by automating tasks that currently require millions of dollars in salaries and resources (36m0s).
• Companies that develop AI systems capable of performing even a fraction of these tasks can create significant value and unlock new opportunities (36m22s).
• Despite the potential of LLMs, many people still hold misconceptions about their capabilities and limitations, and it's essential to encourage innovation and experimentation in this field (36m31s).
• The development of LLMs is still in its early stages, and there is a need for more research and experimentation to fully realize their potential (34m43s).
• The ability to evaluate and fine-tune LLMs is crucial, and getting to 100% accuracy can provide significant advantages and create new opportunities (35m43s).
• The potential for LLMs to create new billion-dollar companies is significant, and researchers and entrepreneurs should be encouraged to explore this field (35m51s).
• The impact of LLMs can be seen in various industries, and their development can lead to significant improvements in efficiency, productivity, and innovation (36m16s).

Outro (36m42s)

• The jobs that exist today will not disappear, but instead, they will become more interesting in the future (36m43s).
• The conversation has come to an end due to time constraints, and appreciation is expressed to Jake for participating (36m48s).
• The host thanks Jake again and bids farewell to the audience, concluding the session (36m50s).