Stanford Webinar - Large Language Models Get the Hype, but Compound Systems Are the Future of AI

Large Language Models and the Rise of Compound AI Systems

• Large language models receive significant attention, but compound systems are the future of AI, and this concept is often overlooked due to the focus on models in headlines (11s).
• The trend of emphasizing large language models began with the GPT-3 paper, which introduced a 175 billion parameter model, an order of magnitude larger than previous models, and demonstrated the potential of scaling to create new systems (56s).
• The announcement of Google's Palm model, with 540 billion parameters, is another example of this trend, where the focus is on the model rather than the system it is part of (1m30s).
• Even companies like Open AI, which focuses on full software systems, tend to announce new models, such as GPT-4, rather than the systems they power (2m7s).
• The emphasis on models can create a misconception that they are the primary focus, but in reality, people only interact with systems, not individual models (42s).
• A large language model is inert on its own and requires additional components, such as a prompt and a sampling method, to function as a system (2m57s).
• The choices of prompt and sampling method are non-trivial and play a crucial role in creating a functional system (3m49s).
• Compound systems, which include models like Gemini and ChatGPT, are already being developed and used, but are often described in terms of their models rather than the systems as a whole (1m50s).
• A minimal system in AI consists of a prompt, a model, and a sampling method, but modern systems are taking this further by giving these minimal systems access to calculators, programming environments, databases, web APIs, and the web itself, making them software systems with the language model as a hub (4m23s).
• The capabilities of these systems are defined by how all components work together, not solely by the language model, which is the essence of the thesis that compound AI systems are the future of AI (4m43s).

The Shift from Models to Systems

• This idea has been explored in a blog post titled "The Shift from Models to Compound AI Systems" from February 18, 2024, which emphasizes the importance of thinking in terms of systems rather than just models (5m0s).
• The idea of shifting from models to systems has been echoed by others in the field, such as Sam Altman, who recently mentioned expecting a shift from talking about models to talking about systems (5m31s).
• Investing in the right things when developing an AI solution requires thinking in terms of systems, rather than just focusing on the model, which is analogous to designing a Formula 1 race car where all components must work together to achieve success (6m16s).
• Focusing solely on the model, like focusing solely on the engine of a Formula 1 race car, will not lead to a good overall system, and it's essential to think in terms of the entire system (7m26s).
• Building the best system for specific goals and constraints will likely emphasize system components, and a small model embedded in a smart system will always be better than a big model in a simplistic system (7m48s).
• When designing AI systems, considerations such as cost, latency, safety, and privacy are crucial, and in such cases, a small model might be the only viable choice, especially when cost is a significant factor (7m58s).
• There is a need to shift the focus from regulating model artifacts to regulating entire systems, as focusing solely on models may lead to electing dangerous systems and overregulation (8m30s).

Sampling Methods and Their Importance

• The method used for sampling when generating output from a model is a critical component of the system, and there are various methods available, including greedy decoding, top-p sampling, beam search, and insisting on token diversity (9m12s).
• These sampling methods can be used to impose high-level ideals, such as ensuring generated output conforms to valid JSON or a specific grammar, and there are innovative ideas in the literature that make use of gradients and other techniques to achieve this (10m9s).
• Researchers are exploring new methods to improve sampling, including adding parameters to the model to adaptively find a good temperature for creative or constrained output, depending on the task (11m1s).
• The concept of sampling can be expanded to include creative exploration, such as majority completion strategies, which involve using the model to complete a prompt with a hard reasoning task (11m31s).
• Large language models can generate answers in one step, but it might be more effective to have them generate a reasoning path and then produce an answer, allowing the model to explore and reason before providing a final answer (11m46s).
• By sampling multiple reasoning paths, the model can produce a distribution of answers, and the most common outcome can be considered the final answer, giving the model the ability to explore and reason before producing a final answer (11m56s).
• This approach is not strictly a sampling strategy, but rather a way to let the model explore and reason before producing a final answer, and it can be made to look like a simple sampling strategy by hiding the intermediate steps from the user (12m27s).
• There is no one true sampling method for a model, and the choice of sampling method is highly consequential for the overall system, as it is what makes the model "speak" and can interact in complex ways with the language model (13m0s).

Prompting: The Heart of Modern AI Systems

• Prompting is a crucial aspect of modern AI system development, and it is the heart of what makes these systems work, allowing for complex and interesting behaviors to be achieved through careful engineering of prompts (13m38s).
• The origins of prompting in AI systems can be traced back to the GPT-2 paper from 2019, which demonstrated that language models can perform downstream tasks in a zero-shot setting without any parameter or architecture modification (14m12s).
• The GPT-2 paper showed that by adding a simple prompt, such as "TL;DR", to the input text, the model could be induced to perform tasks such as summarization, translation, question answering, and reading comprehension (14m36s).
• The concept of large language models has undergone significant development, from GPT-2 with 1.2 billion parameters to GPT-3 with 175 billion parameters, showcasing the impact of scaling on complex in-context learning (15m43s).
• GPT-3 demonstrated successful question-answering capabilities by learning to imitate behavior from a prompt with a context passage, demonstrations, and a target question, leading to striking behavior where the model answers questions as substrings of the passage (16m28s).
• The GPT-3 paper identified a general pattern for this behavior, consisting of context or instructions, a list of demonstrations, and a target, which was applied to various tasks such as QA, reading comprehension, and machine translation (16m41s).
• A template for building systems in this modern mode has been established, allowing for exciting possibilities, but also highlighting the potential dark side of prompting, including sensitivity to prompt formatting (17m20s).
• Research has shown that language models can be highly sensitive to prompt choice, with minor changes leading to significant differences in performance, emphasizing the importance of considering the model-prompt combination when evaluating a model's capabilities (18m12s).
• The concept of systems thinking is essential in this context, where the focus should be on finding the optimal prompt-model combination to achieve specific goals, rather than evaluating the model in isolation (18m59s).
• The idea of Chain of Thought reasoning has been explored in papers such as "Echo prompt," which empirically examines different strategies for step-by-step reasoning, highlighting the importance of considering the prompting strategy when evaluating a model's capabilities (19m17s).
• The performance of a model can vary greatly based on how a question is framed, as shown by the results of an experiment with the Da Vinci 2 model, highlighting the importance of considering model-prompt combinations when evaluating a model (19m38s).
• Evaluating a model requires thinking in terms of model-prompt combinations, which is a systems-level approach rather than a model-level approach, and can bring clarity to development cycles (20m7s).
• A tweet about a person's experience with updating a model from GP4 to 40 and having to re-engineer prompts highlights the inextricable link between models and prompts (20m25s).
• Prompts are not just written in English, but are an effort to communicate with the language model, and understanding this link is crucial for developing effective systems (20m57s).
• The Apple intelligence prompts found in system files demonstrate the tightly knit relationship between prompts and models, with prompts being more like compiled binaries meant to be paired with a particular model (21m41s).
• The dspi library is being promoted as a tool for developing systems that take into account the interaction between models and prompts, and adopting a systems-level approach to AI development (22m37s).

From Prompt Engineering to Language Model Programming with DSP

• The core lessons of artificial intelligence, including modular system design, data-driven optimization, and generic architectures, have been successful in part because they have been adopted throughout the field (22m51s).
• The development of AI systems has moved rapidly, especially in the deep learning era, leading to the creation of exciting large language models, and libraries like Torch, Theano, Chainer, and PyTorch have embodied time-tested lessons that have helped speed up development (23m7s).
• However, the current moment in AI system development is characterized by a lot of manual adjustments to prompts, resulting in complete model dependence, which is tragic given the progress made by time-tested lessons (23m34s).
• DSP is a model and programming library that aims to move away from prompt engineering and towards language model programming, honoring the insight that a prompt, language model, and sampling strategy together design a software system (24m3s).
• DSP allows users to define a system using high-level tools and then compile it down into an actual prompt to a language model, abstracting away some model dependence (24m34s).
• A simple example of DSP is a minimal system for doing basic question answering, which can be achieved with just one line of code, and the system gets compiled down into an actual prompt to a language model (24m56s).
• More complex programs can also be written in DSP, such as a program for multihop question answering, which allows users to freely express the kind of system they want to develop in code (25m43s).
• The design principles of DSP are tightly integrated with PyTorch, and the system can be optimized to find a successful prompting strategy independent of the chosen tools (26m0s).
• The optimizer in DSP can simultaneously optimize the instructions and few-shot demonstrations, moving the burden of finding good ways of doing that onto the optimizer (26m29s).

Evaluating AI Systems: A Framework and Case Study

• A framework for evaluation is presented, consisting of a program, an optimizer, and a language model, to specify a complete system and demonstrate the importance of thinking in systems terms (27m7s).
• The baseline system, which goes from questions to answers, achieves a score of 34 for Turbo and 27.5 for LLaMA 23B, with the metric being exact match on the desired answer (27m47s).
• Adding a simple DSP program for retrieval-augmented generation results in a boost in performance, and using bootstrap few-shot optimization leads to significant gains, with scores increasing to 42 and 38 (28m23s).
• The use of react agents, which enables the model to reflect and think about how to solve the task, shows less success but still demonstrates the power of systems thinking (28m36s).
• Human-written reasoning prompts underperform compared to prompts generated through simple bootstrapping, highlighting the power of data-driven optimization (28m58s).
• A program designed for multihop reasoning, which gathers evidence from multiple passages, achieves high scores, with Turbo reaching almost 55 and LLaMA 23B reaching 50, demonstrating the power of intelligent system design and small models (29m37s).

The Importance of System Design for Small Models

• The importance of designing systems that can get the most out of small models is emphasized, as 77% of enterprise usage of models is at the 13 billion parameter size or smaller, according to an analyst at Theory Ventures (30m16s).
• In industrial systems, latency is a significant concern, with ideal latency being around 18 milliseconds, but as latency increases above 50 milliseconds and up to 750 milliseconds, it can become expensive and potentially prohibitive (30m35s).
• To get the most out of small models, it's essential to think about the systems being designed around them, with the prompt being a crucial factor in system performance (31m19s).

Tool Access and System Consequences

• Tool access is an area where entire systems, not just models, are considered, involving calculators, programming environments, databases, the web, and web APIs (31m36s).
• When designing systems, it's essential to consider the overall consequences, such as reliability, preference, and danger, rather than just focusing on technical details (31m57s).
• A giant large language model with a snapshot of the entire web may be less reliable than a tiny language model working with an up-to-date web search engine (32m20s).
• A small model doing autocomplete tasks locally on a phone may be preferred over a giant large language model doing contextless autocomplete via a centralized service (32m41s).
• A 10 billion parameter language model with access to databases and the web may be more dangerous than GP4 with no access to these tools (33m5s).

The Future of AI: Systems, Not Just Models

• In 2026, it's expected that systems consisting of multiple models and tools working together will be more prevalent than massive foundation models doing everything in terms of their parameters (33m38s).
• Recent legislation, such as SB147, has attempted to regulate models based on their size, with a focus on models that cost over a million dollars to train and have 10 to the 26 flops performed during training (34m1s).
• The concept of regulating large language models is being explored, with the idea that smaller specialized models may emerge as equally or more dangerous than larger models, and that complex systems composed of these models could be more hazardous than a single expensive model (34m35s).
• The focus of future legislation should be on regulating systems rather than individual models, as systems are the entities that can cause harm (35m26s).

Rethinking Evaluation: Focusing on Systems, Not Just Models

• Current research evaluations, such as leaderboards, are flawed as they only evaluate individual models, whereas in reality, a system consisting of a model, prompting strategy, and generation procedure is being evaluated (35m40s).
• The community should reorient leaderboard evaluations to focus on entire systems, considering all components working together, rather than just individual language models (36m42s).

Scaling Up: Multiple Dimensions and the Power of Systems

• The future of AI may involve multiple notions of scaling, including scaling up unsupervised training and scaling up instructive fine-tuning, with the latter driving significant gains in AI progress (37m21s).
• The power of large teams of humans creating good input-output pairs for model updates has been demonstrated, particularly with the emergence of chat GPT in 2022 (38m4s).
• Scaling up large language models is leading to gains, but it's not a silver bullet, and this has led to a rise in the theme of scaling sampling for generation, which involves sophisticated forms of sampling for generation that can be thought of as scaling up inference time processing search (38m24s).
• This theme is expected to continue from 2024 onward, with transformative things happening in virtue of the fact that perfectly good language models, even small ones, are given access to lots of different tools and other things that make them really capable as systems (38m54s).
• The future of AI lies in the scaling up of systems, where language models are given access to various tools and capabilities, making them more productive and leading to bigger gains (39m0s).

Generative Agents and the Importance of Systems Thinking

• Generative agents play a crucial role in this whole thing, as they can be used to take a language model and have it do things it couldn't do on its own, and also make use of tools and tool output (40m5s).
• Systems thinking is essential, and people should not be purists when designing software systems, as they can design agents that depend on the model doing complex things in generation or write code to bridge the gaps between the language model's capabilities and the desired outcome (40m31s).

Reasoning Paths and Transparency in AI Systems

• Modern AI systems are already producing multiple reasoning paths in the background, and the system produces reasoning paths through methods like Chain of Thought, which involves prompting the model to think step by step and generate tokens in response (41m5s).
• To give users more access to the inner workings of these systems, it's essential to understand how the system produces reasoning paths and provide users with more transparency and control over the evaluation of results (41m13s).
• The concept of Chain of Thought reasoning involves generating multiple inference paths to arrive at different outcomes, and then statistically analyzing those outputs to decide on a final generation, which is a systems thinking approach that considers the prompt, overall system structure, and generation methods (42m0s).
• This approach is exemplified in Open AI's 401 models, which perform extensive inference time work before producing an answer, but the details of these processes are considered trade secrets (42m47s).
• Researchers can explore the behaviors of smaller models to understand how to coax out desired behaviors, and this topic is expected to be explored in the research literature going forward (43m16s).

The Future of System-Level Scaling and Tool Access

• The future of system-level scaling will involve increasingly complex systems, similar to Google search, which has evolved from a simple search technology to a complicated software system that functions through teams of people and dynamic behavior (43m41s).
• The development of geni systems will lead to incredible systems being built over the next decades, and a key aspect to watch is when people provide tool access, allowing language models to interact with the web and other systems like humans do (44m17s).
• As language models become more integrated with the web and other systems, they will have significant consequences, both productive and problematic, and it is essential to establish guardrails to mitigate potential negative consequences (44m51s).
• Establishing guardrails requires considering both positive and negative consequences and thinking about how to regulate and manage the development and deployment of these systems (45m28s).

Regulating AI: Systems vs. Models and Human Considerations

• Regulating AI systems is more likely to be effective than regulating the models themselves, as there is already existing legislation that governs software system behavior, which can be applied to the AI realm (45m39s).
• Considering human aspects, such as requiring AI systems to identify themselves as non-human, can help people calibrate to them as agents and control the situations they are allowed to enter (46m1s).
• Fundamental restrictions may be necessary, such as limiting AI models' ability to log into certain websites or interact freely on social networks (46m20s).

The Near-Term Impact of AI and the Need for Vigilance

• The initial disasters caused by AI systems may not be cataclysmic, allowing society to learn from them and figure out how to respond (46m39s).
• AI will impact people's lives in the next five years, even if they are not thinking about it, as more systems that can help with daily tasks and provide companionship, discovery, and creative expression will emerge (47m11s).
• AI can also help with education, providing customized experiences at low costs, but there is a risk of bad actors using AI for malicious purposes, such as social engineering (47m54s).
• Individuals and society need to be on the lookout for AI systems and take steps to prevent malicious activities (48m29s).

Getting Involved in the AI Community and Learning Resources

• For those interested in learning more about AI, there are resources available, such as a Discord community and tutorials for technical individuals, and recommendations for business leaders to grasp the implications of AI for their businesses (48m44s).
• To get involved in the community and contribute to projects, one can start by filing issues or making pull requests, which can lead to positive impacts and learning opportunities, and joining the Discord community to see what others are working on and potentially collaborate with them (49m19s).
• The research community is also open to new contributors, and YouTube can be a helpful resource for learning about different prompting strategies, agent tool usage, and other relevant topics (49m46s).

The Changing Landscape of AI Research and Information Access

• However, the industry is becoming increasingly closed, making it harder to gain insight into the decisions being made and why, even at the level of research innovation (50m4s).
• For leaders in organizations looking to define a generative AI strategy, it's essential to think about what success looks like, what kind of testing to do, and designing a system that balances goals with known risks (50m21s).
• To stay up-to-date with the field, one can dedicate 10 minutes a day to reading and learning, and recommended resources include the ACL Anthology for NLP papers, Semantic Scholar, and tools like ChatGPT that can do retrieval-augmented generation (51m14s).
• Twitter, or X, is no longer as reliable a resource as it once was due to changes, and communities have spread out to other platforms like Bluesky, Threads, and Mastodon (51m48s).
• The rise of generative AI has made it easier to get a sense of an area by typing common-sense questions into search engines like ChatGPT, which can provide a starting point for learning and exploring the literature (52m4s).
• NLP is a well-organized community with a comprehensive literature, and resources like the ACL Anthology and Semantic Scholar can be used to find relevant papers and stay current (52m42s).
• A course on natural language understanding is available, which includes a project development phase that involves building a literature review, forming an experimental protocol, writing a paper, and creating associated code, providing a guided way to do a focused research project and understand the rhythms of research in the domain (52m55s).

Building Generative AI Strategies and Staying Up-to-Date

• For leaders in organizations looking to define a generative AI strategy, it's essential to think about what success looks like, what kind of testing to do, and designing a system that balances goals with known risks (50m21s).
• To stay up-to-date with the field, one can dedicate 10 minutes a day to reading and learning, and recommended resources include the ACL Anthology for NLP papers, Semantic Scholar, and tools like ChatGPT that can do retrieval-augmented generation (51m14s).
• Twitter, or X, is no longer as reliable a resource as it once was due to changes, and communities have spread out to other platforms like Bluesky, Threads, and Mastodon (51m48s).
• The rise of generative AI has made it easier to get a sense of an area by typing common-sense questions into search engines like ChatGPT, which can provide a starting point for learning and exploring the literature (52m4s).
• NLP is a well-organized community with a comprehensive literature, and resources like the ACL Anthology and Semantic Scholar can be used to find relevant papers and stay current (52m42s).

Practical Advice for Building with DSPi and Langchain

• The SPI (Software Platform Interface) is gaining traction in the business world, with various organizations, including Jet Blue and startups, using it in different ways, and its website, dspi.doai, offers documentation, use cases, and starter code (53m35s).
• When starting out with SPI or Langchain, it's essential to make principal choices and avoid designing a system entirely around prompt templates, as this can lead to unintended consequences and make changes difficult (54m26s).
• Using prompt templates can be productive for teaching purposes, but it's crucial to express things as proper software systems to avoid failure modes and ensure flexibility and adaptability (54m32s).
• DSPi is a great choice for building software systems, especially for those experienced in machine learning, as it's tailored to their needs and has PyTorch principles, although it may have a learning curve (55m28s).
• The most important thing to take away is to avoid thinking entirely in terms of models and instead focus on building software systems that can respond to new requirements and changes in the underlying environment (56m16s).

The Importance of Systems Thinking: A Final Emphasis

• Software systems like ChatGPT are often viewed as models, but they are actually compound systems that require a broader focus beyond just the model choice or its properties (56m34s).
• A more effective approach is to concentrate energy on the entire system, similar to an F1 race car design team that focuses on all the complicated pieces working together in concert (56m49s).
• In the industry, most energy is focused on small models, which makes system design even more crucial, as simplistic system design and prompting strategies are not sufficient (57m10s).
• With small models, it is necessary to do everything possible to achieve a significant impact, which places more pressure on system design, but this pressure also presents a huge opportunity (57m29s).

Conclusion and Next Steps

• The discussion concluded with appreciation for the questions and participants, and the session will be posted on YouTube and shared as a recorded session (57m49s).