Stanford CS234 Reinforcement Learning I Emma Brunskill & Dan Webber I 2024 I Lecture 15

Discussion on DPO Model and Human Feedback

• The discussion begins with a review of the DPO model, which assumes a specific model of how people respond to preferences, particularly using the Bradley Terry model. It is noted that this model can also be applied in situations where reward labels are directly provided. (1m46s)
• There is a debate about the effectiveness of using human feedback to learn reward models for board games, such as chess, where the reward is known and occurs at the end of the game. It is suggested that directly using known rewards is more reliable than pairwise rankings of intermediate game states. (2m24s)
• Both DPO and RHF can be utilized with extremely large policy networks. (3m17s)
• The session includes a brief continuation of the previous discussion on Monte Carlo research and AlphaZero, with a focus on clarifying previous points. (3m25s)

Guest Lecture Introduction - Dan Weber

• A guest lecture by Dan Weber is introduced, which will explore the origins of rewards and how judgments about preferred rewards are made. This topic will be discussed further after a quiz. (3m43s)

AI Achievements and Impact on Human Expertise

• The significance of AI achievements, such as surpassing the best human performance in certain tasks, is highlighted. Historical examples like IBM Watson's performance on Jeopardy are mentioned as pivotal moments in AI development. (4m15s)
• The discussion explores the implications of achieving AI on human expertise and excellence, referencing the victory of DeepMind's AlphaGo over Lee Sedol. (4m59s)

Explanation of Monte Carlo Tree Search (MCTS)

• Monte Carlo Tree Search (MCTS) is explained as a method that approximates a forward search tree, which can scale exponentially with the number of states and actions. MCTS uses a dynamics model to sample next states, addressing the state branching factor but not the action branching factor. (8m52s)

AlphaZero's Network and Performance

• In AlphaZero, a single network outputs both a policy and a value, unlike the original AlphaGo, which used two networks. This network has two output heads. (9m59s)
• Even after extensive training, AlphaZero performs additional guided Monte Carlo Tree Search during test time, significantly improving performance, as evidenced by a substantial increase in ELO score. (10m18s)

Self-Play and Curriculum Learning in AlphaZero

• Self-play in AlphaZero provides a form of implicit curriculum learning, as the agent competes against an opponent of similar skill level, enhancing the density of rewards received. (10m45s)

Move Selection Process in AlphaZero

• The process of selecting a move in a single game involves maintaining a Q-value and an upper bound, which is proportional to the policy function from the neural network divided by the number of samples. (11m17s)
• The notation 's' is clarified to represent nodes in a tree search, rather than state space, and it is noted that different game states can occur at different points in the tree, maintaining separate statistics without sharing across nodes for simplicity in architecture and storage. (11m41s)
• When making decisions at the root of the tree, actions are prioritized based on the number of times they have been explored, with the probability of taking an action being proportional to the number of times it has been taken from the root. This is adjusted by a parameter 'tau', which influences whether the approach is more exploratory or follows a "winner takes all" strategy. (12m41s)
• The decision-making process does not consider the value at the root but is based on exploration frequency. (14m24s)

Derivatives of AlphaZero and Games with Hidden Information

• There are various derivatives of AlphaZero, such as MuZero, which does not require knowledge of game rules, and approaches for games with hidden information like poker, where optimal play involves dealing with incomplete information. (14m39s)
• A question was raised about models like AlphaZero, which do not know the rules of a game but can still learn effectively. It was noted that these models can perform well without knowing all the rules, as long as they receive feedback on whether they win or lose. The architecture of the model, such as using convolutional neural networks, significantly impacts the amount of data needed and the quality of results. This approach has been extended to games like chess, demonstrating the versatility of these techniques. (15m18s)

Introduction of Guest Lecturer - Dan Webber

• Dan Webber, a postdoctoral fellow at Stanford, was introduced as a guest lecturer. He specializes in frameworks for understanding rewards and their implications for system development. (16m43s)
• Dan Webber is affiliated with the Stanford Institute for Human-Centered Artificial Intelligence and the Center for Ethics and Society. His role involves integrating ethics into computer science courses. He holds a PhD in philosophy. (18m11s)
• The speaker discusses their background, mentioning a dissertation on moral theory at the University of Pittsburgh and a bachelor's degree in computer science, along with experience in software development. They note that the field of AI has changed significantly since they last studied it, particularly in reinforcement learning. (19m8s)

Understanding Value and Value Alignment

• The focus of the talk is on understanding value and value alignment, acknowledging that the session will not solve deep problems but will provide an overview of the complexities involved. (20m3s)

Examples of Value Misalignment

• An example of value misalignment is introduced through the "paperclip AI" scenario, originally described by Nick Bostrom in 2016. This example illustrates how an AI designed to maximize paperclip production could lead to extreme outcomes, such as converting the Earth and the universe into paperclips. (20m52s)
• The discussion includes more mundane examples of potential issues with AI, such as scheduling workers for excessive shifts, producing low-quality paperclips, or operating inefficiently without considering other goals like efficiency or resource conservation. (22m5s)

The Problem of Value Alignment

• The problem of value alignment in AI involves designing agents that perform tasks according to human intentions, which are often more nuanced than explicitly stated. Humans operate with many background assumptions that are difficult to formalize and easy to overlook. (23m42s)
• An example is given where a factory manager is instructed to maximize paperclip production. Implicit assumptions include adhering to labor laws, ensuring product quality, and managing costs, which are not explicitly stated but are understood by humans. (24m14s)
• Simply providing better instructions to AI, such as specifying paperclip quality or maximizing long-term profits, may not fully address the problem. These instructions might still overlook important considerations like ethical treatment of workers and efficient resource use. (25m17s)
• The challenge is compared to the difficulty of manually specifying reward functions, highlighting that understanding and specifying what is truly desired is complex and often more challenging than initially anticipated. (27m13s)
• Designing AI systems that can take instructions from non-expert users poses challenges, as these users may not foresee issues with incomplete instructions. (27m37s)
• The problem of value alignment involves creating AI agents that accurately interpret and act on the user's true intentions, which are often complex and nuanced. (28m37s)

Technical Challenges in Value Alignment

• A significant technical challenge in AI is understanding the full intention behind instructions, which requires a comprehensive model of human language, culture, and interaction. (29m50s)
• Despite advancements like GPT, there are still concerns that AI models may omit aspects of the world model, leading to potential loopholes in understanding user intentions. (30m42s)
• There is skepticism about whether current language models, such as large language models (LLMs), can fully capture the nuances of human communication and intention. (31m23s)

Philosophical Challenges in Value Alignment

• There is a technical challenge in aligning AI with human intentions, as intentions may not always reflect what individuals truly want, especially in cases of incomplete information or imperfect rationality. (32m6s)
• A philosophical challenge arises when an AI follows expressed intentions, such as maximizing paperclip production, which may not align with the user's actual preferences or best interests. (32m12s)
• To address this, AI should ideally align with what users truly prefer, even if it differs from their stated intentions, to avoid negative outcomes like world destruction or harm to workers. (33m19s)
• Aligning AI with user preferences requires the AI to discern these preferences, which may differ from expressed intentions, potentially through methods like inverse reinforcement learning or reinforcement learning from human feedback. (33m46s)
• A challenge in this approach is inferring preferences from limited observations, especially in unexpected or emergency situations where user preferences are not directly observed. (34m42s)
• There is a philosophical issue where user preferences might diverge from what is actually beneficial for them, such as preferring to smoke despite health risks or prioritizing profit over well-being. (35m32s)
• Preferences may sometimes diverge from what is objectively in a person's best interests, and aligning AI to serve the user's true interests, even when they differ from the user's preferences, is a complex challenge. (36m6s)
• Determining what is objectively good for a person is a philosophical question rather than an empirical one, requiring moral philosophy to address. (37m13s)

Objective Good and User Preferences

• There is significant disagreement about what constitutes objective good, with debates on whether it is a person's pleasure, the satisfaction of desires, or other factors like health, safety, and knowledge. (37m40s)
• Despite disagreements, there is broad consensus that elements such as health, safety, liberty, knowledge, dignity, and happiness are generally good for individuals. (38m52s)
• Autonomy, or the ability to choose how to live one's life, is considered beneficial, even if it leads to suboptimal choices, highlighting the importance of avoiding paternalism. (40m2s)
• The discussion highlights the importance of considering user intentions and preferences when designing AI agents, even when aligning with the user's best interests. This involves fulfilling their intentions and honoring their preferences. (40m32s)

Value Alignment and its Practical Implications

• The concept of value alignment is introduced, which involves designing AI agents to perform tasks that align with what users truly want, intend, or prefer. These aspects can differ and may impose technical or philosophical constraints. (41m8s)
• The practical implications of value alignment are explored through the example of large language model (LLM) chatbots. While everyone interacts with the same fundamental chatbot, providers offer various personas, some designed by users, to cater to different needs and preferences. (41m42s)
• Examples of personalized chatbots include creative writing helpers, emotional support bots, and personas like political figures or fictional characters. These are user-designed and not imposed by the LLM provider. (42m20s)

LLM Chatbots as a News Source

• The idea of using LLM chatbots as a news source is considered, acknowledging that many people already use search engines like Google for news. The potential demand for personalized news chatbots is noted, and the discussion encourages thinking about how to align these bots with user preferences and best interests. (43m31s)

Aligning Content with User Preferences and Best Interests

• A preference optimization approach can be used to align content with a user's preferences by offering choices between different options and optimizing the content based on the user's selections. (47m5s)
• Aligning content with a user's best interests is challenging because it is difficult to determine what is objectively in someone's best interest. A possible solution is to optimize for the best interests of an entire population rather than individual users. (48m13s)
• Understanding a user well, including their data and behavior, can help in prioritizing suggestions and tailoring tools to their needs. Tools should avoid overwhelming users with too much information and should focus on delivering the most important content efficiently. (49m43s)
• The discussion explores the idea of understanding a user's best interests by observing their preferences over time, even if these interests diverge from their immediate desires. This involves maintaining a state that represents different aspects of the user's best interests, which can be personalized and updated with each interaction. (50m43s)
• There is a consideration of aligning responses to the user's preferences by determining what the user wants to achieve from interactions with a bot. This involves writing prompts that maintain the state of the user's best interests and using meta-reasoning to infer these interests. (51m52s)
• A basic question is raised about what kind of news would be in a news-seeking agent's best interest to provide. It is suggested that providing a variety of perspectives and ensuring the information is correct is in the user's best interest, rather than only showing news that aligns with their existing opinions. (52m49s)
• The potential downside of aligning too closely with a user's preferences is discussed, as it may lead to an echo chamber effect where users are only exposed to news that reinforces their existing views. It is argued that exposing users to high-quality, unbiased news and a variety of opinions is more beneficial. (53m43s)
• The conversation touches on the pros and cons of different approaches to designing a news chatbot, considering whether optimizing for a user's best interest might be seen as paternalistic. (54m59s)
• The discussion highlights the challenge of understanding a user's complete state, including their emotional and psychological state, which can affect their preferences and interactions with an app. It suggests that relying on observable user preferences, such as what they click on, might be more effective than trying to infer their internal state. (55m5s)
• There is a recognition that while general assumptions can be made about what is in a person's best interest, individual variations exist due to subjective interests and desires. This creates a risk of paternalism when trying to determine what is truly good for a user. Aligning with user-stated preferences can help avoid this risk. (55m43s)
• A counterargument is presented that focusing solely on user convenience might lead to offering low-quality choices, which can waste time. It is suggested that some aspects of a user's best interests, such as exposure to high-quality news and diverse opinions, are easier to determine and should be considered alongside user preferences. These goals are not mutually exclusive and can be balanced. (57m13s)

Missing Aspects of Value Alignment

• The discussion notes that a significant aspect of value alignment has been missing from the lecture, hinting at a broader topic related to aligning with user intentions and the complexities involved. (58m36s)

Value Alignment and Societal Interests

• The discussion explores the concept of value alignment in AI, emphasizing the importance of aligning AI actions not only with individual user preferences but also with broader societal interests. (59m12s)
• It is highlighted that an AI agent should be considered value-aligned if it acts in a morally right manner, rather than just fulfilling the user's instructions, as actions beneficial to one person might be harmful to others. (1h0m4s)

Determining What is Morally Right

• The conversation acknowledges the complexity of determining what is morally right, given the widespread disagreement on moral issues, such as lying to spare feelings or prioritizing luxury over charity. (1h1m44s)
• Moral theories, such as consequentialism and utilitarianism, are introduced as systematic approaches to address moral questions, suggesting that actions should aim to produce the greatest net good or total happiness. (1h2m50s)
• There is a discussion about aligning AI with morality, specifically focusing on the challenge of determining the correct or best moral theory due to philosophical disagreements. (1h3m51s)
• Various moral theories are mentioned, including consequentialism, prioritarianism, and maximin or minimax views, each with different approaches to evaluating what is morally right. (1h4m43s)
• Consequentialism aims to maximize total good, while prioritarianism gives more weight to the interests of those who are worse off, potentially using a weighted sum approach. (1h6m25s)
• The maximin or minimax view focuses on improving the situation for the person who is worst off, even if it results in less total good. (1h5m23s)
• Satisficing is another approach where an act is considered right if it produces a sufficiently great sum of good, rather than maximizing it. (1h7m51s)
• Deontological views are also mentioned, which prioritize moral rules or rights over the consequences of actions, such as rules against murder. (1h8m7s)
• There is a discussion on the limitations of consequentialism, which may not adequately capture moral actions such as stealing, even if they result in good outcomes. This raises questions about the justification of rules or rights based on their consequences. (1h8m24s)
• The challenge of paternalism in AI design is highlighted, emphasizing the difficulty of aligning AI with a moral theory that users may not share. This could lead to a lack of trust in AI if it imposes moral values that users disagree with. (1h8m58s)
• Despite disagreements on the best moral theory, there is consensus on basic moral principles, such as not killing, lying, or stealing. Aligning AI with common sense or consensus morality, which most people agree on, is suggested as a practical approach. (1h9m52s)
• Aligning AI with common sense morality is seen as more predictable and less likely to lead to surprising or undesirable outcomes compared to strict adherence to a particular moral theory. This approach is more deontological and satisficing, allowing for prioritization of personal interests in some cases. (1h10m33s)
• The potential pitfalls of aligning AI with specific moral theories are illustrated with examples, such as a consequentialist AI making extreme decisions to maximize net good, which may not align with human moral expectations. Aligning AI with common sense morality could result in more human-like and predictable behavior. (1h11m3s)
• The discussion addresses the challenge of aligning AI with common sense morality, particularly in complex moral dilemmas such as whether an AI should kill one person to save a million. (1h12m40s)
• It is suggested that if AI were taught to think about morality like humans, it might be as uncertain as humans are in difficult moral situations. (1h12m57s)

Further Discussion and Meeting Invitation

• The speaker invites further discussion on ethics and offers to set up meetings for those interested in exploring these topics more deeply. (1h13m26s)