Stanford CS329H: Machine Learning from Human Preferences I Guest Lecture: Joseph Jay Williams

Introduction and Goals

• The goal of the talk is to provide a sharable message about human contraction and experimentation, allowing the audience to explain how experimentation can be used in various behavior change contexts and potentially contribute to helping someone win a Nobel Prize for experimentation (52s).
• The speaker's research can be found on their website, and they have a mailing list that people can join to stay updated on lab meetings and special meetings (41s).
• The speaker's objectives for the talk include providing concrete information that the audience can share with others and inspiring the audience to take action or contribute to the field of experimentation (54s).
• The speaker's program aims to coach behavior change by transforming everyday technology touch points into intelligent interventions, using an approach that involves adaptive experimentation tools, personalized and contextualized interventions, and experimentation to exercise science and practice (2m13s).

Behavior Change and Intelligent Interventions

• The speaker's lab has worked on various projects related to behavior change, including papers presented by Anan B Shari and others (2m47s).
• The science of behavior change can help solve many human problems, and the speaker invites the audience to imagine what behaviors they wish they could start or stop doing (3m3s).
• Behavior change is a crucial aspect of various areas, including education and health, and can be influenced by technology to help people make better decisions and change their behavior for good (3m45s).
• The vision for 2034 is to have on-demand intelligent coaches that measurably help people think through many problems and change their behavior (4m11s).
• The ADC framework has been used as a foundation from 2012 to 2024 to design intelligent interventions, which are tools that change behavior and learn over time (4m22s).
• Intelligent interventions can be in the form of text messages, app messages, or explanations of concepts, and they try to come up with the best thing to give a person at a specific point in time (4m31s).
• To build intelligent interventions, multiple disciplines need to be integrated, and continuous learning is essential (4m47s).
• The Adap Comp framework takes components of everyday experience and makes them into micro-labs for intelligent intervention, where ideas can be tested and improved over time (5m17s).
• Examples of intelligent interventions include text messages, such as a daily message that is crowdsourced and tested to figure out which one is best for a person (5m30s).
• Emails can also be turned into intelligent coaches by generating multiple versions and testing what works when, such as getting students to start their homework early (6m4s).
• Any everyday experience, such as explanations on a website, can be turned into an intelligent intervention, and it should be able to learn and adapt over time (6m31s).

A/B Testing and Machine Learning

• Machine learning from human preferences can be applied to various aspects of life, making everything an intelligent intervention to test out different versions and changes over time (7m54s).
• A/B testing can be used to improve learning management systems (LMS) by testing different versions of explanations, messages, and answers to figure out what works best for students (7m37s).
• However, A/B testing in LMS is not being utilized to its full potential, with many opportunities for improvement, such as testing 200 times more versions than currently being done (7m52s).
• One challenge in implementing A/B testing is achieving sufficient statistical power, especially in small exchanges where the impact on metrics like click rates may be very small (8m15s).
• To overcome this challenge, collaborations can be built to conduct A/B testing at scale, such as in education and mental health settings (9m2s).
• Another approach is qualitative A/B testing, which involves gathering feedback and thinking through different options, even with limited data, such as when sending an email to only one person (9m24s).
• A paper titled "Abri by Moi Razer" explores qualitative A/B testing and its potential applications (9m17s).
• A/B testing can also be applied in a more thought-experimental way, using tools like A/V scribe to create different versions of an email and thinking through how they would come together (9m43s).

Qualitative A/B Testing and Thought Experiments

• The concept of A/B testing can be applied qualitatively, rather than just quantitatively, and a new name for this approach is being sought, with suggestions including "qualitative A/B comparisons" and "super A/B comparisons" (10m36s).
• Intelligent intervention involves a process that can work even without quantitative data, consisting of two parts: generating options and exploiting the evaluation of these options, which can be achieved through adaptive experiments that integrate reinforced learning and human judgment (10m56s).
• To generate options, one can use language models (LMS) and human input to create alternative versions, and then explore the space of options to come up with better ideas (11m10s).
• Exploiting the evaluation of options involves explaining the execution of these options, which can be done using adaptive experiments that give people what looks better, integrating reinforced learning and human judgment (11m30s).
• A concrete example of this process is assigning probabilities to different versions of an email, such as 70% to one version and 30% to another, and then giving each version to the corresponding percentage of people to gather more data (11m51s).
• This approach allows for rethinking the world to not just give 100% or 0% of something, but to give things proportional to the probability of having the best outcome (13m47s).

Probabilistic Models and Human Judgment

• Assigning items or actions proportionally to the probability of being the best requires defining what "best" means and deciding on the complexity of the model used to weigh different versions against each other (14m7s).
• Building a probabilistic model in the background is necessary to estimate the belief of the student and plan subsequent actions, but there are many ways to do this and many different ways to plan subsequent actions (14m33s).
• The process of figuring out relevant components of a model and setting probabilities for A/B testing is complex and depends on various factors, with no one-size-fits-all solution (15m8s).
• One possible model for setting probabilities in a two-arm case with binary outcomes will be presented, but it has its limitations (15m18s).
• An alternative approach is to use a crowdsourced mixture of experts, where a group of people, such as marketing experts, provide their opinions and update their beliefs based on the data (15m33s).
• There is a need for more research on getting human beings to set probabilities, and statistical models can be used, but more innovative work is required (15m58s).
• The goal is to make A/B testing accessible to 8 billion people, who may not run formal A/B tests but can still use tools to randomize and test different versions of emails or messages (16m11s).
• The intention is to rethink the way decisions are made and bring elements of testing together to help think about signals that indicate what might be better or worse (17m23s).
• The goal is to encourage randomization, even if it's not a 50/50 split, and to build the habit of A/B testing, which is progress towards making better decisions (17m57s).
• The problem of choosing probabilities is a problem to solve, and worrying about it gives confidence that a solution can be found (18m12s).
• Research has been done on models that can be used for A/B testing, but there is still a lot of room for improvement and innovation (18m21s).
• Quantitative methods are being used to estimate probabilities, but this approach may not be enough to help people make better decisions, and crowdsourced approaches can be more effective in this regard (18m27s).
• The quality of work will be improved with crowdsourced approaches, and one way to see this is by using models that tell us what our probabilities are (18m49s).

AB Scribe and Thought A/B Testing

• An example of an experiment is AB scribe, a tool that enables users to create and test different versions of a message, such as an email, to see which one is more effective (19m21s).
• AB scribe allows users to select pieces of the message and make them an A/B test, and then use a visual language model to generate different versions of the message (20m10s).
• The tool can be used to experiment with different versions of a message and see which one works better for a specific audience, such as someone who is very scientific or someone who believes in the benefits of yoga (20m30s).
• The goal of AB scribe is to help users create more effective messages by testing different versions and seeing which one performs better (20m53s).
• The tool can be used in various contexts, such as writing an email to students before a test, and can help users think through different versions of a message and get feedback from a language model (21m10s).
• The approach used in AB scribe is called thought A/B testing or thought A/B comparison, which involves defining options, thinking through different versions, getting feedback from a language model, and traversing the options (21m25s).
• The future of this approach is promising, and researchers can build on this work to publish high-quality papers in this area (21m47s).

Study on Stress and Test Performance

• A study was conducted to test the effectiveness of a three-minute message in improving test performance, with the message being that stress can actually help improve performance on a test, not hurt it (21m52s).
• The study involved six different elements, including varying the text, showing people instructions, a video of the speaker, and asking participants to explain what they thought they'd learn (22m36s).
• The core message was tested in many ways, and the results showed that the message on its own, without any additional information, resulted in an average grade of 76% (23m7s).
• When the message was elaborated on, such as explaining how stress can help put more resources into the task and make the person pay more attention, the results showed a significant improvement, with an average grade of around 80% (23m38s).
• The study involved running six different experiments in a couple of months, which would have taken a couple of years to run with traditional research methods (24m2s).
• The results of the study suggest that the message could be effective in improving test performance, but it's not a guarantee, and the probability of it working in a particular population is not 100% (24m29s).
• To increase the chances of the message being effective, it's suggested to use crowdsourcing to come up with different versions of the message and then test them using A/B testing and machine learning models (24m36s).
• The data from the study could be used to calculate the probability of the message replicating in a particular setting, and this probability could be used to inform decisions about whether to use the message in a particular context (25m7s).
• Instead of just using statistical models to predict the effectiveness of the message, it's suggested to give the data to someone who is teaching a class and let them decide whether to use the message based on their own judgment (25m30s).
• A class was conducted to demonstrate how quantitative data can be combined with human judgment to make decisions, with the results showing a 76% chance of a particular message being the best to show students (25m34s).

Combining Quantitative Data and Human Judgment

• The process involves taking quantitative data, computing probabilities based on a model, and then giving that data to a human for judgment, allowing for the combination of human judgment with model probabilities (25m49s).
• This approach may seem subjective, but it can be a good way to connect technical tools with human behavior and decision-making (26m23s).
• The approach does not rely on individual estimates of probabilities, but rather on combining different beliefs and models, such as Bayesian combinations of beliefs (26m54s).
• There are different tools that can be used in this process, including statistical models, crowdsourcing opinions from people, and combining mathematical concepts with human behavior (27m25s).
• The goal is to take mathematical concepts and apply them to real-world problems, rather than just focusing on pure math or theoretical models (28m3s).
• This approach requires a combination of technical skills and human intuition, with the goal of designing experiences and interventions that can help people solve problems (28m9s).

Crowdsourcing and Human Feedback

• The idea of "high-dimensional intuition" is mentioned, which refers to the ability to navigate complex and messy problems, and to know which problems are worth working on (28m17s).
• Crowdsourcing opinions from people is a well-defined process that can be used to combine mathematical concepts with human behavior, and can be reproduced and changed as needed (28m50s).
• Human feedback is essential in machine learning, and people are generally bad at quantifying probabilities, making it challenging to work with human preferences (29m25s).
• The importance of using probabilities as signals in human feedback is questioned, and alternative signals or lower-effort methods are considered (29m47s).

Adaptive A/B Testing and Reinforcement Learning

• Deviating from uniform random sampling can increase power in certain cases, and quantitative adaptive AB testing is a valuable approach (30m23s).
• AB testing is a fundamental form of reinforcement learning, and Thompson sampling is a method that samples probabilities to make decisions (31m6s).
• A simple form of AB testing is the beta bandit model, which involves two arms (e.g., explanation A vs. B) and a reward outcome (1 or 0) (31m17s).
• In the beta bandit model, a policy learning algorithm like Thompson sampling assigns probabilities to each arm, and a beta distribution is used to model the probability of a positive outcome (31m50s).

Beta Bandit Model and Thompson Sampling

• A beta distribution with parameters 1 and 1 (beta 1,1) is a uniform distribution, indicating no prior knowledge or data (32m17s).
• Using a beta 1,1 distribution as a prior means that all possible probabilities are equally likely, and it's like having no data or one success and one failure (32m21s).
• A beta distribution can be used to represent the probability that people will think something is helpful, with the reward being a draw from the parameter of that beta distribution, and it is simple and interpretable (32m54s).
• For example, if five people are given arm one and three of them like it, while two do not, the beta distribution becomes beta 1+3, 1+2, which is skewed a bit higher, indicating a higher probability that people think it's helpful (33m19s).
• If five people are given arm two and all of them like it, the beta distribution becomes beta 1+5, 1+0, which is skewed even higher, indicating a high probability that people like it, but still with some uncertainty (33m55s).
• Having more observations, such as beta 60 and 10 versus beta 6 and 1, results in a more peaked distribution, indicating more confidence in the probability, even if the mean is the same (34m22s).
• The mean of a beta distribution can be calculated, for example, the mean of beta 60 and 10 is 6/7, indicating a high probability that people will like something (34m35s).
• Thompson sampling can be used to estimate the probability that one arm is better than another by sampling from the beta distributions and assigning the arm with the highest expected reward (35m48s).
• For example, if the beta distribution for arm A is 6 and 1, and for arm B is 3 and 3, Thompson sampling would sample from each distribution and assign the arm with the highest expected reward, which in this case would be arm A (36m0s).
• Thompson sampling works by sampling from the beta distributions and comparing the expected rewards, and it makes sense because it takes into account the uncertainty of the probabilities (36m35s).
• To determine the probability of one arm being better than another in an adaptive experiment, one can sample from the beta distributions 10,000 times and compute the probability empirically, which can result in a probability of 80% or 70% that one arm is better than the other (37m5s).
• The inference in adaptive experiments is not trivial, and there are many mistakes people make when using these experiments without realizing it (37m32s).
• Sampling works by choosing the arm that gives the highest reward under certain parameters, and one can compute the probability empirically by repeating the process 10,000 times (36m44s).
• The online model can be used to determine the probability of one arm being better than another, and it is possible to formalize human judgment and manual assignment using beta distributions (37m46s).
• If a person has a prior belief about the probability of one arm being better than another, it is possible to back out their prior into beta distributions, which can be represented as fictional successes and failures (38m11s).
• For example, if someone thinks there is a 70% chance that arm A is better than arm B, they might represent this as 8 successes and 2 failures in arm A, and 5 successes and 5 failures in arm B (38m22s).
• There is a need for research on how people express probabilities that one arm is better than another and how to map these probabilities to beta distributions (38m56s).
• It is possible to help people formalize their beliefs about whether one arm is better than another by providing interactive explorers or tools that help them understand how their intuitions can be mapped to beta distributions (39m12s).
• It is also possible to weight people's approaches differently, for example, by giving more weight to the estimates of someone who is considered an expert (39m37s).
• For instance, if someone thinks that an expert's estimates are 10 times more valuable than their own, they can weight the expert's successes and failures 10 times more than their own (40m12s).

Combining Expertise and Opinions

• A method of combining expertise and opinions involves assigning 50/50 using human judgment or a model, then evaluating the results with both quantitative and qualitative metrics to determine the probability of one option being better than the other (40m21s).
• This approach, called Conant experimenting, involves continuously adding new ideas and updating probabilities over time (41m2s).
• The method requires a lot of trust and effort from users, and it's possible for individuals to try to skew the results if they're very confident about something (41m22s).
• A paper titled "Eliciting beliefs from people to get better decision making even when some parties want to sway the vote" investigates ways of gathering feedback from different people to come to a group consensus (41m42s).
• The paper examines policies for combining input from different people and investigates weighting schemes, showing that some schemes can suffer from bias if someone tries to sway the majority (41m55s).
• The "hippo" approach, where decisions are made by the highest paid person's opinion, can be problematic, and a paper on this topic could explore ways to defend against this bias (42m32s).

Language Models and Personalized Interventions

• Using language models to quickly refine priors and simulate human behavior is a potential area of research, with a paper by Rob Willer at the university exploring the use of language models to simulate human behavior (43m45s).
• Writing a paper on using language models to help with decision-making could be a quick, easy, and highly impactful project (44m1s).
• There are numerous potential research papers that can be written on the topic of using large language models (LMs) to simulate human behavior and provide personalized interventions, with at least 200 possible papers in the next 10 years (44m11s).
• Personalized and contextualized interventions are an area of research that involves using LMs to provide tailored advice and support to individuals in specific situations (44m35s).
• Ana's paper on "Small Steps SMS" is an example of a personalized intervention that sends daily messages to help users manage stress and be happier (44m46s).
• Another paper by Ana discusses the use of design prompts for self-reflection, allowing users to ask themselves questions to manage stressful situations (44m53s).
• A paper on "Stories and Messages" presents a system that provides users with stories about others who are going through similar experiences, and allows users to abstract the lessons to their own setting (45m8s).
• A recent paper explores the use of LLMs as a thought partner to help users adapt stories to their own context (45m20s).
• Another paper discusses the use of LLMs as a thought partner to help users overcome procrastination, using prompt engineering and interface design to facilitate effective messaging (45m33s).
• The "Teny Spark" paper or demo presents an interface that allows users to type in their situation and receive a message from an LLM, with options to customize the tone and style of the response (46m40s).
• The interface allows users to edit the prompt and generate multiple messages, and can be used to provide support for a range of situations, including procrastination and stress management (47m13s).
• The development of effective interfaces for LLMs is crucial for unlocking their potential value, and can make a significant difference in the user experience (47m31s).
• The business value of AI can be demonstrated through the development of use cases that show its impact, such as personalized interventions and support systems (47m51s).

Adaptive Experimentation and Statistical Minorities

• Adapting experimentation using psychology and Human-Computer Interaction (HCI) can help discover the best intervention for a statistical minority, which may be the opposite of what's best on average (48m1s).
• A graph is presented showing the outcome of an experiment where students receive messages to generate questions about what they're learning in an online course, with two different messages (blue and red) having different effects on students with higher and lower accuracy (48m31s).
• The data shows that on average, the red message is more effective in getting students to generate questions, but when broken down by accuracy, the blue message is more effective for students with lower accuracy, who are a statistical minority (49m1s).
• This means that taking just the average would actually discriminate against the statistical minority, and using the red message for everyone would hurt the students who would benefit more from the blue message (49m35s).
• The blue message is better for 20% of the data (students with lower accuracy), while the red message is better for 80% of the data (students with higher accuracy), but the blue message is substantially better for the lower accuracy group (50m15s).
• Not running experiments and relying on intuition or qualitative data would make this problem bigger, as it would lead to missing the opportunity to discover the subset of students who benefit from the blue message (50m41s).
• Using contextual bandits, an experiment can be designed to assign the messages separately for the low accuracy and higher accuracy groups, allowing for adaptation over time to help the hierarchy group and collect more data for the low accuracy group (51m15s).
• This approach would allow for a 70% chance that the red message is better than the blue message for the higher accuracy group, while keeping a 50/50 split for the low accuracy group to collect more data (51m32s).
• An approach to experimentation allows for continuous data collection over time, adapting to discover what's better for subgroups, and potentially leading to better outcomes for people in smaller groups, with a 20% increase in statistical power to detect effects (51m55s).
• This method can help identify scenarios where option B is better than option A for most people, but might be hurting a minority, and can increase the statistical power to detect this difference (52m11s).
• In a traditional experiment, outcomes for minorities can be uniform, with 36% of people generating a question, but this approach can lead to better learning outcomes and increased statistical power to detect the best option for minorities (52m53s).
• The statistical power to detect the best option for minorities increased from 87% to 89%, which may not seem like a lot, but can translate to 50 fewer samples needed to be equally confident (53m6s).
• The approach can also reduce the number of people needed in an experiment, with even five fewer people being a good outcome (53m18s).

Epsilon Thompson Sampling and Algorithm-Induced Test Statistic

• The algorithm used is not Thompson sampling, which was previously thought to be effective, but rather Epsilon Thompson sampling, which involves doing uniform random sampling Epsilon of the time and Thompson sampling the rest of the time (54m0s).
• Epsilon Thompson sampling can be thought of as doing some amount of traditional experimentation to reduce the biases of Thompson sampling and guard against changes in the world or missing minority populations (54m32s).
• The analysis of data from Epsilon Thompson sampling can be done using a test statistic, which is a more effective method than debiasing techniques (55m30s).
• When comparing two options, A and B, using Thompson sampling or epsilon Thompson sampling, a Z test can be used to determine if there is a significant difference between the two options, but this assumes uniform random sampling, which is not the case with these algorithms (55m51s).
• The algorithm-induced test statistic takes the Z test statistic and regenerates it using the algorithm to collect the data, resulting in a different distribution for the test statistic and a different cutoff value for the hypothesis test (56m21s).
• Using this approach, the false positive rate can be fixed, whereas Thompson sampling can push the false positive rate to 15% compared to 5% with uniform random sampling, and the power of the test is not bad, with epsilon Thompson sampling giving a power of 66% (56m50s).
• The key takeaway is that when running experiments adaptively, epsilon Thompson sampling or a version called TS postf can be used, and then the algorithm-induced test statistic can be applied to get better results (57m22s).
• Traditional Thompson sampling has a specific tradeoff between false positives and other errors, but modifying the algorithm can result in other tradeoffs that might be more practical for a study, giving good inference (57m43s).
• Thompson sampling is not necessarily on the Pareto frontier, and changing the algorithm can result in better uniformly more statistically sensitive results (58m3s).
• The issues with Thompson sampling have been characterized, and using epsilon top sampling ideally adaptive epsilon top sampling will give better statistically sensitive results, but the way statistical tests are done should also be modified using algorithm-tuned analysis (58m19s).
• The Z test is used to compare means by subtracting the sample mean of people who said they like explanation A from B, and then getting the standard error of the difference, which largely depends on the sample sizes (59m7s).
• The paper shows how Thompson sampling affects positive rates compared to uniform random, and other tests were done to show that factors such as T test, inverse propensity, and waiting also cause issues (59m31s).
• The AL induced test has a power of 17% and controls for a positive rate of 5%, but this is not great, and the probability technique difference exists when it does not, resulting in false positives (59m44s).
• This occurs due to sampling issues, where a random low mean is observed, and sampling is stopped, leading to a false conclusion of a difference when there is none (1h0m11s).
• The vertical axis represents a sample estimate of the mean, and in reality, both arms have a mean of 0.5, but due to sampling, one arm appears to have a lower mean, leading to a false positive (1h0m26s).
• The adaptive algorithm stops sampling from the arm that appears worse, resulting in a false conclusion of a difference, even with 800 participants (1h0m40s).
• This issue also explains why there is lower power in experiments, as the sample size is not evenly distributed between arms, leading to a lack of confidence in the results (1h1m26s).
• Even when there is no difference in means, the assign probability can be 80% or higher in 53% of the time, indicating a high probability of one arm being better when there is no actual difference (1h1m55s).
• Empirical exploration of this issue is necessary, and using a tuned analysis algorithm can help control the false positive rate to 5% and increase power (1h2m50s).
• The algorithm allows for better behavior in experiments, with lower false positive rates and better power, especially when there is a small difference between arms (1h2m40s).

Uniform Random Sampling and Hypothesis Testing

• Under uniform random sampling, the wall Z test is distributed in a way that allows for hypothesis testing, and the probability of getting a test statistic of 1.96 or bigger can be calculated to determine the alpha level (1h3m12s).
• Power is the probability of getting a test statistic bigger than 1.96 when there is an arm difference of 0.1, and in this case, power is 80% with a sample size of 785 (1h3m41s).
• Thompson sampling (TS) can give extreme values of the test statistic, but it's not the reality, resulting in lower power because it reduces the test statistic when there is a difference (1h3m58s).
• TS can't tell apart when there's a difference versus when there's not, reducing the signal from the two distributions (1h4m28s).
• The algorithm-induced test statistic can be adjusted to set the cut-off values to be higher, giving a 5% false positive rate, but this results in lower power, at 17% (1h4m52s).
• The problem is that the data are not discriminating, and TS or adaptive epsilon Thompson sampling drives the distributions apart (1h5m21s).
• Statistically sensitive algorithms are needed, and a paper is being worked on to investigate hybrid combinations of reward-maximizing bands like Thompson sampling with traditional uniform random (1h5m40s).

TS Positive and Adaptive Epsilon Thompson Sampling

• Epsilon Thompson sampling has a fixed probability, but it's not clear what epsilon value to use, and it's not the statistician's business to decide (1h6m3s).
• A clever idea is to use the posterior probability of the arm difference being less than a certain value (e.g., 0.1) to determine epsilon, which is called TS positive (1h6m56s).
• TS positive says that if the probability of the arm difference being smaller than the specified value is X, then that X is epsilon, and this adaptive epsilon Thompson sampling adjusts epsilon based on the posterior probability (1h7m31s).
• The algorithm provides a trade-off across different sample sizes and arm differences or effect sizes, and it is believed to beat existing algorithms because it adapts epsilon to sampling, with a uniform random sampling rate of 10% regardless of the sample size (1h8m2s).
• Using adaptive epsilon to sampling Thompson sampling (TS) positive results in a better trade-off between reward power and false positive rate, with a fixed false positive rate of 5% (1h8m31s).
• The algorithm's power is comparable to other methods, with TS positive having a power of 82%, uniform random having a power of 87%, and Thompson sampling having a power of 66% (1h8m50s).
• The use of TS positive and the algorithm allows for a good trade-off between reward inference and false positive rate, making it suitable for use in experiments (1h9m3s).

Future Research and Vision

• The development of more statistically sensitive algorithms, such as adaptive epsilon Thompson sampling and algorithm-tuned analyses, is a potential area of future research (1h9m22s).
• The vision of experimentation and coaching involves the use of intelligent interventions, such as adaptive experimentation tools like ABscribe, and the use of TS positive and algorithm-tuned tests (1h10m11s).
• The goal of this research is to accelerate science and impact practice through the development of more effective experimentation and coaching methods (1h10m23s).