Stanford Seminar - Responsible AI (h)as a Learning and Design Problem

Introduction: The Problem of Bias in Generative AI and Existing Resources (Lines 1-5)

• Generative AI can lead to various harms, such as amplifying or introducing biases in society, as demonstrated by newspaper headlines and research from experts like Sophia Noble, Ziad Obermeyer, and Joy Buolamwini. (31s)
• The issue of biases in algorithms is not new and dates back to the 1990s, with work from researchers like Bachcha Friedman and Helen Nissenbaum on bias and computing systems. (1m12s)
• To address these issues, various responsible AI resources have been developed, including software toolkits for fairness analyses, transparent reporting resources like model cards and data sheets, and organizational processes for responsible AI. (1m22s)
• Companies like Microsoft have developed responsible AI standards and processes, such as the Microsoft Responsible AI Standard, which outlines goals for fairness and other principles in responsible AI. (1m41s)
• Public sector organizations, like the US National Institute for Standards and Technology (NIST), have also developed responsible AI processes, including the AI Risk Management Framework to govern, map, measure, and manage various risks in AI development. (1m58s)

The Need for Better Support and Guidance for AI Developers (Lines 6-7)

• Research has shown gaps in AI developers' knowledge and skills when working on fairness in AI, including a study where 10 AI product teams struggled to complete a fairness evaluation process due to a lack of understanding of who to evaluate fairness for. (2m44s)
• This lack of understanding highlights the need for better support and guidance for AI developers in addressing fairness and bias in their work. (3m46s)

Gaps in AI Developers' Knowledge and Training in Responsible AI (Lines 8-12)

• AI developers may not be trained for new forms of work involved in responsible AI, and there is evidence to suggest this from various studies, including one published in 2022 (4m30s).
• Many universities are developing tech ethics courses, such as the Embedded Ethics program, to address this issue, and researchers like Casey Fiesler at the University of Colorado Boulder are surveying these courses to understand who is teaching ethics and how it is integrated with computer science concepts (5m18s).
• However, working AI developers may not have taken these courses during their training and may not have the time or inclination to take them after work, leading to a gap in their knowledge about responsible AI (5m51s).
• AI practitioners are taking on extra work to educate their team members about responsible AI, with some even teaching their teammates about harms they saw in user studies, despite it not being part of their job description (6m13s).
• Research has found that cross-functional collaboration between teams from different roles can be challenging due to a lack of shared language and understanding of concepts like fairness, leading to the need for education and clarification (6m52s).

A Study on AI Practitioners' Learning and Challenges in Responsible AI (Lines 13-15)

• A study was conducted to investigate what and how AI practitioners are currently learning about responsible AI, as well as the goals and aspirations of AI practitioners and responsible AI educators, and the challenges that get in the way of these goals (7m32s).
• Semi-structured interviews were conducted with 40 participants from 16 companies, including AI practitioners and responsible AI educators, to gather data for the study (8m3s).
• The study aims to understand the current state of knowledge and education about responsible AI among AI practitioners and educators, and to identify areas for improvement (8m2s).

Target Audiences for Responsible AI Training and the Scope of Current Resources (Lines 16-20)

• AI practitioners, including developers of machine learning or AI models and developers of applications that those models are embedded into, are the primary focus of learning resources or trainings for responsible AI, but there is a larger set of potential audiences, including third-party AI applications, marketing, PR, corporate leadership, and broader audiences such as community organizations, civil society, public sector policymakers, and universities (8m58s).
• A study was conducted to understand how AI practitioners learn and teach others about responsible AI, including the settings, reasons, effectiveness, topics, and skills involved, as well as how educators develop trainings and resources (9m51s).
• The study found that practitioners are learning about responsible AI concepts such as recalling company principles, defining dimensions like fairness, and procedural knowledge like computing fairness metrics and creating data cards or model cards (10m18s).
• However, the study also found that fewer resources are designed to help develop skills to identify new potential harms or proactively design generative AI to avoid those harms for new use cases (11m12s).
• Many learning objectives focus on understanding and recall, rather than applying knowledge to new skills or use cases, according to Bloom's taxonomy of cognition (11m34s).

Learning Modalities and Pathways for Responsible AI (Lines 21-26)

• Developers are learning about responsible AI through various modalities, including text documentation, notebooks, toolkits, training videos, YouTube, and collaborative learning settings like discussion groups (11m58s).
• The study identified different pathways for learning, including adapting knowledge from other domains, such as participants who learned about ethics in courses outside of computer science (12m21s).
• Many developers may not have the skill to apply what they learned in education or other fields to their work in building AI, and this application is a skill that needs to be fostered and cultivated (12m49s).
• Practitioners are often searching for responsible AI (RI) resources internally and externally, including on social media, and are creating their own curriculum through self-study and online resources (13m10s).
• This self-directed learning approach can be challenging, as learners may struggle to find legitimate resources or verify the credibility of information found on social media (14m8s).
• Some learners are also finding resources and learning about ethics and responsibility through social media, but are often unclear about the legitimacy of these resources or their creators (14m3s).

Collaborative Learning and Informal Knowledge Sharing (Lines 27-29)

• Learners are also learning from their coworkers in structured and unstructured settings, such as code reviews and conversations (14m31s).
• A follow-up study led by J. Marie Soless explored how informal learning sites, such as code reviews and reading groups, can facilitate collaborative learning and raise awareness about AI risks (14m37s).
• The study found that learners are often exposed to AI risks and ethics through personal experiences, such as family members protesting AI-related issues on social media (15m12s).

Framings of Responsible AI and the Computational Approach (Lines 30-32)

• The study identified two main framings for responsible AI: a computational framing, which is a technical approach, and another framing that is not specified in this text (15m37s).
• The computational framing is a technical approach that focuses on fairness metrics and software toolkits, but may overlook the human impact of AI systems or model failures (15m43s).
• This computational orientation can affect every part of an educator's choices, including learning objectives, goals, and assessment methods (16m18s).

Challenges in Assessing Qualitative Concepts and the Need for Training (Lines 33-35)

• Participants discussed the challenges of assessing the impact of AI systems on communities, with some suggesting that methods from the humanities and social sciences could be used to evaluate less quantifiable concepts (17m1s).
• Many participants were aware of the importance of assessing qualitative concepts, but felt they lacked the training to develop such assessments, having primarily received computer science training (17m29s).
• Educators recognized the need for a "train the trainers" approach to address the lack of training in assessing qualitative concepts and understanding which communities might experience fairness harms (18m18s).

Shifting Perspectives on Responsible AI and the Need for a Comprehensive Approach (Lines 36-38)

• Initially, participants viewed responsible AI as a problem to be solved by someone else, but later recognized the scope of the issue and the need for a more comprehensive approach (18m24s).
• There was a recognition that teaching easily quantifiable concepts, such as fairness metrics, was not the same as understanding the underlying issues and communicating them effectively (18m44s).
• Educators knew that broader questions, such as training data and model cards, were important, but struggled to make time to teach or upskill in these areas (19m6s).

Procedural Orientation vs. Outcome-Oriented Approach in Responsible AI Training (Lines 39-40)

• A procedural orientation was observed in many trainings and learning resources, where participants felt they were teaching company policies or toolkits rather than focusing on outcomes (19m25s).
• This procedural orientation was described as a "process objective" rather than an "outcome objective", with some ambivalence about the need for culture change (19m42s).

Proactive Fairness Evaluation and the Challenges of a Procedural Approach (Lines 41-43)

• Proactive playbooks for fairness evaluation involve asking questions at different stages, including problem formulation and model training, to address potential issues (20m2s).
• Participants were ambivalent about requiring processes for fairness evaluation, as it might lead to a sanitized version of ethics that avoids normative questions about what should or should not be designed (20m16s).
• Some participants felt that following company policies or processes might not help raise larger questions about the underlying root issues, such as military applications or policing technologies (20m51s).

The Need for Social and Cultural Considerations in Responsible AI (Lines 44-47)

• There is a desire to move beyond technical approaches and focus on social and cultural elements of responsible AI, including sociological and anthropological angles (21m10s).
• Recent reports from the Data and Society Institute and the Center for Democracy and Technology suggest that AI governance needs more technical expertise and sociological approaches (21m32s).
• Participants wanted to know how to engage with communities to identify potential harms, understand what harms mean for them, and co-design AI systems with them (21m55s).
• Many developers knew that engaging with communities was a good thing to do, but they didn't feel like they had the skills to do it (22m16s).

Applying Responsible AI Principles to Specific Products and Cultural Contexts (Lines 48-51)

• A recent paper with collaborators from Microsoft explored the impacts of language technology on speakers of different language varieties (22m19s).
• Participants aspired to build capacity for responsible AI by applying what they learned to their AI applications, but felt they lacked the skills to apply high-level principles to specific products and cultural contexts (22m48s).
• Practitioners wanted to apply AI principles, such as fairness and transparency, to their specific products, but felt they needed more guidance on how to do so in different cultural contexts (23m26s).
• Participants expressed a desire for customized trainings on responsible AI and fairness, tailored to specific contexts such as South Asian or Latin American settings, but technology companies felt they lacked the time, resources, or knowledge to develop these trainings (23m59s).

The Need for Context-Specific Training and Practical Examples (Lines 52-53)

• There was a need for more case studies, scenarios, and examples of responsible AI in practice, as current examples were often irrelevant or overly reused (24m26s).
• The application of learning about responsible AI is particularly challenging for generative AI models, which can be used in multiple contexts, putting more responsibility on product teams to consider potential harms (24m45s).

Organizational Pressures and the Tension Between Scalability and Depth in Learning (Lines 54-57)

• Organizational pressures and tensions impacted the aspirations of educators, including the need to scale learning resources and trainings, which often prioritized self-study courses over more effective collaborative learning methods (25m46s).
• Educators wanted to develop longitudinal curricula with increasing depth, but learners felt pressure to quickly ship products, leading to a preference for brief, superficial learning experiences (26m11s).
• The lack of structured pathways for continued learning in depth was a significant issue, with few resources available for learners to engage in more in-depth exploration of responsible AI (26m48s).
• Educators aimed to foster mindsets rather than provide prescriptive guidance, but organizational pressures often incentivized the latter, with learners seeking clear, actionable advice to navigate AI review processes (26m56s).

Moving Beyond Checklists and Compliance to Foster Responsible AI Mindsets (Lines 58-59)

• There is no one-size-fits-all approach to achieving Responsible AI (RI), and it cannot be reduced to a simple checklist of six things to ensure fairness (27m28s).
• A compliance-oriented approach to RI may not be effective, and instead, educators want to foster mindsets that allow people to reflect on their own practices (27m47s).

The Importance of Learning Environments and Integrating Social and Technical Skills (Lines 60-62)

• The learning environments for RI, including the sites for learning, impact what is learned and how, with large tech companies and open-source platforms having different approaches (28m0s).
• Informal sites for learning, community-based learning, and integrating social and technical skills and concepts can help avoid a procedural approach to RI (28m16s).
• Designing technical learning opportunities that integrate social and technical skills and concepts is crucial, and this can be achieved by shifting professional norms and the identity of what it means to be an AI developer (28m37s).

Shifting the Culture of AI Development Towards Responsibility (Lines 63-65)

• The hierarchy of knowledge in learning about RI needs to be resisted, and the culture of AI development needs to be shifted towards responsible AI (28m52s).
• Developing pedagogical provocations to destabilize hegemonic values in AI development is necessary, but this can be challenging, especially in corporate contexts (29m34s).
• Supporting learning in situ or in context, and fostering mindsets that apply RI concepts in development, is essential for meeting learners where they are (29m54s).

Supporting Responsible AI During Prototyping with Large Language Models (Lines 66-67)

• A study on supporting responsible AI during prototyping processes, led by Jay Wang, explored how to develop AI applications using large language models (LLMs) in a responsible manner (30m14s).
• The study found that developing AI applications using LLMs is now easy, but it raises questions about who is an AI practitioner and how to ensure responsible AI practices (30m30s).

Late-Stage Fairness Evaluations and the Need for Proactive Design (Lines 68-72)

• A survey of 300 machine learning practitioners found that nearly half encountered fairness issues in their products, with 99% of those issues discovered after model deployment (31m26s).
• The same survey found that many machine learning practitioners believe they have to release their model and then address fairness issues if someone raises concerns (31m16s).
• A study analyzed 30 responsible AI or AI ethics toolkits and found that most focus on downstream development phases, such as model training, testing, deployment, and monitoring, rather than design phases (32m8s).
• The study aimed to support responsible AI by fostering a mindset during the design phase, moving away from development and deployment towards ideation and prototyping phases (32m45s).
• Most learning resources focus on development phases, with few focusing on design phases or design skills, which motivated the focus on proactive design for generative AI (33m0s).

The Farsight Tool for Proactive Harm Ideation (Lines 73-78)

• A formative study and codesign study were conducted with AI prototypers using Google's AI Studio to generate and critique design ideas for proactive ideation about potential harms (33m21s).
• The study used the API for the AI incident database, which contains news articles about AI incidents, to extract embeddings and compute similarity between prompts and headlines (33m49s).
• The study developed a tool called Farsight, which can be used to analyze the similarity between system prompts and AI incident headlines, helping designers proactively identify potential harms (34m46s).
• A tool called Farsight has been developed to help identify potential harms of AI models by analyzing the cosine similarity between a given prompt and AI incidents from a database, and suggesting potential use cases and misuses based on embeddings (34m54s).
• The tool also includes an interactive tree visualization that suggests use cases, stakeholder groups, and potential harms based on a harms taxonomy from Renee Shelby (35m40s).
• The goal of Farsight is to foster ideation about potential harms early in the prototyping phase, before a product has been built or launched (35m24s).

Evaluating the Effectiveness of Farsight in Harm Ideation (Lines 79-82)

• A study was conducted with 42 participants to understand how Farsight changes the approach to identifying harms, and to see if it can augment the harm ideation process (36m38s).
• The study had three conditions: a full version of Farsight, a light version called Farsight Light, and a control condition using a PDF of a harms taxonomy (37m7s).
• The study found that after using Farsight, users were able to envision more harms independently, and focused more on users and use cases than the control group (38m28s).
• The tool helped users think more about potential harms, and the study suggests that Farsight can be an effective tool for augmenting the harm ideation process (38m45s).

Addressing Longer-Term Harms and Mitigations (Lines 83-84)

• Longer-term or second-order harms, such as cascading failures, can occur when one event leads to another, resulting in unforeseen consequences (38m47s).
• The tool Farsight does not provide mitigations for these harms, but users who utilized it considered more possible mitigations during the prototyping phase (39m8s).

The Need for Further Research on Farsight's Impact and In-Situ Interventions (Lines 85-86)

• While participants found Farsight useful and usable, it is essential to conduct studies on its effectiveness in changing development practices or outcomes in ecologically valid settings (39m42s).
• There is a need for in-situ interventions to motivate learners during development, especially for practitioners who may not seek out training or learning resources independently (40m1s).

The Role of Subjectivity and Community Engagement in Harm Assessment (Lines 87-90)

• The trade-offs between automation and human agency are raised, as there is a risk that developers might offload harm identification to the tool, undermining its purpose of supporting reflexivity and independent ideation (40m27s).
• The study highlights the role of subjectivity, including people's unique positionality, backgrounds, lived experiences, and domain expertise, in identifying harms and evaluating their likelihood and severity (40m52s).
• The low interrater reliability among third-party raters who evaluated the likelihood and severity of harms suggests that subjectivity and lived experiences play a significant role in these assessments (41m33s).
• Engagement with communities impacted by AI systems through participatory design or co-design is encouraged to foster mitigation and address the limitations of technical mitigations (42m11s).

Design Choices as Mitigation Strategies and the Importance of Early Intervention (Lines 91-93)

• Design choices can be a mitigation strategy, but this aspect was not supported by the tool, and further exploration is needed to understand its potential (42m20s).
• Different design choices at the beginning of a project could potentially avoid some of the harms associated with AI, and intervening in the prototyping phase is crucial (42m40s).
• The focus should be on designing the right thing, rather than just designing the thing right, as emphasized by a Bill Buxton quote (42m50s).

Integrating Social and Technical Topics and Assessing Sociotechnical Skills (Lines 94-95)

• Integrating social and technical topics in responsible AI is essential, rather than siloing them in learning objectives, assessment, or resources (43m9s).
• Engineers working in AI need to move towards higher-order cognition, including application, analysis, evaluation, and creation, and assessing these sociotechnical skills is crucial (43m28s).

Balancing Scalability and Collaboration in Responsible AI Learning (Lines 96-97)

• Balancing scalability and collaborative learning approaches is necessary, and determining what all developers need versus different roles is essential (43m47s).
• Designing learning environments that incorporate responsible AI principles is vital, and moving this work upstream from pre-launch review to early-stage problem formulation or design is necessary (44m1s).

Adapting Participatory AI Methods for Pre-trained Foundation Models (Lines 98-100)

• Participatory AI methods, such as codesign and value-sensitive design, need to be adapted for the current paradigm of pre-trained foundation models, which can impede responsible AI work (44m54s).
• Research has shown that nearly all papers on participatory AI focus on the user interface, rather than broader design questions, highlighting the need for more methodological approaches (44m30s).
• Emerging policy and regulatory work, such as the EU AI Act and US NIST, calls out human factors in risk management, emphasizing the importance of design methods in responsible AI work (46m4s).

The Learning and Design Problem of Responsible AI and Open Questions (Lines 101-102)

• The field of responsible AI has a learning and design problem, and it is essential to conceptualize it as such to address the challenges it poses (46m19s).
• There are open questions about how to support the work of responsible AI and how to evaluate its effectiveness in actual products (46m16s).

Evaluating the Impact of Policy and Responsible AI on Product Development (Lines 103-109)

• A question was raised about the impact of policy and responsible AI (RI) on the actual product being built and whether there is an assessment or evaluation of each stage of the process and its effectiveness on the product (46m59s).
• Another question was asked about whether users should be aware of biased models or problems with the current system and if they should be warned or educated about these issues (47m14s).
• Evaluations are happening at each stage of the responsible AI process, including model-level and product-level evaluations, but more cross-pollination across these stages is needed (47m50s).
• Laura Whinger and her collaborators at Google Deep Mind proposed a framework for sociotechnical evaluations that includes risks that any product application using a base model or pre-trained model might encounter (48m0s).
• Evaluations should include risks, interaction evaluations, and broader societal or systemic evaluations to ensure that responsible AI is effective and improves the user experience (48m31s).
• There is a need for more mechanisms to surface risks or harms encountered during user studies or product development and to develop more effective evaluations that improve the user experience (49m24s).
• The effectiveness of evaluations in improving the user experience is a distinct question that requires further exploration (49m45s).

Supporting User Awareness and Education about Responsible AI (Lines 110-112)

• In the US, the NIST framework is not binding legislation or regulation, but federal agencies such as the Federal Trade Commission and the Equal Employment Opportunity Commission are working to enforce violations within their specific purview (49m59s).
• There is a need for more support for users in learning about responsible AI harms and guidance on how to use it responsibly, but users should not be solely responsible for using systems responsibly, especially if they were not developed responsibly (50m31s).
• Communicating the limitations of language models, such as hallucinations, through broader education or AI literacy could inform users on the appropriate use cases for AI (51m0s).

Resources and Toolkits for Community Organizations and the Broader Public (Lines 113-114)

• Resources and toolkits, such as the Algorithmic Equity Kit led by U Peaks Craft, Mike Catel, and Meg Young, are available for community organizations to advocate for responsible AI (51m33s).
• There is a lack of AI ethics toolkits geared towards community organizations and the broader public, with most being targeted towards developers (52m0s).

The "Ship It Now" Mentality and the Challenges of Bias and Fairness (Lines 115-117)

• The "ship it now" mentality in the industry can lead to models being deployed without proper consideration for bias and fairness, resulting in issues being discovered later on (52m38s).
• The design phase for models is still an art and science, and figuring out the data distribution to ensure fairness is an unsolved problem (53m31s).
• Bias, fairness, and responsibility are unsolved problems not only in generative AI but also in supervised learning and classical machine learning (53m55s).

Resource Availability and Organizational Incentives in Responsible AI (Lines 118-119)

• There are more resources available to support responsible AI, but their effectiveness is uncertain, and organizational incentives often prioritize shipping products over responsible AI practices (54m7s).
• Many teams focus on data collection and processing, which provides opportunities for intervention, but pre-trained models pose a challenge as they are often trained by different companies or teams on undisclosed data sets (54m20s).

Challenges with Pre-trained Models and Data Transparency (Lines 120-121)

• Third-party companies using pre-trained models through APIs may not know what data was used to train the base model, making it difficult to evaluate, remediate, or retrain the model (55m12s).
• In high-stakes settings, traditional machine learning models may be more widespread due to their auditability and traceability, but there is still room for improvement in responsible AI practices (55m41s).

The Importance of Cross-functional Collaboration and Shared Language (Lines 122-129)

• Differences in how teams think about implementing responsible AI and where they begin from can impact the learning resources provided, highlighting the need for a shared language and understanding of terms like fairness (56m31s).
• Cross-functional collaboration is essential for responsible AI, but teams from different disciplines may have different knowledge and perspectives on the issue, making it challenging to develop a shared approach (57m6s).
• A lack of shared language and the use of "suitcase terms" like fairness can hinder collaboration and responsible AI practices, emphasizing the need for clearer definitions and meanings (57m43s).
• Different professionals, such as data scientists and UX researchers or social scientists, may have varying interpretations of the term "fairness," with data scientists possibly referring to equalized odds or demographic parity, while UX researchers or social scientists may have a different understanding (57m55s).
• Interdisciplinary collaboration can be challenging due to differences in terminology and understanding, highlighting the need for more learning resources to facilitate effective communication across disciplines (58m10s).
• There is a need for both role-specific and general trainings to address the complexities of responsible AI, as well as to enable professionals to speak across disciplinary boundaries and understand the basics of model training and design (58m19s).
• Every role involved in AI development should have a basic understanding of AI concepts, such as model training and fairness, to facilitate effective communication and collaboration across disciplines (58m41s).
• Having a shared understanding of AI concepts can help identify potential interventions or design opportunities, ultimately contributing to the development of more responsible AI systems (58m58s).