Stanford Seminar - From Flat to Phantasmal

Introduction to Spatial Computing

• The speaker, Jasmine, provides an overview of augmented reality (AR), virtual reality (VR), and mixed reality (MR), clarifying terminology and discussing contributing fields such as computer graphics, optics, AI/ML, HCI, psychology, neuroscience, and sensors.
• Jasmine shares her personal journey into AR/VR, explaining how her background in chemistry and physics led her to question the nature of perception and reality.

Challenges in AR/VR Adoption

• Jasmine addresses the question of why AR/VR hasn't gained widespread traction despite being around since the 1980s, suggesting that onboarding difficulties and lack of intuitive controls are key issues.

Smartphones as an Entry Point to MR

• The speaker emphasizes the potential of smartphones as an accessible entry point to mixed reality experiences and proposes three ways in which a smartphone can interact with MR: as a window, a scanner, and a controller.
• Jasmine provides examples of each use case, showcasing various AR apps and projects, including her work on the Depth API at Google.

Object Detection and Large Language Models (LLMs)

• The speaker discusses the importance of object detection for AR/VR, especially in the context of contextualizing and adding information to the user's environment.
• They introduce the concept of "Speaking the World into Existence," which involves using LLMs to generate virtual worlds based on text prompts.
• Challenges of using LLMs for game design are highlighted, emphasizing the need for a balance between deterministic and non-deterministic components to avoid complete hallucination by the model.

Generative AI for Game Design

• The speaker discusses the use of generative AI for importing and scaling 3D assets, simplifying the process for developers.
• They demonstrate a multiplayer pong game created using LLMs, showcasing the ability to dynamically change objects and interact with the game world through text prompts.
• The speaker concludes by emphasizing the potential of LLMs as ontology engines, capable of generating content based on the relationships and combinations of objects in the model's knowledge base.

Cross-Pollination of Ideas and Research

• The speaker discusses the lack of cross-pollination between game design, game discourse, game pedagogy, HCI, and computer science pedagogy.
• They emphasize the importance of considering research from outside one's immediate field and provide an example of how integrating OpenAI models into game engines has become less novel due to recent developments.

Material Science and Haptics in AR/VR

• The speaker expresses their passion for understanding how objects interact with each other in the natural world and how this can be replicated in generative AI.
• They mention their work on using material science databases to inform the physical properties of objects in game engines.
• The speaker emphasizes the importance of haptics in AR/VR experiences and expresses surprise that it is not being prioritized in the current iteration of AR/VR technology.

Social Implications and Ethics in Spatial Computing

• The speaker briefly discusses the psychology and neuroscience of social VR, including the importance of ethics, psychology, and human-to-human interaction in creating positive and non-toxic environments.
• They highlight the need for diversity and inclusion in tech and the challenges of relying solely on speech-based input systems.

Overlooked Aspects of Spatial Computing

• The social implications of spatial computing, such as ethics and historicity, are often overlooked in the industry due to its hyper-individualistic culture.
• Research institutions prioritize citing and building upon previous work, while industry tends to overlook historical research, leading to the reinvention of existing concepts.
• The goal of replicating reality in virtual spaces may not be the best use of technology, and stylized representations can be more effective.

Augmented Virtuality and Accessibility

• Augmented virtuality experiences, which combine digital and tangible elements, have gained attention due to their grounding in physical reality and potential for haptic feedback.
• Challenges such as comfort, battery life, and motion sickness are still barriers to widespread adoption of virtual reality headsets.
• Apple's AR/VR headset is a step in the right direction because it works well and is easy to set up.
• Web-based experiences and easy onboarding are key to making AR/VR more accessible.

Insights from Theater and Gaming

• Theater and stage design can provide valuable insights into user attention and navigation in VR.
• The gaming community has made significant contributions to spatial computing, and HCI researchers should pay more attention to their work.

Education and Learning in Spatial Computing

• Spatial learning is not prioritized in education, and there is a need for more courses on spatial reasoning and computer science.
• Learning materials for AR/VR often only cover basic concepts and do not provide enough depth or interactivity.

Current State of AR/VR and Future Considerations

• The speaker believes that the field of AR/VR is still trapped in prototyping and ideation, and that the general public already has a mental model of what AR/VR