Stanford Seminar - Living Scenes: Creating and updating 3D representations of evolving indoor scenes

Introduction and Motivation

• The speaker is from the Civil and Environmental Engineering department, but their team consists mainly of roboticists and computer scientists working to solve problems in the building industry using computer vision (22s).
• The team's work focuses on understanding what exists in a space, how it was constructed, and how it changes over time, with the goal of creating sustainable, inclusive, and adaptive built environments that prioritize human needs (46s).
• The team is also exploring the intersection of physical and digital spaces, including immersive technologies like VR and AR, and designing physical spaces that allow interaction between the physical and virtual worlds, which they call "gradient realities" (1m12s).
• The team's lab is called "Gradient Spaces" and is working on creating and updating digital replicas of evolving indoor scenes, which they call "living scenes" (1m31s).

Creating and Updating 3D Representations of Evolving Indoor Scenes

• The concept of living scenes is based on the idea that buildings are like living organisms that evolve over time, and the team is working on developing methods to realistically make, maintain, and update their representations throughout their lifespan (1m53s).
• Agents that exist in the environment can navigate, act, and interact with their surroundings, but need to be able to map and understand the environment to do so (2m4s).
• The team is working on developing methods to align and merge spatial and temporal data collected by agents to create evolving representations of indoor environments (2m53s).
• By acquiring data from agents performing repetitive tasks within a single scene, the team can create a cumulative scene understanding and representation that improves in geometric completeness and accuracy over time (3m20s).
• This can enhance interaction with objects within the scene by having a more accurate geometry, and is particularly useful for scenes or parts of scenes that have not been seen before (3m51s).
• Creating and updating 3D representations of evolving indoor scenes involves understanding how objects move within the scene and having a foundational understanding of the scene's geometry and semantics (4m7s).

Methods for Acquiring 3D Representations

• Two methods for acquiring 3D representations at a single temporal point are Loop Splat and Adaptive Realtime 3D Semantic Understanding (4m41s).
• Loop Splat uses the 3D cion Splat representation to reconstruct the scene with great accuracy and can perform Loop detection and Loop closure by registering 3D cion Splats together (4m47s).
• Loop Splat's innovation is its ability to minimize drift from the ground truth trajectory by changing the color of the trajectory when Loop closure is detected (5m2s).
• Adaptive Realtime 3D Semantic Understanding creates a single map with adaptive quality, meaning certain areas can be in high fidelity while others are in low resolution, depending on user-defined semantics or geometric complexity (5m31s).
• This adaptive approach to 3D semantic understanding prioritizes sustainability by collecting only necessary data and reducing resolution in areas of less importance (5m38s).

Relocalizing and Reconstructing Objects in Evolving Environments

• For evolving environments, the goal is to relocalize objects within the scene and reconstruct them on an individual object level given sparse 3D observations (6m48s).
• A method for achieving this involves instance matching, registering different point clouds, and relocalizing objects within the scene (7m16s).
• This method assumes instance segmentation and is robust to noise, as it has been tested with both ground truth and predicted instances (7m29s).
• The method takes as input the scene at two temporal points where changes have occurred and aims to reconstruct the scene and relocalize objects (7m20s).
• A 3D representation of evolving indoor scenes is created by reconstructing point clouds of objects over time to achieve a more complete and accurate geometry and accuracy of the object instance (7m52s).
• The representation is able to solve three tasks - matching, relocalization, and reconstruction - by utilizing different embedding spaces, specifically equivariant and invariant feature spaces (8m6s).
• The model is trained only on synthetic data from the ShapeNet database and evaluated zero-shot on a real-world Noy data set, demonstrating its ability to reconstruct unseen parts of objects by understanding geometric priors (8m19s).
• The model uses a vector neuron encoder to provide an equivariant and invariant feature space, and a DSDF encoder for shape completion, making it category-agnostic but trained on seven categories from ShapeNet (8m42s).
• The equivariant embedding space provides information about the pose of objects in the scene, while the invariant embedding space provides information about the actual geometry and shape of the objects (9m8s).
• Qualitative results on synthetic data sets show the model's ability to track objects and reconstruct their geometry and shape over time (9m28s).
• The model is also evaluated on a real-world 3RScan data set, which includes multiple scans of the same scene over time, demonstrating its ability to handle temporal changes and reconstruct the geometry and completion of objects (10m1s).
• Experiments show that accumulating point clouds from different viewpoints and temporal times improves the model's ability to reconstruct the geometry and completeness of objects and minimize registration error (10m51s).
• The model's performance is compared to a baseline, demonstrating its ability to outperform it in terms of geometry and completion accuracy (10m27s).

Scene Graph-Based Representations and Alignment

• A second approach to evolving scene representation uses sing graphs, a work done by a student named Shiyen, which was submitted to ICV 2023 (11m27s).
• A map can be created in a low-level manner using representations such as occupancy maps, voxel grids, octo maps, hash grids, point clouds, and others, but these methods have limitations, including decision-making taking place on the metric space, which can limit higher-level understanding and generalization (11m46s).
• Building a map in a higher level can be achieved using 3D scene graphs, which allow for both high-level and low-level information, enabling decision-making on a more abstract space, and are lightweight and privacy-preserving (12m28s).
• D scene graphs are used in agents to build maps on the fly, perform robotic navigation, or task completion, and are a representation that many robotics agents are already benefiting from and using (12m51s).
• The goal is to leverage the information that agents are already building in the background to create 3D maps of environments, which can be static or changed scenes with overlaps between zero to partial to full (13m22s).
• The SGN Aligner is a method that takes scene graphs as input and performs node matching to identify how two graphs are aligning together, providing a great initialization for tasks such as point cloud registration or point cloud mosaicing (13m45s).
• Existing methods for point cloud registration have limitations, including focusing on local feature descriptors, which can lead to issues with changes in the scene, low overlap, point cloud density, and large scenes (14m29s).
• The SGN Aligner forms the alignment of scene graphs as multimodal knowledge graphs, which have three types of information within them (15m7s).
• SC graphs represent semantic entities in a scene, including object instances with attributes such as category, size, and material, as well as relationships between entities like relative position or attribute similarity (15m19s).
• Entity alignment methods from the multimodal Knowledge Graph domain can be redesigned to align spatial maps together, but existing works assume overlapping and accurate information, which is not the case with 3D graphs built by agents (15m51s).
• SG liner is a method that takes 3D Sy graphs as input and uses unimodal embeddings, including Point Cloud encoders, structure encoders, and met encoders, to encode modality information separately and interact in a joint space (16m39s).
• The goal of SG liner is to align the same object instances closer together and different object instances further apart, enabling tasks such as Point Cloud registration (17m12s).
• The method was evaluated on the 3R scan data set and its extension, 3D SSG, and achieved robust performance in matching nodes together, even with predicted sing graphs and low overlap conditions (17m42s).
• SG liner also performed well in aligning 3D Sy graphs with temporal changes and varying overlap conditions, outperforming usual 3D Point Cloud registration methods (18m11s).
• The method can match nodes together and figure out whether they align together, even with a large number of nodes, and can handle scenarios with 50% or all nodes matched (18m47s).
• The performance of the system is improved when at least two nodes are matched for the particular graphs, and this information is used to perform 3D point cloud registration (18m56s).
• In contrast to previous approaches, the system registers object instances instead of entire scenes by calculating the registration between object instances that are matched with each other (19m34s).
• This approach allows for a more robust and faster alignment of the entire scene, resulting in a 49% improvement in chamfer distance and a 40% improvement in relative translation error (20m4s).
• The system also performs well with noisy point cloud predictions and can handle cases with 10 to 30% low overlap (20m17s).
• The system can identify overlapping pairs of point clouds more correctly and three times faster than prior art, which is useful for robotics platforms (21m21s).
• The system can also handle scenes with zero overlap, where previous methods are not able to perform robustly (21m2s).
• The system's geometry-based and semantic-based alignments are designed for scenes that evolve with minimum changes in geometry, such as furniture being relocated or added/removed (21m42s).

Spatiotemporal 3D Point Cloud Registration for Large-Scale Changes

• However, the system can also handle more drastic changes in the scene, such as those that occur over time, through spatiotemporal 3D point cloud registration (22m9s).
• This approach involves finding pairwise correspondences from the static parts of the scene and excluding temporal changes (22m13s).
• The system is tested on datasets with small-scale scenes, such as rooms, and captures standard daily human interaction activities (22m28s).
• A student, Tan, has performed work on spatiotemporal 3D point cloud registration, which is an extension of the original system (22m3s).
• Existing methods for 3D Point Cloud temporal registration can only handle small changes in the geometry of a scene, such as those found in self-driving car scenarios, but struggle with large changes, like those found in construction sites (22m52s).
• Construction sites are a particular scenario where large, drastic changes in the world occur in a small amount of time, making them a challenging environment for existing methods (23m46s).
• The "Nothing Stand Still" benchmark was created to evaluate the performance of existing special temporal Point Cloud registration methods on scenes with large changes, such as construction sites (24m7s).
• The benchmark dataset was collected from different construction sites over time, using a tripod-based device to capture the scene at multiple temporal points (24m26s).
• The dataset includes interior layout construction scenes, with a focus on the slabs, ceilings, walls, and empty spaces, but excludes exterior elements like foundations and excavators (25m36s).
• The dataset is challenging due to the large changes in the scene, inconsistent capture over time, and inaccessible areas, making it difficult to align the Point Clouds temporally (25m5s).
• The benchmark evaluates both pairwise and multi-way registration of the scene, allowing for the assessment of existing methods' performance on very large scenes (25m21s).
• The dataset includes snapshots of the interior layout construction scenes, showcasing the drastic changes that occur over time, from empty spaces to the addition of walls, insulation, pipes, air ducts, and materials (25m54s).
• Indoor scenes have repetitive elements, such as studs in walls, which can make registration algorithms struggle to match corresponding elements due to their similar appearance (26m19s).
• The environment is also very uniform, with most elements being gray or brown, making it difficult to perform tasks in a robust manner (26m46s).
• The data set shows the interior layout being constructed, with changes over time, including the addition of static furniture (27m2s).
• Explorations of the meshes in virtual reality demonstrate how spaces change over time and the complexity of the scenes (27m20s).
• Pairwise registration methods are typically used, taking two small point clouds as input, performing correspondent estimation, and then using RANSAC and ICP to determine the final transformation (27m36s).
• Multi-way registration is also used, connecting point clouds over space and time with edges, and minimizing the weighted RMSD of the poses of the POS graph (28m1s).
• Most existing methods struggle to handle multi-way registration, but a recently developed method has shown better performance (28m23s).
• Before and after multi-way registration, the best-performing algorithm at the time shows improvement, but still with some failures (28m36s).
• The colors in the visualization represent different temporal points, with each color indicating a different point in time (28m52s).
• Better algorithms are needed to perform tasks involving large, drastic changes in the environment, and solving this problem could lead to solutions for other related tasks (29m1s).
• The assumption is that if this hard problem can be solved, other related problems can also be solved, and more people should work on this using the provided data set (29m10s).

Applications in the Building Industry and Circular Economy

• The goal is to create and update representations of evolving indoor scenes, and this technology has potential applications in various fields, including civil and environmental engineering (29m19s).
• The building industry is a significant area of focus, with renovation being a major scope in architecture, engineering, and construction (29m46s).
• The construction industry aims to increase sustainability and create a circular build environment by reusing materials from demolished buildings in new designs, extending the life of existing buildings, and utilizing existing resources without depleting new ones (29m51s).
• Most buildings lack digital information, as computer-aided design software became widespread only after the 1980s, leaving billions of buildings on Earth without digital representations (30m21s).
• The construction industry is extremely expensive, with 50% of construction costs increased due to rework, and poor estimations of costs due to a lack of knowledge about the building process (30m55s).
• Construction workers accounted for 20% of all occupation fatalities in 2020 in the US, with around a thousand people killed in construction sites due to errors (31m28s).
• % of non-hazardous construction and demolition material is either reusable or recyclable, but often ends up in landfills (31m44s).
• Understanding spatial and temporal information can have a significant impact on human life and planet sustainability, motivating researchers to work on these problems (32m1s).
• The model can be applied to improve the circular economy and reduce construction costs by capturing information about existing materials in buildings, allowing for better planning and harvesting of materials during demolition or new building design (32m50s).
• Potential application scenarios include taking down old buildings, capturing information about existing materials, and planning ahead to harvest materials from demolished buildings for use in new designs (32m39s).
• The goal of circularity is to plan ahead and know what materials will be available when creating a new building, allowing for the harvesting of materials from demolished buildings and designing with those materials in mind (33m7s).
• To understand evolving indoor scenes, it's essential to consider not only new construction but also the deterioration of materials and their current condition, especially in areas with no new construction, and to build and update a map of the space (33m56s).
• For these operations, both geometric and semantic information are necessary, which involves understanding where things are and what they are, enabling better planning for sustainable building construction (34m27s).

Knowledge Graphs and Scene Comparison

• Knowledge graphs are used to represent relationships between entities, with nodes representing objects categorized using object categories, and relationships including semantic and geometric connections (35m5s).
• These relationships can be relative, such as "in front of" or "to the left of," and are typically taken from a particular viewpoint, including object instances, attributes like size and material, and relative relationships within the space (35m39s).
• Comparing point clouds or 3D scene graphs to existing CAD or BIM models is challenging due to issues with completeness, scale, and level of detail, making geometric alignment difficult and often inaccurate (36m32s).
• While 3D scene graphs might be more robust for alignment, current methods are insufficient, and researchers are working on addressing these challenges (36m14s).
• The method can be used to compare a CAD model to a point cloud, allowing for the comparison of a built environment to its designed state, but this is more challenging in construction settings where detailed building information models or CAD models are not always available (37m15s).
• The method does not infer any physics from the scene, such as whether an object is brittle or will break if it falls, and assumes that any changes to the scene have already occurred and will not evolve further (37m54s).
• The scene graphs created by the method are in JSON format and could potentially be loaded into Unity or other simulation software, but this has not been attempted (38m40s).
• The method has not been used for simulations or virtual reality applications, but it could potentially be used for these purposes (39m14s).

Data Sets and Future Work

• The data sets used to test the method are existing data sets that were not produced by the researchers, and are available on GitHub (39m36s).
• The researchers are working on tools to automate the annotation of 3D geometry in construction sites, but this work is still in its early stages (40m6s).
• The method could potentially be integrated with Internet of Things (IoT) devices, such as sensors that store information about the materials and condition of objects in the environment, but this is not currently being explored (40m25s).
• The deconstruction process can be enhanced by using living scenes to understand the materials present in a building and what can be harvested, such as the size of panels, windows, and ducts, providing an initial estimate and hypothesis for potential reuse (41m11s).
• Connections between materials, such as glues and nails, play a significant role in deconstruction, but this information is often not captured in the data, making it harder to have an accurate estimate of what can be reused (41m47s).
• The deterioration of materials behind what is visible is also a challenge, as it is unknown what is happening behind the wall 50 or 100 years later, making assumptions necessary (42m32s).
• Workers can use the information from the models to detect which parts can be reused and how to cut them, but an on-the-spot survey is still necessary for accurate information (41m32s).
• The approach does not eliminate the need for on-site surveys, but it helps narrow down the best options for new designs by matching the existing building's characteristics (43m27s).
• For construction progress monitoring and deconstruction, even a basic sensor would be sufficient, as there is currently a lack of data, but laser scanners are also useful, although they need to be fast and efficient to collect data on a large scale (44m29s).
• Current methods for capturing 3D representations of indoor scenes, such as using a laser scanner, can take a long time to complete one rotation, making them not scalable for large areas (44m54s).
• Backpack systems are faster and similarly accurate but may have issues with potential drift, and they capture a large number of points that can be hard to process (45m0s).
• A method like Adaptive Reconstruction can be helpful for construction progress monitoring as it focuses on the newly installed elements and their correct position and time of installation (45m20s).
• Imagery can also be used to capture the installation of elements, but it may not provide perfect point clouds, and sizes could be off; however, it can still provide valuable information on what has been installed and when (45m41s).
• Laser scanners with adaptive reconstruction capabilities and the ability to perform fast iterative processing, combined with images, can provide the best solution for capturing 3D representations of indoor scenes (46m1s).
• Images are necessary to understand materials, as point clouds cannot provide this information, and high-frequency information from images is needed to characterize materials (46m19s).
• Material characterization can be done using visual information from images, but it may not always be accurate, and additional documentation or information on the planned installation of elements can help make this decision easier (47m15s).
• The model can also work in non-confined spaces, such as outdoor areas like squares, roads, or parks, where there is no clear ending edge around the whole space (48m25s).
• A model can be used to create a 3D representation of an urban area, such as a city, to determine potential locations for solar panels, for example on rooftops, rather than on the ground (48m43s).
• Aerial scanning or imagery, such as that provided by Google Street Maps, which has 3D information in certain cities, can be helpful in this task (49m4s).
• The Living Scenes model works by doing instance matching, and while it has not been tried outdoors, the Singra work, which operates in the semantic space, would likely generalize more easily to outdoor environments (49m20s).
• The Living Scenes model is restricted to being trained on certain categories, which would need to be expanded to accommodate outdoor environments, and one potential workaround is to consider open scene understanding (49m32s).
• Open scene understanding is a possible extension of the Living Scenes model, but it has not yet been explored (49m41s).