Augmented Reality (“AR”) refers to any technology that ‘augments’ the user’s experience of the real world by superimposing computer-generated information into the user’s field of view in the form of 3D holograms, images, or text, with the help of an AR device like the Microsoft Hololens products.
We are exploring the role that these immersive technologies can play in improving visual inspections when they are combined with data-intensive digital twins. A digital twin is a virtual representation of a physical asset such as a bridge, dam, or tank, that is updated with information from the real world as regularly as possible so that the analysis of the digital twin will continue to apply to the physical asset. Aerial scans by drones are often the ideal way to collect this data and Qii.AI is the ideal tool to manage and analyze it, allowing users to detect and evaluate changes over time.
Experimentation is a vital part of our developmental process. The purpose of our augmented reality pilot project is to explore how it can power the inspection workflow with the Qii.AI software suite. Using AR, we can blend physical and digital worlds in terms of interaction and visualization, so that we can analyze and review visual data in a 3D environment, allowing us to get past the restrictions imposed by the 2D screens that are using today. We are developing a minimum viable product to determine the feasibility of eliminating this restriction and providing a more realistic, ‘digital twin’ like experience to interact with these assets.
Our application is built for Microsoft HoloLens 2, a wearable head-mounted AR display. The HoloLens uses a set of cameras and sensors to understand spatial information and applies that information to enhance the immersive experience. It was developed using the Unity game engine, with scripting done in C#. The open-source Mixed Reality Toolkit scripts by Microsoft are used to provide foundational components and building blocks for spatial interaction.
The application will enable a user to access inspection data in Qii.AI over the cloud. The user can then initialize a session to visualize and interact with the digital twins in AR. Due to the highly complex nature and level of detail of these digital twin assets, it is not feasible to load the models onto the device and render it locally yet. Rendering these huge datasets locally will result in poor performance and highly optimized models with reduced triangles. This is due to the limited computation rendering power on mobile devices. To address this issue, the application connects to a remote service that renders high-quality models in real-time and streams them to the device as video. By moving the rendering workload to capable GPUs in the cloud, a smoother experience can be achieved while visualizing high-fidelity models. The rendered model can be scaled up and down, rotated along different axes and placed anywhere in the space around you to better understand the dynamics of the digital twin. This model can be populated with tooltips representing different anomalies on the structure sourced from the inspection. A virtual overview of the site and assets along with the data associated with it can be visualized in any indoor space with internet connectivity.
The value created through the ability to do this in AR is that companies can optimize their inspection workflows to increase efficiency in different ways. Bringing the digital representation of a structure into the office can save time for the employees and reduce decision-making times. A clear picture of the situation can be analyzed without physically being there. They can use the information from the digital twin’s inspection for predictive maintenance and adapt their schedules efficiently. Scenarios and solutions concerning the site can be simulated and evaluated to reduce cost, time, and downtime. The AR experience also has utility for training new employees and routine equipment maintenance in a safe environment. Furthermore, the application can be extended for real scale, on-site visualizations, aligning the digital twin against its physical counterpart, and overlaying relevant data. The integration of IoT sensors can also stream real-time data regarding the physical environment into its AR view. There is also scope for collaborating with different users in AR with shared virtual spaces.
The major strength of AR lies in how it can use data to enhance the way we visualize and interact with the real world, rather than shape it. This enhanced, immersive experience surely has a role to play towards improved and digitally transformed industrial and infrastructure asset inspections. This is a quick preview of our experimental AR project in its early stages. Stay tuned for demos and updates on our progress!
