XR Brings Humans and Robots Closer on the Factory Floor

As collaborative robots move onto manufacturing lines, extended reality (XR) is fast emerging as a practical tool to ensure they work safely and efficiently alongside humans. The European Union Horizon project MASTER (Mixed reAlity ecoSystem for TEaching Robotics in manufacturing) is exploiting the latest XR tech to develop an Open XR platform, which is set to reshape how workers are trained, how robots are programmed, and how the two collaborate on factory floors. “XR has now reached a level in which you can deploy it fairly economically,” said Johan Kildal, human-computer interaction researcher and project manager at Spanish research and technology organization Tekniker, a technology-providing partner for MASTER. “For example, headsets are easily available with disorientation issues vastly improved, and it’s also more straightforward to create any virtual or augmented environment—the time is right for this.” MASTER’s Open XR platform is based on an enterprise XR platform, VIROO, developed by Spanish virtual-reality tech firm Virtualware. As Kildal pointed out, the platform helps organizations to build and run immersive VR and mixed-reality training and simulation environments, in which several remote users can interact in real time within the same virtual scene. “Its setup is ideal for training groups of people—this is one of the greatest assets of the platform,” he said. Building blocks To get the Open XR platform ready for XR training in robotics, Kildal and project colleagues have added three features to VIROO: safe robotic environments, code-free robot programming, and gaze-based interaction. As part of the safe robotic environments extension, Panagiotis Karagiannis, MASTER coordinator and project manager at the Laboratory for Manufacturing Systems and Automation (LMS) at the University of Patras (Greece), has been developing virtual safety zones for factory robots for the platform. Industrial robots are typically enclosed by physical fences to create a safety boundary. But instead, virtual safety zones rely on safety-rated sensors, such as laser scanners and vision-based monitoring systems, and controllers on the robot to trigger a speed reduction or protective stop when a human enters the defined space. To reduce workspace restrictions while maintaining safety, Karagiannis and LMS colleagues took this concept further by developing dynamic safety zones that use sensors to track the robot’s real-time position and trajectory and create smaller, localized safety zones that move with the robot. They have now recreated these zones within the Open XR platform so that operators can be trained to understand virtual zone behavior and learn safe movement patterns before entering real production. “Our research has always focused on how humans and robots can collaborate, and this approach in MASTER will now ensure both the robot and human can work in the same space safely,” Karagiannis said. “By introducing XR to the safety training, we are also providing a much more affordable educational tool to factories.” Alongside the safety feature, Kildal and colleagues at Tekniker have been working on codeless programming of robots, in which robots are taught to perform tasks without manually writing traditional source code. Instead, the robot’s control logic is generated automatically from natural interactions between a human operator and the robot—or, in this case, via its digital representation, or digital twin, in the Open XR platform. According to Kildal, in VR, the operator can grab and move the robot arm to demonstrate, say, simple pick-and-place actions. The Open XR platform’s programming system, connected to the robot’s digital twin, can then capture the motion trajectories, object positions, and spatial relationships, which are translated into structured robot instructions. Then, once validated on the platform, the new program is translated to the real robot ready for execution. “The robot can follow your gestures, it can understand your speech [simple commands], and all of this interaction generates code,” Kildal said. “So at the end of the day, you’ve actually programmed that robot, but you haven’t had to write the code and you don’t have to debug it later. And because you’ve done all this in virtual reality, the productivity of the actual factory line hasn’t suffered.” Eye-tracking tools In recent years, gaze interaction has made inroads into XR, thanks to the integration of eye-tracking sensors into head-mounted displays. Given this, László Kopácsi and Michael Barz, researchers for interactive machine learning at the German Research Center for Artificial Intelligence (DFKI), have developed several tools for implementing eye tracking on MASTER’s Open XR platform. For example, their Gaze Interaction Toolkit is designed to ease the integration of gaze-based interaction to Open XR and other open-source XR platforms. Modules, including gaze-contingent information displays, visual attention monitoring, eye-tracking accuracy assessment, and tools for onboarding, are designed to help users build gaze-responsive interfaces in XR training applications. At the same time, a balloon-popping VR game called GazeDrift is designed to help users troubleshoot gaze-interaction artifacts and system errors, such as jitter, systematic shift, and reduced peripheral accuracy. “Gaze-based tracking is still a bit of a novel interaction method but is instrumental for people using training platforms [such as Open XR],” Kopácsi said. “We’ve introduced users to issues that can arise whilst using gaze to interact with a platform that might be a bit noisy.” With their tools developed, Barz believes the next step could be to analyze the sensor data coming off the headsets of Open XR platform users to better understand training needs. “We could understand exactly what the users are doing and looking at during their training and adjust to this in real time,” he said. “Then we could carry out inferences on how concepts are learned, which would enable the educators to react even better to their students or workforces.” Extending and validating With the fundamental Open XR platform and assets firmly in place, EU-funded XR researchers across Europe have been developing technologies to further enhance the platform, from both a technical and human perspective. For example, in a project called AAXLP, a voice-prompted tool for spawning, moving, and deleting objects such as robotic arms and forklifts replaces traditional button-based or drag-and-drop methods. Then, in MANIPULAY XR, users interactively assemble robot components, such as bases, joints, links, and grippers, and program their movements using command blocks in VR, helping them to better understand robot kinematics manipulator design. Taking a different tactic, i-MAR-XR has delivered a remarkably photorealistic digital twin of a cargo ship using cutting-edge 3D capture technologies to boost maritime safety training. Haptics have featured highly in the platform development, with Kildal pointing to two key haptic-glove-related projects: HEART and Magos Robot Programming (MRP). In HEART, users wear a SenseGlove force-feedback glove to teleoperate a humanoid gripper, developed by Aeon Robotics and modeled on the human hand, integrated into the Open XR platform. Users can feel the interaction forces exerted by the robot as it handles objects, helping them to quickly master force-based teleoperation in VR. Then in MRP, users wear Haptikos force-feedback, five-fingered gloves with finger tracking to interact with digital simulations of factory processes. “Like HEART, the haptic glove in MRP also lets you feel what the robot hand is feeling,” Kildal said. “These projects show that the dexterity of a humanoid robot is an important topic in robotics.” Kopácsi also highlighted how WAVE has developed an entire haptic shirt, as well as a glove, for collaborative robot training in XR. “This project sticks out as being really unique—the researchers developed the haptic glove and a blue worker shirt with a flexible inner layer containing inertial-measurement-unit sensors,” he said. “This could be used for gesture recognition as well as interacting with the virtual environment and robot itself.” Haptics aside, Karagiannis described how a software development kit, Dreamer, reconstructs 3D objects to be embedded in virtual scenes, so that users can create 3D content for populating simulated worlds for realistic, risk-free training scenarios with robotics. The software uses a self-supervised AI learning agent trained on streams of images that can rapidly extract a 3D mesh representation of a physical object. “For me, this is an interesting tool to create realistic 3D objects from the factory—it has great potential,” he said. “For example, in an office scenario for safety training, you could create 3D objects of, say, a desk, chair, and fire extinguisher—[and in the simulation,] it’s as if you are seeing the real objects.” With this array of novel XR tools developed, the Open XR platform and its related technologies are now being validated—with a new round of EU-funded projects. Project researchers hail from nonprofit educational institutions and are creating content that uses at least one of the existing tools. “MASTER is going to provide this large catalogue of educational assets, which future users can continue to extend,” Kildal said. “The idea is to sow the seeds for this open platform to grow and escalate in diversity so companies can realize the power of XR for training.” Karagiannis agrees and hopes that SMEs will soon be customizing the Open XR platform tools, with larger industry players following. “Our final client should be large industries,” he said. “Unlike many projects that struggle to reach industry, the Open XR platform can be fully exploited because it’s software sitting in the cloud, so companies can come, use it, and adapt it to their own commercial scenarios.” See also: EE Times Europe Magazine – March 2026 The March 2026 Edition of EE Times Europe Magazine analyzes how AI is transforming factory automation and operations and reviews Europe’s de-risking semiconductor strategy. Leave a Reply You must Register or Login to post a comment.