The Avatarm: Interacting in the Physical Metaverse via Robotics, Diminished Reality, and Haptics

The Metaverse is a three-dimensional digital space where users interact and communicate through their avatars, creating a sense of presence and immersion. The use of avatars allows to convey body language and to establish an object-based dialogue, improving the expressiveness of communication between users. However, existing metaverses are limited to digital interaction, as the avatar is not able to touch, perceive, grasp or manipulate the physical objects surrounding the remote interlocutor. In a previous work, we laid the foundation for the “Avatarm”, an avatar able to manipulate both the physical and the digital worlds. This was achieved through a robotic manipulator that remains hidden from the user’s view by means of diminished reality techniques. Building upon this groundwork, in this work we have advanced the capabilities of the Avatarm with the integration of force and vibrotactile haptic feedback, empowering the user to tangibly perceive manipulated objects, and gain a heightened sense of situational awareness. In addition, to increase the realism of the interaction, we have implemented a robot control algorithm capable of matching the position and orientation of physical objects with their digital counterparts. Furthermore, a digital avatar has been integrated into the augmented environment along with a newly designed virtual hand for more realistic manipulation of virtual objects. The enhanced version of the “Physical Metaverse” is tested in this work through an extensive experimental campaign considering both objective and subjective measures of performance.