Augmented reality (AR) provides immersive interactions by blending real and virtual worlds, making human-machine interactions more interesting and convenient than ever. A common application of AR involves placing a virtual object in a real environment, where the user is free to control or interact with the virtual object. However, there is so much more AR can do beyond that.

To make interactions easier and more immersive, many mobile app developers now allow users to control their devices without having to touch the screen by identifying the body motions, hand gestures, and facial expressions of users in real time and using the identified information to trigger different events in the app. For example, in an AR somatosensory game, players can trigger an action by striking a pose, which spares them from frequently tapping keys on the control console. Likewise, when shooting an image or short video, the user can apply special effects to the image or video by striking specific poses without even having to touch the screen. In a trainer-guided health and fitness app, the system powered by AR can identify the user’s real-time postures to determine whether they are doing the exercise correctly and guide them to exercise in the correct way. All of these would be impossible without AR.