For more sharp and insightful business and economic news, subscribe to The Daily Upside newsletter. It’s completely free and we guarantee you’ll learn something new every day.
As AR and VR slowly but surely gain traction in gaming, tech companies want to know what your head is on: Sony and Meta have both filed patent applications for tech that tracks facial expressions while inside an artificial reality device.
Let’s start with Meta. The company is seeking a patent for facial expression tracking that, in some examples, does not require “cameras or complex image data processing.” Instead, Meta’s tech uses an illuminator and a photon detector.
As the name suggests, the headset’s lighting illuminates a user’s face, and the photon detector then picks up the ways in which that light is reflected off of it. The information from these two devices is then sent to a processor, which determines the appearance of a user’s face using an algorithm that can “reconstruct facial expressions.”
Along with avoiding the need for an inward-facing camera system in an AR or VR headset, Meta’s said its tech “provides a low-cost, low-computing overhead facial recognition system.”
Sony, meanwhile, is working on tracking users’ facial movements in an artificial reality game using barometric pressure sensors. Using headset-mounted sensors, this tech captures “pressure deviations” from the movement of a user’s facial features, including eye movements, breathing patterns, blinking or speaking. These variations are then analyzed by a processor to detect and evaluate “motion metrics,” or a user’s facial expressions, and are used to “fine-tune the user’s interactions.”
Sony said this type of tracking helps monitor user engagement metrics with AR and VR games, to “measure user interest or lack thereof” in the content being shown. Don’t let them catch you gaping.
“Measuring the interest level of the users to the content of the various applications will help the developers and content providers to provide content that is useful and/or interesting to the user,” Sony said in its submission.
As we’ve seen in several editions of Patent Drop, tech companies seem very interested in creating tech that can track users’ faces, whether it’s making video calls less awkward with simulated eye contact or making rendering easier for VR – headsets.
What Meta and Sony’s patent applications have in common, however, is that they use sensors instead of cameras to track your facial movement, potentially preserving privacy in a way that camera-based facial tracking methods don’t, Jake Maymar, VP of innovation at De Glimpse Group, told me. Shooting cameras also creates a better financial picture for the companies. Given the plethora of patents for camera-based methods of expression tracking that already exist, sensors provide a solution for these companies to get their hands on unique face-tracking tech, he said.
“It makes perfect sense. They’re just trying to figure out the solution to ‘How do I get facial recognition without using cameras?’ Maymar said. “Because cameras are patented and researched; they are expensive and also processor intensive. There are also privacy and security issues with that.”
Sony and Meta are hardly newbies in the augmented reality gaming space. Sony has long been an industry leader in console gaming with the PlayStation, and has its own headset offering with PS VR. Meta, of course, has its line of Quest VR and mixed-reality headsets to support its lofty metaverse goals, which supports a handful of games, including Among Us and Beat Saber.
As for the facial tracking itself, there are a few potential reasons that might interest Sony and Meta. In their applications, both companies achieve the unique engagement metrics that can be deciphered by watching the expression of a user in-game, as well as monitoring their emotional reactions to certain content. But another possible use, Maymar said, is how these expressions could enhance the VR experience.
Maymar gave the example of an in-game VR quest with friends: “Let’s say you’re in this scary corridor, and you can see the faces of your friends and they can see your face. You can see these expressions – that excitement – not only through gestures, but their actual face. It feels like a bonding moment, because it feels like you’re there.”
At this point, augmented reality experiences represent a small fraction of the gaming industry, Maymar said. Part of the reason is because it can be a lonely experience, devoid of connection or emotion. But adding in a way to convey emotions in gameplay, such as through facial expressions, could be a key to growing user interest.
But until that happens, Maymar said, “XR probably won’t really have a big impact on gaming.”
Do you have any comments, tips or suggestions? Drop us a line! E-mail at (email protected) or shoot us a DM on Twitter @patentdrop. If you would like to receive Patent Drop in your inbox, click here to subscribe.
0 Comments