Using face tracking AI to drive Unreal Engine cameras
- August 6, 2024
So much great tech is now available for artists and technologists to assemble in novel ways. We all have the means to invent the grammar of tomorrow’s language. In this case, can we make the interaction with virtual spaces more natural and intuitive? The clip below shows a prototype that uses an AI face tracker to move a virtual camera in Unreal Engine. The Unreal camera moves left and right when I move my head left and right, and I can rotate the camera by turning my head back and forth. Moving forward and backward is achieved by moving my head towards or away from the camera.
This setup uses the OAK-D AI assisted computer vision camera. The camera supports Google’s Mediapipe library of edge AI models and has a robust python API which is used to capture, calibrate and stream the data to Unreal Engine in real time. Using the OSC protocol which makes the data available in blueprints via a plugin, the information is applied to the camera transforms and to a UI overlay that provides the necessary visual feedback. While not necessarily designed for permanent installations, this setup makes a very efficient toolset for rapid prototyping of ideas, particularly since Mediapipe includes many other models such as full body pose estimation or hand gesture recognition amongst others.
Featured -----
Relentless Play VFX Delivers The Little Render of Bethlehem
2023-04-19 -- In the delivery of over 30 VFX shots, Thomas Hollier and Relentless Play LLC created digital environments for the cities of Bethlehem and Jerusalem in ancient times, and handled all aspects of VFX on the project from visual development and previsualization to on-set supervision and final shot production.
NeRFs in Unreal Engine
2023-04-07 -- NeRFs in Unreal Engine 5 are finally here. Check out [Read More...]Archive -----
Bringing VFX environments to an LED volume stage
2024-03-04 -- Relentless Play put together a presentation for Turning Point Productions [Read More...]
Video Datagrams
2021-10-19 -- Compositions featuring stacks of "chronographs" generated from digital video data. [Read More...]
Synced Video Tryptic
2020-11-29 -- Filmmaker Natasha Maidoff approached me needing a way to display [Read More...]
Electronic Time Piece with Cement and Light
2020-11-28 -- Building a binary clock is a rite of passage for [Read More...]