A hand tracking and gesture classifier for your 3D Application. Hand tracking and gesture classifier will run in Python, the data will be sent through an UDP connection into Unity locally. The model is trained using Dense Neural Network, optimized and quantized using Tensorflow.
It only needs a single RGB camera to operate as a hand tracking and can also simulate z-space (forward and backward) movement on the 3D application. The dataset is collected by recording each gesture individually from the Python application and saved into a .csv file.
The application in Unity uses the Interactor and Interactables concept to interact with the object. The interaction experience is augmented using the hand gesture classifier input from the Hand Gesture Capturer, enabling it to elevate more complex interactions to be made inside the application.
See it on action here!
- PlatformUnity
- StackPython, C#, Tensorflow
