Gesture BeatBox
Machine learning experiment, interaction design
Work with Anne-Sophie Charroin, Mellie Laroque, Marcelo Coelho - 2017

In the context of a thematic cycle “Arts, design & artificial intelligence” I took part in a machine learning workshop Machine learning workshop in Stereolux Arts&Tech Lab. The workshop was supervised by Martial Geoffre-Rouland.
The various experimentation and example help me to understand how machine learning broadly works.
Using creative coding tools such as Visual Studio and OpenFrameworks (a open library in C++),  we experiment a t-SNE (t-Distributed stochastic neighbor embedding) which sort images with similar content side by side and forming cluster of images.
Then in team, we prototype an idea that based on images recognition using machine learning. The concept was a visual Beatbox : make a gesture to trigger a sound.
We set 4 different gesture for 4 different sound.
Then we train a programme taking many photos of the four gestures with differents people in order to increase the success of image recognition. We use face OSC on OpenFrameworks.
Finally we get a simple result which approximately works. Only the second gesture was not well recognizable, probably because the hands are in front of the head.
next work