Face Music Control

Кликните, чтобы прочитать этот документ на русском

Face Music Control is a python program for controlling the sound of a musical instrument via emotion recognition. It uses a virtual MIDI port to send MIDI CC Messages of amplitudes proportional to the probabilities of emotions recognised by a convolutional neural network based on facial expression. To control DAW parameters, we usually move sliders of a MIDI controller. Face Music Control's approach is similar, except the MIDI controller is virtual, operated by a neural network.

Motivation

When improvising and creating new music, the existing interfaces for controlling the sound (pedals, sliders, etc.) are inconvenient and rarely applicable. The fact is that the musician, when creating music, lacks concentration to look for the right tone and control the sound of the instrument. At the same time, in these contexts it is particularly important that the character of the sound is appropriate to the musician's emotional state. I have not found any means for controlling the sound of an instrument based on the musician's emotions. Therefore, I wrote this program.

Additional applications

A new kind of interaction with an instrument can be a self-contained performance
Since the lighting equipment is controlled by MIDI events, Face Music Control can be used to complement the visuals of concerts

Requirements

Obvious requirements are

A camera on your computer
A python interpretor
A DAW

Libraries

Install all the required libraries with this command:

pip install -r requirements.txt

Virtual MIDI port driver (Windows only)

Windows requires a driver installation to create virtual MIDI ports. Possible solutions:

LoopBe1
Once LoopBe1 is installed, a virtual MIDI port will be on your computer until the driver is uninstalled.
loopMIDI
It is a program that allows you to manipulate virtualMIDI driver that comes with loopMIDI. If you chose loopMIDI over LoopBe1 you have to run loop MIDI to have a virtual MIDI port on your computer.

If you use linux or macOS, Face Music Control creates a virtual MIDI port by itself, and three is no need for third-party drivers.

Launching

Run main.py. The program was tested with python 3.10.4

Setting mode

To bind a sound parameter and an emotion, you need to:

Open the setting mode in Face Music Control and MIDI mapping mode in a DAW
Successively press the DAW graphic interface element responsible for the sound parameter and the button with a graphic representation of the emotion, which should control the sound parameter. When all the required sound parameters are mapped to a corresponding emotion, you should exit MIDI mapping mode, and then exit Face Music Control setting mode.

Play mode

The sound is controlled in play mode.

In the video demonstration below, happiness controls the echo, and anger controls the distortion.

happiness-echo_anger-distortion.mp4

Sound on the video in the README is off by default, but it can be turned on.

*All demonstrations are in the project directory: READMEmaterials/demonstrations. If the video does not display, you can find it there (happiness-echo_anger-distortion.mp4).