Keep Smile

“Smile! Frown! Only use your facial expressions to overcome platforming challenges.”

Game Designer, Game Programmer | 4-Person Course Project for “Human Computer Interaction” | April 2021

In this project, we design and implement an experimental platformer, exploring the effect and the experience of playing 2D platform game only by facial expressions.

For the facial input, we choose two common facial expressions: smile and frown. Accordingly, we simplify the control scheme of traditional platformers, leaving only two dimensions of control: move forward and jump forward. Thus, we match the inputs and the controls. To be specific, when players make a happy face such as a smile, the character will run forward. Similarly, when players make an angry face such as a frown, the character will jump forward.

The arduous part of the implementation is to recognize player’s facial expression in real-time. We use Python to build a CNN network model. Then we train the model on JAFFE dataset as well as CK+ database. After several tests, we conclude that the model meets the needs of the real-time detection of expressions and the results are accurate. Later, we use a clever solution to solve the problem of data communication between Unity and Python. Eventually, the prototype is completed and we have a small-range playtest.