Customer Emotion Recognition Through Facial Expression

Customer Emotion Recognition

Through Facial Expression

by

Hoa T. Le

Bachelor of Information Technology

Thai Nguyen University of Information and

Communication Technology – Vietnam, 2012

A Thesis Proposal Submitted to the School of Graduate Studies

in Partial Fulfillment of the Requirements for the Degree of

Master of Science in Computer Science

Mapúa Institute of Technology

June 2016

ii

iii

ACKNOWLEDGEMENTS

The Author would like to express her sincere gratitude to God and to other significant

persons for giving the opportunity to complete this study;

To the greatest Adviser, Sir Larry A. Vea, for the continuous support of this Master

thesis study and related research, for his patience, motivation, and immense knowledge. His

guidance made this research in completion;

To the Thesis Committee, Dean Kelly Balan, Sir Joel De Goma, and Sir Aresh

Saharkhiz, for their time, insightful comments and encouragement, and for the hard questions

which incented the author to widen and improve her research from various perspectives;

To the School of Graduate Studies, Dr. Jonathan Salvacion, and Sir Omar Ombergado,

for their instruction to complete the format of this paper and other requirements needed;

To Ms. Grace Panahon – Star Circle manager and Ms. Rizza Faustino, for the help to

have the permission to gather data in the stores;

To the Editor, for the time spent in patiently checking the errors and reviewing this

manuscript;

To the Family, Parents, Brother and Sister-in-law, for the support that they provided

through the entire life of the author;

To the Friends and Housemates, especially Jocel Marie T. Gebora, for the support and

provision of food and prayers to have this thesis achieved in full completion.

Hoa T. Le

iv

TABLE OF CONTENTS

TITLE PAGE i

APPROVAL PAGE ii

ACKNOWLEDGEMENTS iii

TABLE OF CONTENTS iv

LIST OF TABLE vii

LIST OF FIGURES ix

ABSTRACT xi

Chapter 1: INTRODUCTION 1

Chapter 2: REVIEW OF RELATED LITREATURE 5

Emotion Typologies 7

Customer Emotion 9

Expression of Interest 9

Expression of Happiness 10

Expression of Sadness 11

Expression of Boredom 12

Expression of Surprise 12

Facial Affect Analysis 13

Microsoft Kinect SDK and Face Tracking Outputs 13

Kinect for Xbox 360 Face Tracking Outputs 16

Kinect v2 – High Definition Face Tracking 20

Comparison of Face Tracking Results Between Kinect v1 and Kinect v2 24

Piecewise Bezier Volume Deformation 24

v

Candide-3 25

Classifiers for Emotion Detection 25

Related Works 26

Chapter 3: CUSTOMER EMOTION RECOGNITION THROUGH FACIAL

EXPRESSION 29

Abstract 29

Introduction 29

Methodology 33

Research Paradigm 34

Methodology Parameters 35

Data Collection 36

Gathering Setup 36

Feature Extraction and Annotation 38

Feature Selection 41

Annotation 42

Training Classifiers 43

Model Testing 44

Prototype Development 45

Prototype Testing 45

Real World Testing 45

Analysis of the Results 45

Machine Learning and Classification 47

Results and Discussion 48

vi

Dataset Description 48

Animation Unit Interpretation 49

Annotation Results 52

Correlation Between the AUs. 54

Test Machine 55

Definition of Terms 55

Model Development 56

Model Testing 58

Model Performance of thirty-three (33) customers of Kinect 2. 59

Feature Selection 60

Classifier Analysis 62

Prototype Testing Result 66

Real World Testing 67

Conclusion 67

References 68

Chapter 4: CONCLUSIONS 76

Chapter 5: RECOMMENDATIONS 77

REFERENCES 78

vii

LIST OF TABLE

Table 1: Basic Emotions 8

Table 2: The Angles are expressed in Degrees 18

Table 3: Action Units [AUs] which represent “deltas “from the neutral shape of the face 19

Table 4: Shape Units [SUs] which determine head shape and neutral face 20

Table 5: Face Shape Animations Enumeration 22

Table 6: Kinect v1 and Kinect v2 Face Tracking Outputs 24

Table 7: Emotion Behaviors 42

Table 8: Instances in Kinect v1 and v2 dataset 49

Table 9: Animation Unit Interpretation (Microsoft) for Kinect 1 49

Table 10: Animation Unit Interpretation (Microsoft) for Kinect 2 50

Table 11: AUs detected from Sample Face by Kinect 1 51

Table 12: AUs detected from Sample Face by Kinect 2 52

Table 13: Features Observed by the Dataset 53

Table 14: Comparison of Magnitudes of “Happy", “Interest”, "Bored”, ”Surprise”" and “Sad”

in the Dataset. 53

Table 15: AUs Correlation 54

Table 16: Selected features using CfsSubsetEval and BestFrist 61

Table 17: Accuracy result by using CfsSubsetEval and BestFrist Kinect 2 61

Table 18: Accuracy result by using CfsSubsetEval and BestFrist Kinect 1 62

Table 19: Base Classifiers of the Random Committee 63

Table 20: Movements considered by the Classifier 63

Table 21: Movements considered by the Classifier 63

viii

Table 22: New Patterns Discovered of Customer’s Affect via the Notable Features 65

Table 23: Prototype Testing Results 66

ix

LIST OF FIGURES

Figure 1: Camera Space 14

Figure 2: Kinect-1-vs-Kinect-2-Tech-Comparison 16

Figure 3: Tracked Points 17

Figure 4: Head Pose Angles 18

Figure 5: Candide -3 face model 25

Figure 6: The Conceptual Framework 34

Figure 7: Research Paradigm 35

Figure 8: Star Circle, Starmall, Alabang 36

Figure 9: Camera Set-Up 37

Figure 10: Setup for Kinect Sensor Captures Full Body 38

Figure 11: Setup for Two (2) Kinect Sensors. 38

Figure 12: 3D Face Mask 40

Figure 13: Tracked Face 41

Figure 14: Annotation of Videos 43

Figure 15: Sample Face captured by Kinect 1 51

Figure 16: Sample Face Captured by Kinect 1 52

Figure 17: Accuracy of Model Development Results 57

Figure 18: Kappa of Model Development Results 57

Figure 19: Accuracy of Model Testing Results 58

Figure 20: Kappa Statistic of Model Testing Results 59

Figure 21: Accuracy of Model Testing 60

Figure 22: Kappa of Model Testing 60