Development Of Real Time Systems To Detect And Track On Duty Injured Firefighters Using Advanced Signal Processing Techniques

VIETNAM NATIONAL UNIVERSITY HA NOI

UNIVERSITY OF ENGINEERING AND TECHNOLOGY

PHAM VAN THANH

DEVELOPMENT OF REAL-TIME SYSTEMS TO DETECT AND

TRACK ON-DUTY INJURED FIREFIGHTERS USING ADVANCED

SIGNAL PROCESSING TECHNIQUES

DOCTORAL THESIS IN ELECCTRONICS AND

TELECOMMUNICATIONS ENGINEERING

Ha Noi - 2022

VIETNAM NATIONAL UNIVERSITY HA NOI

UNIVERSITY OF ENGINEERING AND TECHNOLOGY

PHAM VAN THANH

DEVELOPMENT OF REAL-TIME SYSTEMS TO DETECT AND

TRACK ON-DUTY INJURED FIREFIGHTERS USING ADVANCED

SIGNAL PROCESSING TECHNIQUES

Major: Electronic Engineering

Code: 9 52 02 03.01

DOCTORAL THESIS IN ELECCTRONICS AND

TELECOMMUNICATIONS ENGINEERING

Supervised by: Assoc. Prof. Tran Duc Tan

Ha Noi - 2022

1

DECLARATION

“I hereby declare that the work contained in this thesis is of my own, and it has

been written by myself under the supervison of Professor Tran Duc Tan at Faculty of

Electrical and Electronic Engineering, Phenikaa University, Hanoi, Vietnam during

the period from September 2017 to August 2021.

The thesis has not been previously submitted for a degree or diploma at this or

any other higher education institution.

I have duly acknowledged all the sources of information which have been used

in the thesis.

The thesis content has been partly published in my list of publications as below:

1. Pham Van Thanh, Tuan Khai Nguyen, Duc Anh Nguyen, Nhu Dinh Dang, Huu

Tue Huynh, Duc-Tan Tran*, “Adaptive Step Length Estimation Support Indoor

Positioning System using Low-Cost Inertial Measurement Units”, 2020 IEEE Eighth

International Conference on Communications and Electronics, pp.271-275, 13-15

Jan.2021.

2. Pham Van Thanh, Le Quang Bon, Nguyen Duc Anh, Dang Nhu Dinh, Huynh

Huu Tue, Tran Duc Tan, "Multi-Sensor Data Fusion in A Real-Time Support System

for On-Duty Firefighters", Sensors 2019 (ISSN: 1424-8220 – SCIE).

3. Van Thanh Pham, Duc Anh Nguyen, Nhu Dinh Dang, Hong Hai Pham, Van

An Tran, Kumbesan Sandrasegaran and Duc-Tan Tran, “Highly Accurate Step

Counting at Various Walking Speeds Using Low-Cost Inertial Measurement Unit

Support Indoor Positioning System”, Sensors. 2018; 18(10):3186. (ISSN: 1424-8220

– SCIE).

4. Pham Van Thanh, Duc-Tan Tran, Dinh-Chinh Nguyen, Nguyen Duc Anh,

Dang Nhu Dinh, S. El-Rabaie and Kumbesan Sandrasegaran, “Development of a

Real-time, Simple and High-Accurate Fall Detection System for Elderly Using 3-

2

DOF Accelerometers”, Arabian Journal for Science and Engineering. 2018 (ISSN:

2191-4281 – SCIE).

5. Pham Van Thanh, Anh-Dao Nguyen Thi, Quynh Tran Thi Thuy, Dung Chu

Thi Phuong, Viet Ho Mau and Duc-Tan Tran, “A Novel Step Counter Supporting For

Indoor Positioning Based On Inertial Measurement Unit”, 7th international

conference on Integrated Circuit, Design, and Verification (ICDV), IEEE, pp. 69-74,

5-6 Oct. 2017.

6. Nguyen Van Duong, Pham Van Thanh, Tran Van An, Nguyen Tuan Khai,

Duong Thi Thuy Hang, Hoang The Hop and Tran Duc Tan, “Elevator Motion States

Recognition Using Barometer Support Indoor Positioning System”, The 7th

International Conference in Vietnam on the Development of Biomedical

Engineering, IFMBE Proceedings, Springer, pp.581-587, 27-29 Jun.2018.

7. The Hop Hoang, Van Thanh Pham, Thuy Quynh Tran Thi, Huu An Nguyen,

Tuan Khai Nguyen and Tan Tran-Duc, “Xây dựng hệ thống xác định độ cao bên

trong nhà và công trình sử dụng đa cảm biến áp suất”, Hội nghị Quốc gia lần thứ

XXI về Điện tử, Truyền thông và Công nghệ Thông tin (The 21st National

Conference on Electronics, Communications and Information Technology), 2018, pp.

193-197.

Ha Noi, 16

th January, 2022

Author

Signature:……………………………………………….

3

ACKNOWLEDGEMENT

I would like to express my sincere thanks to my advisor Assoc. Prof.

Tran Duc Tan, Faculty of Electrical and Electronic Engineering, Phenikaa University

for the guidance and support throughout the completion of my thesis.

My thanks go to all lecturers and people in Faculaty of Electronics and

Telecommunications, Universisty of Engineering and Technology, Viet Nam

National University Hanoi for their teaching and useful help.

I give my special thanks to leaders, colleagues and Mr. Tran Van An at

University of Fire Prevention and Fighting for their help, guidance and financial

support in the entire thesis completion process.

I am grateful to all people in MEMS lab as well as my students in Universisty

of Engineering and Technology, Vietnam National University Hanoi and University

of Fire Prevention and Fighting for their contribution.

I would also like to greatly thank the Vingroup Innovation Foundation

(VINIF) - Vingroup Big Data Institute (VinBigdata) for their grant support and

encouragement, which help me overcome financial problems and difficulties.

Last but the most important, I would like to thank my parents, my brother,

my sister-in-law because their comfort and support are the power for me going to

success.

“This work was supported by the Domestic Master/ PhD Scholarship

Programme of Vingroup Innovation Foundation”.

Ha Noi, 16th January, 2022

4

CONTENTS

CONTENTS............................................................................................................4

LIST OF ABBREVIATIONS.................................................................................7

LIST OF FIGURES ..............................................................................................12

INTRODUCTION......................................................................................... 18

1. Defining Problem............................................................................................18

2. The purpose of thesis ......................................................................................19

3. Objectives and Scope of the Thesis................................................................21

4. Research Methodology ...................................................................................22

5. Scientific significance and Contributions of the Thesis .................................22

6. Thesis structure ...............................................................................................23

CHAPTER 1. OVERVIEW OF THE RESEARCH.................................. 25

1.1. Literature review............................................................................................25

1.2. Related Studies on Injured Detection ............................................................26

1.3. Related Studies on Indoor Positioning ..........................................................29

1.4. The challenges in study on injury detection and indoor positioning.............29

1.5. Summary........................................................................................................30

CHAPTER 2. SYSTEM DESCRIPTION, SENSOR ERRORS

ELIMINATION AND MAP PROCESSING.............................................. 31

2.1. System Description........................................................................................31

2.2. Sensors Errors Elimination ............................................................................33

The 3-DOF Accelerometer................................................................33

The Magnetic Sensor.........................................................................35

The Barometer...................................................................................36

The MQ7 Sensor ...............................................................................38

2.3. Map Processing..............................................................................................38

Map Preprocessing ............................................................................38

Map Simplification............................................................................39

5

Map Scale ..........................................................................................43

2.4. Summary........................................................................................................45

CHAPTER 3. DEVELOPMENT OF A METHOD TO DETECT

INJURED FIREFIGHTERS........................................................................ 46

3.1. Fall Detection Method ...................................................................................46

Fall Detection Module.......................................................................46

Post-fall Recognition Module ...........................................................48

The Posture Recognition Estimation.................................................49

The Vertical Velocity Estimation......................................................49

3.2. Injury Detection for On-Duty Firefighters ....................................................52

The Proposed Fall Detection Algorithm for Firefighter....................53

The Proposed Loss of Physical Performance Detection Algorithm for

Firefighter.........................................................................................................57

The CO Detection Algorithm............................................................60

3.3. Result and Discussion....................................................................................61

The Experimental Results .................................................................61

Fall Detection Results .......................................................................63

Loss of Physical Performance Detection ..........................................64

The High CO Level Alerting Algorithm...........................................66

3.4. The Comparison.............................................................................................67

The Comparison on the Experimental Data ......................................68

The Comparison on Public Datasets .................................................71

3.5. Summary........................................................................................................77

CHAPTER 4. DEVELOPMENT OF A METHOD TO TRACK ON￾DUTY INJURED FIREFIGHTERS ........................................................... 79

4.1. The Step Counting Method............................................................................80

The Results........................................................................................95

Discussion .........................................................................................99

4.2. Step Length Estimation................................................................................107

6

The Proposed Method .....................................................................107

Results and Discussion....................................................................110

4.3. Turning Time and Direction Estimation......................................................113

Turning Time Estimation ................................................................113

Turning Direction Estimation..........................................................116

4.4. Vertical Position Estimation ........................................................................121

4.5. Summary......................................................................................................124

CHAPTER 5. INDOOR FIREFIGHTER POSITIONING AND

TRACKING USING MULTI-SENSOR DATA FUSION AND MAP

MATCHING ALGORITHM..................................................................... 125

5.1. Data Fusion ..................................................................................................125

5.2. Combining Data Fusion and Map Matching to Detect Indoor Position ......126

Experiment Setup ............................................................................126

The Scenarios Testing .....................................................................126

5.3. Summary......................................................................................................135

CONCLUSIONS AND FUTURE WORK................................................ 136

LIST OF PUBLICATIONS........................................................................ 138

THE RELATED PUBLICATIONS .......................................................... 140

REFERENCES............................................................................................ 141

7

LIST OF ABBREVIATIONS

3-DOF Three Degrees Of Freedom

ADLs Activities of Daily Living

CO Carbon Monoxide

CO2 Carbon Dioxide

COHb Blood Carboxyhemoglobin

FFT Fast Fourier Transform

GPS Global Positioning System

iOS iPhone Operating System (Apple)

I

2C Inter-Integrated Circuit

IC Incident Commander

IMU Inertial Measurement Unit

MEMS Micro-Electro-Mechanical systems

NFPA National Fire Protection Association

OADs On-Duty Activities

PAA Piecewise Aggregate Approximation

PASS Personal Alert Safety System

ppm Parts Per Millions

RMS Root Mean Square

ROC

curve

receiver operating characteristic curve

SAX Symbolic Aggregate approximation

SCBA Self-Contained Breathing Apparatuses

SpO2 Saturation of Peripheral Oxygen

SVM Support Vector Machine

UFPF University of Fire Prevention and Fighting

US United States

8

US

OSHA

CFR

U.S. Occupational Safety and Health Administration Code of Federal

Regulations

������ Accuracy

���� False Negative

���� False Positive

������ Sensitivity

�������� Specificity

���� True Negative

���� True Positive

9

LIST OF SYMBOLS AND THEIR MEANINGS

SYMBOLS MEANINGS

Ax, Ay, Az Acceleration Along Ax, Ay, And Az Axes

G

The Acceleration Of Gravity On The Surface Of The

Earth At Sea Level

K

��

+ and ��

−

The Accelerations Measure In Positive (��

+) And

Negative (��

−) Of Ax, Ay And Az Axes

��

′+ and ��

′−

The Accelerations Measure In Positive (��

+) And

Negative (��

−) Of Ax, Ay And Az Axes After Multiple

With K Factor.

��

+ and ��

− Acceleration In Each Axes After Calibrated

A����(��)

The Root Mean Square (Rms) Of Acceleration Along

Ax, Ay, And Az Axes At Time ��

���� The Kalman Gain

��̂�� The Estimated Signal On The Current State

��̂��

− The Estimated Signal On The Previous State

���� The Posteriori Error Covariance

����

− The Priori Error Covariance

���� The Measured Value

R The Environment Noise

���� The Calculated Altitude In Meters

���� The Measured Pressure

P0 The Pressure At The Sea Level (P0 = 1013.25 Hpa).

�� ⊕ �� The Dilation Of A By The Structuring Element B

�� ⊖ �� The Erosion Of A By The Structuring Element B

TFE

The Duration Time From Acc Values Exceed The LFT

Threshold Until It Exceeds The UFT Threshold.

10

LFT Lower Fall Threshold

UFT Upper Fall Threshold

�� The Angle Between Ay And Gravity

�� Vertical Velocity

����ℎ������ℎ������

The Threshold To Distinguish Between The Rest And

The Active States

����ℎ The Upper Threshold

U���� The Upper Threshold to check the post-fall condition

L���� The Lower Threshold to check the post-fall condition

T The Theta Angle

��, ��, �� The Yaw, Pitch, And Roll Angles

����������

The Upper Threshold To Check Loss Of Physical

Performance Condition

����_������

The Lower Threshold To Check Loss Of Physical

Performance Condition

������������������ The Altitude Variations

∆�� The Altitude Change

D The Minimal Number Of Samples

������_�������� The Window Size

���� The Average Time Period To Perform Each Step

AverD The Average Dynamic Thresholding

��ℎ�� Fluctuated Value

������������

The Difference Between ������(��) And The Gravity

Acceleration

���� The Time Range Between Two Neighboring Peak

������������+1

And ������������

The Time Of Peak(I+1) And Peak(I)

���� The Similarity Between Two Neighboring Peaks

Kg Kilogram

11

�� The Number Of The Estimated Steps

�� The Number Of Reference Steps

12

LIST OF FIGURES

Figure 0.1. US firefighter injuries by type of duty during 2015 [45]. ......................18

Figure 2.1. The block diagram of the proposed system. ...........................................32

Figure 2.2. The recorded data with and without using the simple Kalman filter ....35

Figure 2.3. The Magnetic fields along Ax, Ay and Az before and after using the

simple Kalman filter..................................................................................................35

Figure 2.4. The signal from the magnetic sensor before and after calibration .........36

Figure 2.5. The altitude signal comparison between with and without using the simple

Kalman filter .............................................................................................................37

Figure 2.6. The position of CO sensor on the mask..................................................38

Figure 2.7. The binary image of a floor used in experimental testing ......................39

Figure 2.8. The map simplification algorithm ..........................................................40

Figure 2.9. The Erosion and Dilation operations [17] ..............................................41

Figure 2.10. The result of applying Erosion and Dilation with structuring element size

of (7 × 7) for keeping wall. .......................................................................................42

Figure 2.11. The result of applying Erosion and Dilation with structuring element size

of (4 × 4) for keeping windows and stairs. ...............................................................42

Figure 2.12. The Map simplification achieved after using Erosion and Dilation

operations..................................................................................................................43

Figure 2.13. The floor size detected after applying the flood fill algorithm.............44

Figure 2.14. The unclosed floor structure .................................................................45

Figure 2.15. The floor size detected after applying the dilation operation and flood

fill algorithm..............................................................................................................45

Figure 3.1. The proposed fall detection algorithm....................................................47

Figure 3.2. An example of a fall event and the UFT, LFT and tFE thresholds........48

Figure 3.3. The time cycle of 6 walking steps..........................................................49

Figure 3.4. The different states of the user along three axes Ax, Ay and Az ...........50

Figure 3.5. The values of acceleration, Velocity and Theta angle of a volunteer when

transiting between activities: standing – walking – sitting – walking – lying..........51

Figure 3.6. The injury detection algorithm for firefighters.......................................52

Figure 3.7. The proposed fall detection algorithm....................................................56

Figure 3.8. Firefighters move through the narrow paths or spaces...........................58

Figure 3.9. The proposed loss of physical performance detection algorithm...........59

13

Figure 3.10. The high CO level alerting algorithm...................................................60

Figure 3.11. The volunteer is carrying the support device in his trouser pocket in the

crawling state viewed from a side (a) and from above (b)........................................62

Figure 3.12. (a) The RMS of acceleration of a fall forward from standing, first impact

on knees; (b) the theta angle; (c) the pitch and roll angles. ......................................64

Figure 3.13. The loss of physical performance because of the accident (crawling then

falling); (a) the RMS of accelerometer data; (b) the barometric data.......................65

Figure 3.14. The loss of physical performance because of moving up in an elevator;

(a) the RMS of accelerometer data; (b) the barometric data.....................................66

Figure 3.15. a) Testing and measuring the CO level in the fire; b) the measured CO

values.........................................................................................................................67

Figure 3.16. The RMS of acceleration of a fall forward from standing. ..................69

Figure 3.17. The RMS of acceleration of crawling then falling as the scenario of

Figure 3.11a...............................................................................................................69

Figure 3.18. The RMS of acceleration of crawling then falling as the scenario in

Figure 3.11b. .............................................................................................................70

Figure 3.19. The pressure signal of the public dataset [75]; (a) the raw pressure data

and estimated pressure data after being filtered by the simple Kalman filter and

complementary filter, and (b) the zoom in raw pressure data and estimated pressure

data after being filtered by the simple Kalman filter and complementary filter.......75

Figure 3.20. The altitude variations of the fall events based on the different mounting

positions [75].............................................................................................................76

Figure 4.1. The flowchart depicting our step counting method................................83

Figure 4.2. The signal before and after applying the proposed low-pass filter: (a)

Before applying the low-pass filter; and (b) After applying the low-pass filter.......85

Figure 4.3. The acceleration magnitude of three different kinds of walking............85

Figure 4.4. The d value on the peak detection in our normal walking data: (a) d = 5; (b)

d = 14; (c) d = 20. ......................................................................................................87

Figure 4.5. The minimal peak prominence definition...............................................88

Figure 4.6. The true peaks and the false peaks of a data sample. .............................90

Figure 4.7. The minimal peak prominence analysis. ................................................90

Figure 4.8. The dynamic thresholding and the zoom in dynamic thresholding for our

recorded data: (a) The dynamic thresholding for our recorded data; and (b) The zoom

in of dynamic thresholding for our recorded data.....................................................93