Development of machine vision systems on object classification and measurement for robot manipulation

i

國立高雄科技大學

機械工程系 博士班

博士論文

操縱機器人所需物件分類和量測機械視覺系統之開發

Development of machine vision systems on object

classification and measurement for robot manipulation

研究生：吳玉武 Graduate student: Ngo Ngoc Vu

指導教授：許光城 教授 Advisor: Prof. Quang-Cherng Hsu

中華民國 108 年 1 月

ii

操縱機器人所需物件分類和量測機械視覺系統之開發

Development of machine vision systems on object classification and

measurement for robot manipulation

研究生：吳玉武 Graduate student: Ngo Ngoc Vu

指導教授：許光城 教授 Advisor: Prof. Quang-Cherng Hsu

國立高雄科技大學

機械工程系

博士論文

A Dissertation

Submitted to

Department of Mechanical Engineering

National Kaohsiung University of Science and Technology

in Partial Fulfillment of the Requirements

for the Degree of Doctor of Philosophy

in Mechanical Engineering

January, 2019

Kaohsiung, Taiwan, Republic of China

中華民國 108 年 1 月

iii

i

操縱機器人所需物件分類和量測機械視覺系統之開發

研究生：吳玉武 指導教授：許光城 教授

國立高雄科技大學

機械工程系 博士班

中文摘要

本研究介紹了機器視覺系統在物體分類和機器人操縱測量方面的研究。首先，

在不同的照明條件下開發了一套可用於金屬零件的自動分類和量測的機械視覺系

統，並將其應用於具 6 個自由度（DOF）的機器手臂。為了獲得準確的定位資訊，

整個圖像藉由工作平台上方的 CMOS 攝影機來進行取像。本研究中亦探討不同照明

條件下對本系統的影響，在正向打光的條件中有四種不同的打光狀態，且在每種條

件，皆使用全域和局部閾值來獲得較佳的圖像品質。研究中主要利用張氏方法、線

性轉換函式與二階轉換函式來取得影像座標與世界座標的關係。

實驗結果顯示，在背向打光相較於前向打光其所獲得的物體中心更為準確，而

在校正結果的部分，二階轉換函式比其他校正方法更準確。透過使用二階轉換函式

的校正偏差，在 X 和 Y 軸其最大正偏差分別為 0.48 mm 和 0.38 mm 而最大負偏差分

別為-0.34 mm 和-0.43 mm。

本研究亦開發了一套機械視覺系統於六自由度(DOF)的機械手臂用以進行物件

顏色的分類和座標的量測。整個圖像藉由工作平台上方的兩個相機以藉此獲得準確

的定位資訊，並在二維(2-D)和三維(3-D)的校正過程中採用二階轉換函式與透視投影

法來建立影像坐標和世界坐標間的關係。在二維校正的部分，於 X 和 Y 軸上的最大

正偏差分別為 1.29 mm 和 1.12 mm；而最大負偏差分別為-1.48 mm 和-0.97 mm。在

三維校正的部分，X，Y 和 Z 方向的偏差分別為 0.07、-0.418 和-0.063 mm。本研究

所提出的視覺識別系統可以對物件進行分類並取得該物件之三維坐標。

關鍵字：機械視覺、機械手、相機校正、影像分析、物件分類、光源

ii

Development of machine vision systems on object classification and

measurement for robot manipulation

Graduate student: Ngo Ngoc Vu Advisor: Prof. Quang-Cherng Hsu

Department of Mechanical Engineering

National Kaohsiung University of Science and Technology

ABSTRACT

This research presents development of machine vision systems on object

classification and measurement for robot manipulation. Firstly, a machine vision

system for the automatic metal part classification and measurement process is

developed under different lighting conditions, and has been applied to the operation

of a robot arm with 6 degrees of freedom (DOF). In order to obtain accurate

positioning information, the overall image is captured by a CMOS camera which is

mounted above the working platform. The effects of back-lighting and front￾lighting conditions to the proposed system were investigated. With the front￾lighting condition, four different conditions were performed. For each condition,

global and local threshold operations were used to obtain good image quality. The

relationship between the image coordinates and the world coordinates was

determined through Zhang’s method, the linear transformation and the quadratic

transformation during the calibration process. Experimental results show that in a

back-lighting environment, the image quality is improved, such that the positions of

the centers of objects are more accurate than in a front-lighting environment.

According to the calibration results, the quadratic transformation is more accurate

iii

than other methods. By calculating the calibration deviation using the quadratic

transformation, the maximum positive deviation is 0.48 mm and 0.38 mm in the X

and Y directions, respectively. The maximum negative deviation is -0.34 mm and

-0.43 mm in X and Y directions, respectively. The proposed system is effective,

robust, and can be valuable to industry.

The second, a machine vision system for color object classification and

measurement process for robot arm with six degree of freedom (DOF) is developed.

In order to obtain accurate positioning information, the overall image is captured by

a double camera C615 and a camera C525 which are mounted above the working

platform. The relationship between the image coordinate and the world coordinate

is performed through calibration procedure. The quadratic transformation and

generalized perspective transformation algorithms were used to transform

coordinates in 2-D and 3-D calibration process, respectively. According to

calibration results, with 2-D calibration, the positive maximum deviation is 1.29

mm and 1.12 mm in X and Y directions, respectively. The negative maximum

deviation is -1.48 mm and -0.97 mm in X and Y directions, respectively. With 3-D

calibration, the deviation is 0.07 mm, -0.418 mm, and -0.063 mm in X, Y and Z

directions, respectively. The proposed system can catch the three dimensional

coordinates of the object and perform classification and assembly automatic

operations by the data from visual recognition system.

Keywords: machine vision, robot arm, camera calibration, image analysis, object

recognition, lighting source.

iv

ACKNOWLEDGMENTS

The fulfillment of over three years of study at National Kaohsiung University of

Science and Technology (NKUST) has brought me into closer relations with many

enthusiastic people who wholeheartedly devoted their time, energy, and support to

help me during my studies. Therefore, this is my opportunity to acknowledge my

great debt of thanks to them.

I wish to express my thanks and gratitude to my academic supervisor,

Prof. Dr. Quang-Cherng Hsu, for his continuous guidance, valuable advice, and

helpful supports during my studies. He has always been supportive of my research

work and gave me the freedom to fully explore the different research areas related

with my study.

I wish to acknowledge my deepest thanks to Vietnam Ministry of Education and

Taiwan Ministry of Education for giving me a great opportunity, necessary

scholarships to study at NKUST via VEST500 scholarship which is corporation

between Vietnam Government and Taiwan Government, and many enthusiastic

helps during my time in NKUST. I am also particularly grateful to Thai Nguyen

University of Technology (TNUT) provided me unflagging encouragement,

continuous helps and support to complete this course.

My gratitude also goes to all of the teachers, Dean and staffs of Department of

Mechanical Engineering at NKUST for their devoted teaching, great helping and

thoughtful serving during my study.

v

I would also like to express my sincere gratitude to all of my colleagues at the

Precision and Nano Engineering Laboratory (PANEL), Department of Mechanical

Engineering, NKUST.

I want to express my sincere thanks to all my Vietnamese friends in NKUST for

their helpful sharing and precious helping me over the past time.

I also wish to express my gratitude to all those who directly or indirectly

helped me during my study in NKUST.

Finally, my special thanks to my dad Ngo The Long and my mom Vu Thi Hai,

to my older sister Ngo Thi Phuong, to my adorable wife Duong Thi Huong Lien, to

two lovely little daughters Ngo Duong Anh Thu and Ngo Phuong Linh, who are the

most motivation for me over years in Taiwan!

vi

CONTENTS

中文摘要…………………………………………………………………………….i

ABSTRACT………………………………………………………………………...ii

ACKNOWLEDGMENTS………………………………………………………...iv

CONTENTS………………………………………………………………………..vi

LIST OF FIGURES……………………………………………………………....xii

LIST OF TABLES…………………………………………………………….....xvi

NOMENCLATURE……………………………………………………………..xvii

Chapter 1 Introduction…………………………………………………………….1

1.1 Motivation of the research……………………………………………………1

1.2 Scopes of the research………………………………………………………..8

1.3 Contributions…………………………………………………………………9

1.4 Organization of the dissertation……………………………………………...9

Chapter 2 Theory of image processing and machine vision……………………12

2.1 Image processing system……………………………………………………12

2.1.1 Basics in image processing……………………………………………..13

2.1.1.1 Pixels……………………………………………………………..13

2.1.1.2 Resolution of image……………………………………………...13

vii

2.1.1.3 Gray level………………………………………………………...14

2.1.1.4 Histogram………………………………………………………...15

2.1.1.5 Image presentation……………………………………………….16

2.1.1.6 Color models……………………………………………………..17

2.1.1.7 Neighbors of pixels………………………………………………18

2.1.2 Morphological image processing ……………………………………...19

2.1.2.1 Erosion operation………………………………………………...19

2.1.2.2 Dilation operation………………………………………………..19

2.1.2.3 Opening and closing operations …………………………………20

2.1.3 Blob analysis…………………………………………………………...21

2.1.3.1 Goal of blob analysis.……………………………………………21

2.1.3.2 Feature extraction .………………………………………………21

2.1.3.3 Steps to perform blob analysis. …………………………………22

2.2 Machine vision system..…………………………………………………….22

2.2.1 Lighting design ....………………………………………………….......23

2.2.2 Lens....…………………………………………………………….. …..25

2.2.3 Image sensors....…………………………………………………... …..27

Chapter 3 Coordinate calibration methods and camera calibration………….30

3.1 Two-Dimensional coordinate calibration…………………………………...30

viii

3.1.1 Linear transformation ...………………………………………………..30

3.1.2 Quadratic transformation……………………………………………….32

3.2. Three-Dimensional coordinate calibration.. . ………………………………36

3.2.1 Stereo imaging ....…………………………………………………........36

3.2.2 The generalized perspective transformation.... …………………….......37

3.2.3 The perspective transformation with camera model..... ………………..39

3.3 Camera calibration…………………………………………………………..44

3.3.1 Intrinsic parameter......…………………………………………………44

3.3.2 Extrinsic parameter....…………………………………………….........45

3.4 Lens Distortions......………………………………………………………...46

3.4.1 Radial distortion........ ………………………………………………….46

3.4.2 Tangential distortion........ …………………………………………… 47

3.5 Camera calibration using Matlab tool……………………………………….48

Chapter 4 Development of a metal part classification and measurement system

under different lighting conditions………………………………………………51

4.1 Materials and Experimental Setup…………………………………………..51

4.1.1 Platform Description........……………………………………………...51

4.1.2 Robot arm ........………………………………………………………...53

4.2 Research methodology………………………………………………………54

ix

4.2.1 Image processing……………………………………………………….54

4.2.2 Camera calibration……………………………………………………..55

4.2.3 Coordinate calibration method…………………………………………57

4.2.4 Resolution of the Measurement System…….………………………….58

4.2.5 Calculating the Coordinate Deviation ........ …………………………....58

4.3 Experimental Procedures……………………………………………………59

4.3.1 Illumination conditions........…………………………………………...59

4.3.2 Calibration work ........………………………………………………….59

4.4 Image Segmentation…………………………………………………….......61

4.4.1 Using backlighting........………………………………………………..61

4.4.2 Using front-lighting ........……………………………………………....63

4.5 Algorithm for object classification…………………………………….........65

4.6 Algorithm for the bolt head determination ..... ..…………………………....66

4.7 Results and Discussion......………………………………………………….68

4.7.1 Calibration results……........…………………………………………...68

4.7.2 Coordinate deviation among different lighting conditions …………….70

4.8 Implementation ……… .....…………………………………………………73

Chapter 5 Development of a color object recognition and measurement system…..77

5.1 Platform Description………………………………………………………...77

x

5.1.1 Experimental setup … ........…………………………………………....77

5.1.2 The implementation process of the proposed system…………………..79

5.2 Calibration work…………………………………………………………….80

5.2.1 2-D coordinate calibration using the quadratic transformation .............. 80

5.2.2 3-D calibration coordinate using the perspective transformation........... 81

5.3 Image analysis and object segmentation…………………………………….83

5.4 Algorithm for color object classification……………………………………86

5.4.1 Algorithm for solid classification………………………………………86

5.4.2 Algorithm for hole classification……………………………………….87

5.5 Determination of orientation for triangular holes…………………………...90

5.6 Results and discussion………………………………………………………93

5.6.1 2-D calibration and spatial position measurement results……………...93

5.6.2 3-D calibration and spatial position measurement results……………...95

5.6.3 Determination process for the first points of triangular holes………….97

5.7 Implementation……………………………………………………………...98

Chapter 6 Conclusions and future works……………………………………...100

6.1 Conclusions………………………………………………………………..100

6.1.1 Conclusion for the classification and measurement system for metal parts.. 100

6.1.2 Conclusion for the classification and measurement system for color objects101