A 3D panoramic image guiding system design and implementation for Minimum invasive surgery

逢 甲 大 學

電機與通訊工程博士學位學程

博士論文

一個使用於微創手術的三維影像導引系統之

設計與實作

A 3D Panoramic Image Guiding System Design and

Implementation for Minimum Invasive Surgery

指導教授：鄭經華博士

研 究 生 ：金泰廷

中華民 國 一 百 零 九 年 二 月

A 3D Panoramic Image Guiding System Design and Implementation for Minimum Invasive Surgery

i

FCU e-Theses & Dissertations (2020)

Acknowledgements

I cannot complete the work of this thesis without the aid of many people. First of all, I

would like to thank my advisor Prof. Dr Ching-Hwa Cheng. He allowed me to join his lab

“Information and Technology Laboratory” at Feng Chia University and to work on this

challenging research project. That allowed me to study and discover a lot of exciting things

in medical imaging processing. I especially appreciate his dedicated guidance as well as

financial support and lab equipment.

I am very grateful to Prof. Dr Tang-Chieh Liu for taking the time to participate in our

weekly meetings and his helpful scientific advice. Besides, I would like to thank him for

reading my paper manuscripts and giving comments and guidance.

Special thanks go to Dr Kai Che Jack Liu and Dr Wayne Shih Wei Huang from the IRCAD

Taiwan/AITS, Taiwan, for enabling me to conduct in-vio animal experiments. Their

collaboration on the medical aspect of our research project was a great source of motivation.

I am also thankful to friends in my laboratory for the great time spent together and

especially to Thang, Toan, Tuan, and Tai for their suggestions and motivation.

Last but not least, I want to express gratefulness to my family and especially my wife for

their unlimited support and belief in me. Their continuous spiritual support is the

motivation for me to complete the work of this thesis.

A 3D Panoramic Image Guiding System Design and Implementation for Minimum Invasive Surgery

ii

FCU e-Theses & Dissertations (2020)

Abstract

Minimally invasive surgery (MIS) is gradually replacing traditional surgical methods

because of its advantages, such as causing less injury, preventing unsightly surgical scars,

and resulting in faster recovery time. In MIS, the surgeon performs surgery by observing

images transmitted from the endoscope. Therefore, there are three significant challenges in

MIS, which is a limited field of view (FOV), lack of depth perception, and viewing angle

control during surgery.

This thesis aims to explore how to solve these challenges using only images provided by

the endoscopic camera without the requirement for the additional device in the operating

room. We proposed a 3D Panoramic Image Guiding System for MIS to provide surgeons

a broad, optimal and stable view field with Focus‑Area 3D‑Vision in the surgical area. We

have designed a new endoscope that consists of two endoscopic cameras attached to the tip

of one tube. With the two-view images captured simultaneously from the two lenses, the

proposed algorithm can combine two camera’s FOV into one larger FOV. The overlap area

of the two cameras was also displayed in the 3D space. Besides, our system could be a 3D

measurement tool for endoscopic surgery. Finally, a surgical tool detection algorithm is

proposed to evaluate the surgical skills and control camera position during MIS.

The experiments of the proposed system were performed on phantom model images and

in-vivo animal images. Experimental results confirm that our system is feasible and give

promises to improve existing limitations in laparoscopic surgery.

Keywords: minimally invasive surgery (MIS), image stitching, 3D reconstruction, surgical

tool detection.

A 3D Panoramic Image Guiding System Design and Implementation for Minimum Invasive Surgery

iii

FCU e-Theses & Dissertations (2020)

Contents

Chapter 1. Introduction.....................................................................................................1

1.1 Minimally Invasive Surgery ......................................................................................1

1.2 Computer Assisted Interventions...............................................................................3

1.2.1 Image stitching....................................................................................................3

1.2.2 3D reconstruction................................................................................................4

1.2.3 Surgical Tool Detection ......................................................................................5

1.3 Thesis Overview ........................................................................................................6

1.3.1 Proposed Endoscope ...........................................................................................6

1.3.2 Problem Description ...........................................................................................7

1.3.3 Thesis Structure ..................................................................................................8

1.4 Thesis Contributions..................................................................................................9

Chapter 2. Image Stitching in Minimally Invasive Surgery ........................................10

2.1 Introduction..............................................................................................................10

2.2 Features-based Image Stitching ...............................................................................11

2.2.1 Feature Detection ..............................................................................................12

2.2.2 Feature Matching ..............................................................................................14

2.2.3 Find homography matrix...................................................................................14

2.2.4 Image Warping..................................................................................................15

2.2.5 Seam Estimation ...............................................................................................15

2.2.6 Image Blending.................................................................................................16

2.3 Proposed Video Stitching ........................................................................................16

2.3.1 Accelerating Image Registration.......................................................................17

2.3.2 Accelerating Image Composition .....................................................................19

2.4 Results......................................................................................................................20

2.4.1 Video-stitching Results.....................................................................................20

2.4.2 Run Time Estimation ........................................................................................21

2.4.3 Discussion.........................................................................................................23

2.5 Conclusions..............................................................................................................25

Chapter 3. 3D Reconstruction in Minimally Invasive Surgery ...................................26

3.1 Introduction..............................................................................................................26

3.2 Proposed Stereo Reconstruction ..............................................................................27

3.2.1. Image Rectification..........................................................................................27

A 3D Panoramic Image Guiding System Design and Implementation for Minimum Invasive Surgery

iv

FCU e-Theses & Dissertations (2020)

3.2.2 Disparity Map ...................................................................................................31

3.2.3 Dense Reconstruction .......................................................................................33

3.3 Results......................................................................................................................35

3.3.1 Experimental Description .................................................................................35

3.3.2 Evaluation of the Disparity Map and 3D Reconstruction.................................36

3.3.3 Evaluation of Distance Measurement ...............................................................39

3.3.3 Run time Evaluation .........................................................................................40

3.4 Discussion................................................................................................................41

3.5 Conclusion ...............................................................................................................41

Chapter 4. Video Stitching in Minimally Invasive Surgery.........................................42

4.1 Introduction..............................................................................................................42

4.2 The Proposed Image-Stitching Algorithm...............................................................44

4.2.1 Image Registration ............................................................................................44

4.2.2 Image Compositing...........................................................................................46

4.3 The proposed Video Stitching Algorithm................................................................46

4.3.1 Stitching Video at increased Speed...................................................................46

4.3.2 Increasing the Stability of the stitched video....................................................48

4.4 Experimental Results...............................................................................................48

4.4.1 Video-stitching results......................................................................................49

4.4.2 Comparison with the Previous Method.............................................................50

4.5 Discussion................................................................................................................53

4.4 Conclusions..............................................................................................................54

Chapter 5. Surgical Tool Detection in Minimally Invasive Surgery ...........................55

5.1 Introduction..............................................................................................................55

5.2 Surgical Tool Detection ...........................................................................................55

5.2.1 Dataset...............................................................................................................56

5.2.2 Method ..............................................................................................................56

5.2.3 Results...............................................................................................................58

5.3 Surgical Tool-Instance Segmentation ......................................................................59

5.3.1 Dataset...............................................................................................................59

5.3.2 Method ..............................................................................................................60

5.3.3 Results...............................................................................................................62

5.4 Conclusions..............................................................................................................63

Chapter 6. 3D Panoramic Image Guiding System for Minimum Invasive Surgery..64

6.1 Introduction..............................................................................................................64

6.2 Hardware..................................................................................................................64

6.3 Software ...................................................................................................................65

A 3D Panoramic Image Guiding System Design and Implementation for Minimum Invasive Surgery

v

FCU e-Theses & Dissertations (2020)

6.3.1 Video Stitching .................................................................................................65

6.3.2 3D Image...........................................................................................................65

6.3.3 Measurement.....................................................................................................66

6.3.4 Tool Detection ..................................................................................................66

6.3.5 Tool Tracking....................................................................................................66

6.3.6 Robot Control....................................................................................................66

6.4 Results......................................................................................................................67

6.5 Conclusion ...............................................................................................................69

Chapter 7. Conclusion .....................................................................................................70

7.1 Contributions............................................................................................................70

7.2 Limitations...............................................................................................................71

7.3 Future Work .............................................................................................................71

A 3D Panoramic Image Guiding System Design and Implementation for Minimum Invasive Surgery

vi

FCU e-Theses & Dissertations (2020)

List of Figures

Figure 1.1: The open surgery procedure. .............................................................................1

Figure 1.2: (a) The minimal invasive surgery procedure. (b) The set up for MIS...............2

Figure 1.3: The robotic-assisted MIS...................................................................................2

Figure 1.4: Examples of the image stitching from a moving camera: (a) Behrens et al. [2],

(b) Liu et al. [6] and (c) Ali et al. [4]. ..................................................................................4

Figure 1.5: Example of the 3D reconstruction base on moving a monocular endoscope [12]

..............................................................................................................................................5

Figure 1.6: Example of surgical tool detection [23] ............................................................6

Figure 1.7: The proposed endoscope system consisted of two cameras, a mechanical tube,

and a push-button. The figure depicts (a) the endoscopic cameras, (b) the geometric

arrangement between the two cameras, (c) the primary state of the device and (d) the

working state of the device. .................................................................................................7

Figure 1.8: The schematic diagram of our endoscope system. The two images on the left

side indicate the input images obtained from the two lenses. Through the USB ports on the

PC in the center, three outputs can be derived by our algorithm. The three images at the

right side indicate a window displaying a 3D image, a window showing an extended 2D

view, and another window showing the instrument detection result. ..................................8

Figure 2.1: The combination of the two limited camera’s FOV into a wider FOV...........11

Figure 2.2: Flowchart of image-stitching process..............................................................11

Figure 2.3: Calculating the sum of pixel intensities inside any rectangular region will only

require three additions and four memory accesses by using the integrated image: ∑=A-B￾C+D....................................................................................................................................12

Figure 2.4: Left to right and top to bottom: The Gaussian second-order derivatives Lxx, Lyy

, Lxy (top row) and their approximations Dxx, Dyy and Dxy (bottom row)..........................13

Figure 2.5: Conventional video-stitching algorithm..........................................................17

Figure 2.6: Proposed video-stitching algorithm.................................................................17

Figure 2.7: Overlap region during image stitching at frame t and frame t+1 (red), and ROI

region during image stitching at frame t (left, yellow), and small region during image

stitching at frame t+1 (right, yellow). ................................................................................17

Figure 2.8: ROI of Frame-1. Four corners of Frame-2 are transformed into four points P1,

P2, P3, and P4. Red rectangle is rectangle surrounding Frame-2*’s edges and is parallel to

Frame-1. ROI of Frame-1 is intersection of Frame-1 within red rectangle (green rectangle).

............................................................................................................................................18

A 3D Panoramic Image Guiding System Design and Implementation for Minimum Invasive Surgery

vii

FCU e-Theses & Dissertations (2020)

Figure 2.9: The image-stitching result (phantom model). The result expands the original

FOV of the input image by 60%. .......................................................................................20

Figure 2.10: The image-stitching result (animal experiment). The result expands the

original FOV of the input image by 55%...........................................................................20

Figure 2.11: Comparison of image registration times for conventional method (blue) and

proposed method (green) on the CPU computer (a) and the computer with an additional

GPU (b)..............................................................................................................................21

Figure 2.12: Comparison of seam estimation times for conventional method (blue) and

proposed method (green) on the CPU computer (a) and the computer with an additional

GPU (b)..............................................................................................................................22

Figure 2.13: Comparison of the stitched image for the conventional method and the

proposed method: (a) input images; (b) matching feature points and (c) stitched image by

conventional method; and (d) matching feature points and (f) stitched image by our method;

(e) ground truth. .................................................................................................................23

Figure 2.14: Transformation of Frame-2 into Frame-2

*

: (a) quadrilateral and (b) non￾quadrilateral. ......................................................................................................................24

Figure 2.15: Images captured by four cameras..................................................................24

Figure 2.16: Result of image stitching of four input images (area expansion ratio is 300%).

............................................................................................................................................24

Figure 3.1: 3D reconstruction of the cameras’ overlap by our endoscope. .......................26

Figure 3.2: The stereo reconstruction algorithm. There are three steps: 1. Image

rectification, 2. Disparity calculation, 3. 3D reconstruction. .............................................27

Figure 3.3: Pinhole camera model is used in this study.....................................................28

Figure 3.4: Radial distortion of the lens: (a) No distortion, (b) Positive distortion and (c)

Negative distortion.............................................................................................................29

Figure 3.5: Image rectification: (a) two input images, (b) two output aligned images......30

Figure 3.6: Disparity computation algorithm by stereoBM. The sum of absolute differences

(SAD) and Winner Takes All (WTA) are used for the disparity computation..................31

Figure 3.7: Disparity map calculation algorithm consists of three steps: (1) Compute

disparity map by StereoBM, (2) Compute WLS disparity map and confidence map by

WLS, (3) Compute WLS-FBS disparity map by FBS......................................................32

Figure 3.8: A stereo camera model ....................................................................................33

Figure 3.9: 3D reconstruction from ROI and disparity map..............................................34

Figure 3.10: Phantom model datasets................................................................................35

Figure 3.11: In-vivo animal datasets..................................................................................36

Figure 3.12: The qualitative evaluation results of the disparity map. Column1: Roi image

is the overlaped area of two input images. Column 2: Raw disparity map, computed by

StereoBM. Column 3: WLS disparity map, filtered by WLS. Column 4: WLS-FBS

disparity map, filtered by WLS+FBS. ...............................................................................37