A study of energy monitoring and road-adaptive vibration reduction system for electric bicycles

Instilling Computational Thinking through making Augmented Reality application

by

The Vinh Nguyen

A Dissertation

In

Computer Science

Submitted to the Graduate Faculty

of Texas Tech University in

Partial Fulfillment of

the Requirements for

the Degree of

DOCTOR OF PHILOSOPHY

Approved

Dr. Tommy Dang

Chair of Committee

Dr. Kwanghee Jung

Co-chair of Committee

Dr. Yuanlin Zhang

Dr. Susan A. Mengel

Dr. Akbar Siami-Namin

Dr. Yo Woon Chong

Graduate Dean’s representative

Mark Sheridan

Dean of the Graduate School

October, 2020

Copyright 2020, The Vinh Nguyen

Texas Tech University, The Vinh Nguyen, October 2020

ii

ACKNOWLEDGMENTS

The purpose of this page is to recognize scholarly and professional aid and

advice; however, the inclusion of references to persons who provided clerical help,

help with field studies, financial assistance, and permission to use copyrighted

materials is also acceptable.

Acknowledgments should be brief, in a professional style, and should not

exceed two pages.

Texas Tech University, The Vinh Nguyen, October 2020

iii

TABLE OF CONTENTS

ACKNOWLEDGMENTS ...................................................................................II

ABSTRACT.........................................................................................................VI

LIST OF TABLES ............................................................................................ VII

LIST OF FIGURES .........................................................................................VIII

1. INTRODUCTION............................................................................................ 1

1.1 Motivation and Issues ................................................................................. 1

1.2 Research Contributions............................................................................... 5

2. BACKGROUND............................................................................................... 8

2.1 Framework for presenting complex models................................................ 8

2.2 Potential capability of generating web-based AR/VR applications in the

classroom setting............................................................................................. 10

2.3 Alternative ways of producing Web-based AR/VR applications through a 3-

part use case study........................................................................................... 12

2.4 Instilling Computational Thinking through using a Visual Programming

Interface........................................................................................................... 13

3. FRAMEWORK FOR PRESENTING COMPLEX MODELS.................. 15

3.1 Contributions............................................................................................. 15

3.2 Introduction............................................................................................... 17

3.3 Related work ............................................................................................. 19

3.4 Design ....................................................................................................... 21

3.4.1 Material contents.............................................................................. 22

3.4.2 Vuforia package for Unity3D........................................................... 24

3.4.3 Google Cardboard Package.............................................................. 25

3.4.4 Unity3D............................................................................................ 26

3.4.5 Application....................................................................................... 26

3.5 Evaluation ................................................................................................. 28

3.6 Challenge and Discussion ......................................................................... 29

3.7 Conclusion ................................................................................................ 30

Texas Tech University, The Vinh Nguyen, October 2020

iv

4. POTENTIAL CAPABILITY OF GENERATING WEB-BASED AR/VR

APPLICATIONS IN THE CLASSROOM SETTING FROM LEARNERS’

PERSPECTIVES ................................................................................................ 31

4.1 Contributions............................................................................................. 32

4.2 Introduction............................................................................................... 33

4.3 Related work ............................................................................................. 36

4.4 Methodology ............................................................................................. 37

4.4.1 Goal and objectives.......................................................................... 38

4.4.2 Study design..................................................................................... 38

4.4.3 Project assessment............................................................................ 40

4.4.4 Survey .............................................................................................. 41

4.4.5 Case Study........................................................................................ 42

4.5 Results....................................................................................................... 48

4.6 Lessons learned and challenges ................................................................ 54

4.7 Recommendations..................................................................................... 56

4.8 Conclusion ................................................................................................ 57

5. ALTERNATIVE WAYS OF PRODUCING WEB-BASED AR/VR

APPLICATIONS THROUGH A 3-PART USE CASE STUDY .................... 58

5.1 Contributions............................................................................................. 59

5.2 Introduction............................................................................................... 61

5.3 Related work ............................................................................................. 65

5.4 Methodology ............................................................................................. 66

5.4.1 Goal and objectives.......................................................................... 66

5.4.2 Study design..................................................................................... 67

5.4.3 Participants....................................................................................... 70

5.5 Use Case Study ......................................................................................... 71

5.5.1 Use Case 1: Developing a WebVR application ............................... 71

5.5.2 Use Case 2: Developing a VR/AR application on a given topic ..... 75

5.5.3 Use Case 3: Developing a VR/AR application on any topic ........... 78

5.6 Results and Discussion.............................................................................. 79

5.6.1 Research question 1: What VR/AR development framework/library best￾allowed users (novice to expert) to stimulate their interest in creating and

sharing VR/AR content in both perceived utility and ease of use?........... 79

5.6.2 Research question 2: Did the library/framework that students favored

afford them the ability to solve more complex problems?........................ 83

Texas Tech University, The Vinh Nguyen, October 2020

v

5.6.3 Research question 3: When given the choice, which library/framework

did students employ to develop a VR/AR application (based upon their

interests)? .................................................................................................. 84

5.6.4 Research question 4: Based upon reported learners' perspectives, what

were the pros and cons of WebVR compared to other app-based VR/AR tools?

................................................................................................................... 84

5.6.5 Lessons learned and discussion........................................................ 88

5.7 Conclusion and Future work ..................................................................... 91

6. INSTILLING COMPUTATIONAL THINKING THROUGH MAKING

AUGMENTED REALITY APPLICATION .................................................... 93

6.1 Contributions............................................................................................. 94

6.2 Introduction............................................................................................... 95

6.3 Related work ............................................................................................. 99

6.4 Methods................................................................................................... 101

6.4.1 System Design................................................................................ 101

6.4.2. Use Case........................................................................................ 114

6.4.3. Evaluation ..................................................................................... 116

6.5 Results..................................................................................................... 121

6.5.1 Qualitative Analysis....................................................................... 121

6.5.2 Quantitative Analysis..................................................................... 122

6.6 Discussion ............................................................................................... 125

6.7 Conclusion .............................................................................................. 127

7. CONCLUSION............................................................................................. 128

REFERENCES.................................................................................................. 130

Texas Tech University, The Vinh Nguyen, October 2020

vi

ABSTRACT

It is widely recognized that instilling and inculcating computational thinking

skills (CTS) such as problem formulation, effective representation of big data, and

identifying, analyzing and implementing possible solutions are essential for succeeding

in STEM disciplines. There is also a recognition that technology and human behavior

are tightly interrelated and leveraging computational thinking to understand complex

human-computer interactions is vital to foster systemic sustainable developments.

Augmented Reality is a technology that expands the physical world with additional

digital information. The central value of AR is that the components of the digital world

blend into a person's perception of the real world. It is not just simply showing the data

but through the integration of immersive sensations, which are perceived as natural parts

of an environment.

Traditional approaches involving making an AR application are heavily

dependent on a programming language in which the syntax of programming is not easy

to master for non-computer science users. Recent research has produced some insights

that describe how to lessen the issue of mastering a certain programming language for

young learners and enthusiasts. Block-based programming is a type of programming

language where instructions are mainly represented as blocks (or visual cues) and users

drag and drop the cues to form a set of instructions. This programming paradigm enables

developers to focus on logical programming rather than memorizing the syntax of

coding. However, in the existing studies, the interactions between 3D objects are

limited.

The purpose of this dissertation is to help students enhance computational

thinking skills for a successful future career through making an Augmented Reality

application. To tackle the aforementioned issues, we provide students with an

interactive web-based tool that allows them for experimenting, testing, abstracting,

modularizing, reusing, and remixing the application ideas.

Texas Tech University, The Vinh Nguyen, October 2020

vii

LIST OF TABLES

4.1 Pearson correlation test scores produced by SPSS software..................... 54

5.1 Participant distribution by gender vs graduate level................................. 71

5.2 Research questions for the survey............................................................. 73

5.3 Survey questions for peer evaluation in Project Case 2............................ 75

5.4 Pearson Correlation test scores produced by SPSS .................................. 82

5.5 A summary of the VR/AR application types and hardware...................... 83

6.1 Construct and items................................................................................. 119

6.2 General information about the participants............................................. 120

6.3 Means and standard deviations of TAM measures (N = 66). ................. 122

6.4 Internal Consistency and Convergence Validity..................................... 123

6.5 Estimates of loadings. ............................................................................. 124

6.6 Estimates of path coefficients ................................................................. 125

Texas Tech University, The Vinh Nguyen, October 2020

viii

LIST OF FIGURES

Figure 3.1 A comprehensive framework to build VR/AR application. ................ 21

Figure 3.2 Example of some free3D models that will be used in the

application..................................................................................... 22

Figure 3.3 Heightmap generator from terrain.party.............................................. 23

Figure 3.4 Terrain generated from heightmap in Unity ........................................ 24

Figure 3.5 Birch tree 3D model............................................................................. 25

Figure 3.6 Example of using tree in the inventory to plant and build the

city................................................................................................. 25

Figure 3.7 Main menu of the VR/AR game that allows to switch

between VR/AR mode .................................................................. 27

Figure 4.1 A collage of WebVR applications created by sampled

students.......................................................................................... 44

Figure 4.2 Sampled Students' WebVR project examples: (a) A moon

dream house (b) A 2-level dream house (c) A New York

skyline dream condo ..................................................................... 47

Figure 4.3 Students' grade distribution of the WebVR dream house

project: Level 1 is equivalent to a C while Level 3 is

equivalent to a A. .......................................................................... 48

Figure 4.4 Responses from students (regarding questions 1-3)............................ 49

Figure 4.5 Responses from students (regarding questions 4-6)............................ 50

Figure 4.6 Responses from students (regarding questions 7-8)............................ 52

Figure 4.7 Responses from students (regarding questions 9-10).......................... 53

Figure 5.1 The study design: 16-week activities................................................... 68

Figure 5.2 A collage of WebVR applications created by sampled

students.......................................................................................... 74

Figure 5.3 Three good WebVR project examples: (a) A moon dream

house (b) A relaxing dream house (c) A bar dream house............ 74

Figure 5.4 Students' grade distribution of the WebVR dream house

project: Level 1 is equivalent to a C while Level 3 is

equivalent to a A. .......................................................................... 75

Figure 5.5 A collage of VR/AR applications created by students. ....................... 77

Figure 5.6 A collage of VR/AR applications created by students in

Project 3. ....................................................................................... 78

Texas Tech University, The Vinh Nguyen, October 2020

ix

Figure 5.7 Survey results from students in Project 1 from question 1 to

10................................................................................................... 80

Figure 6.1 The visual interface of Blockly for generating JavaScript

from blocks ................................................................................. 100

Figure 6.2 The coding editor of BlocklyAR: it enables users to drag and

drop a palette of commands into the working space ................... 103

Figure 6.3 Different shapes of blocks allow users to stack or wire up

components together ................................................................... 105

Figure 6.4 The visual AR component enables enthusiasts to experience

their coding schemes in the mixed 3D space .............................. 113

Figure 6.5 Tutorial section where learners are guided on how to use

blocks and the connections among them..................................... 114

Figure 6.6 Use case of using BlocklyAR to recreate the Palmito Battle

Ranch........................................................................................... 116

Figure 6.7 The conceptual research model with extensions of Task￾Technology Fit and Visual Design variables. Each set of

ellipses represents a construct and an arrow denotes a

hypothesis.................................................................................... 118

Texas Tech University, The Vinh Nguyen, October 2020

1

CHAPTER 1

1. INTRODUCTION

1.1 Motivation and Issues

It is widely recognized that instilling and inculcating computational thinking skills

(CTS) such as problem formulation, effective representation of big data, and

identifying, analyzing and implementing possible solutions (Sykora, 2014; Wing, 2006)

are essential for succeeding in STEM disciplines (Assaraf & Orion, 2010; Kokkelenberg

& Sinha, 2010). Evaluation and understanding of highly complex systems with

extensive inter-connection and feedback has become an integral focus of many STEM

fields as the world we live in is increasingly becoming dynamic, self-organizing and

continually adaptive (Assaraf & Orion, 2010; Bower et al., 2017). A strong grounding

in CTS is an essential requirement to study these complex adaptive systems. There is

also a recognition that technology and human behavior are tightly interrelated and

leveraging computational thinking to understand complex human-computer interactions

is vital to foster systemic sustainable developments.

Computer programming is not only writing a code but the process of analyzing

a situation, identifying its key components, modeling the data and processes, and

creating or refining a program through an agile design-thinking approach to accomplish

a specific computing result. These strategies could be considered under the umbrella of

the CT concept (Wing, 2006), thus having and understanding creative programming

skills would improve computational thinking. Basic concepts in computer programming

include sequence (identifying an ordered series of steps), loop (running the same set of

sequences multiple times), parallelism (running multiple sequences at the same time),

events (one thing causing another thing to happen), conditional (making decisions based

on criteria), operator (mathematical and logical expressions), data (storing, retrieving,

and updating values).

Texas Tech University, The Vinh Nguyen, October 2020

2

Human Computer Interaction (HCI) focuses on the interaction between human

and computer and it has been applied in my fields such as user customization, embedded

computation, augmented reality, social computing and knowledge and knowledge

driven (Biseria & Rao, 2016). Augmented Reality (AR) is a technology that expands

the physical world with additional digital information such as sound, image, model, etc

(R. T. Azuma, 1997). The central value of AR is that the components of the digital world

blend into a person's perception of the real world. It is not just simply showing the data

but through the integration of immersive sensations, which are perceived as natural parts

of an environment. In recent years, the growth of AR applications can be attributed to

solutions that focus on contextualizing information (e.g., annotating different parts of a

physical object (Bruno et al., 2019), displaying artifacts at a given place (K. Jung et al.,

2020), aligning virtual objects with the real world (Norouzi et al., 2019)--that is,

automatically position an object on the detected table or floor). In the educational

setting, AR technology can be incorporated in the classroom to enhance

teaching/learning efficiency and the motivations of both educators and learners (R.

Azuma et al., 2001; V. T. Nguyen, Hite, & Dang, 2018).

Traditional approaches involving making an AR application are heavily

dependent on a programming language in which the syntax of programming is not easy

to master for non-computer science users. For example, developers rely on available

tools/frameworks such as ARCore, ARKit, Vuforia, Unity (Linowes & Babilinski,

2017), and ARToolkit (Kato, 2002) to create and manipulate AR applications. ARCore

is a framework from Google for building augmented reality applications for both

Android and iOS devices. Apple also provides ARKit for making AR apps and games

for their iOS devices. Both of these frameworks use a handheld device's sensors for

motion tracking, light estimation, and environmental understanding. Unlike ARCore

and ARKit, ARToolkit and Vuforia are computer tracking libraries that overlay virtual

objects on the real world based on markers. The above frameworks and libraries can be

integrated in Unity for porting an AR application on different operating system devices

(e.g., Android, iOS). To use these frameworks, it is necessary to have an integrated

Texas Tech University, The Vinh Nguyen, October 2020

3

development environment (e.g., XCode for iOS and Android Studio for Android, or

Unity for both iOS and Android), a compatible device (e.g., ARCore requires that the

device must be running Android at least 7.0, or ARKit requires device to be run on iOS

11 with an A9, A10, or A11 processor), and knowledge of a specific programming

language (e.g., Objective-C, C#, C/C++, Java). These requirements are obstacles for

beginners who want to create an AR experience on their own. In their study, Nguyen et

al. (Vinh T Nguyen, Jung, & Dang, 2019) showed that API version incompatibility in

the integrated development environment (IDE) is a major obstacle students face while

working with the application in terms of both coding and deploying. The study also

indicated that students faced difficulty in analyzing “800+ line scripts”.

To alleviate difficulty of having an IDE and a compatible device, web-based AR

(e.g., WebVR, WebXR) is an alternative approach for users to experience virtual objects

in the real world only by using a web browser on their handheld device. A web browser

engine supports Hypertext Markup Language (HTML), Cascading Style Sheets (CSS),

and scripting languages such as JavaScript that can be programmed with a simple text

editor, thus releasing users from the need for an IDE. Furthermore, web-based AR

toolkits such as ARToolkit for web (Kato, 2002) are compatible with a wider range of

devices (e.g., running on OS 4.0.3 or higher for Android and 7.0 or higher for iOS). The

use of ARToolkit has been widely adapted in other libraries such as ThreeJS (Danchilla,

2012), and A-Frame.io (Mozilla, 2019). Currently, although web-based AR has some

limitations due to its low framerate and not leveraging the full capacity of in-app AR,

the continual development of other technologies will help web-based AR keep growing

in the future. For example, the presence of an open binary instruction format

WebAssembly (WebAssembly, 2020) would allow the browser engine to run native

code in a browser and this provides the capability to access the in-app AR features

through a web-based VR.

Recent research has produced some insights that describe how to lessen the issue

of mastering a certain programming language for young learners and enthusiasts. Block￾based programming (Weintrop, 2019) is a type of programming language where

Texas Tech University, The Vinh Nguyen, October 2020

4

instructions are mainly represented as blocks (or visual cues) and users drag and drop

the cues to form a set of instructions. This programming paradigm enables developers

to focus on logical programming rather than memorizing the syntax of coding. Scratch

(Resnick et al., 2009) is an example of using this visual paradigm in K-12 education.

By using Scratch, K-12 students are able to program a 2D game and experience it

immediately on a web-browser. Radu and Blair (Radu & MacIntyre, 2009) extended

Scratch to make it possible to use to create AR. However, their work stopped at

rendering 2D images on the screen only; 3D objects and spatial position manipulation

have not been implemented. CoSpaces (CoSpaces, 2020) is a commercial product

created for education; it enables students to create virtual reality (VR) applications

through context-based language. It also can be used to create a simple AR app by

superimposing 3D objects onto the physical environment. However, the interactions

between 3D objects are limited. In addition, Laine (Laine, 2018) pointed out that the

majority of AR apps were developed through Vuforia SDK with a few occurrences of

ARToolkit, which would not be suitable for non-programmers, but expert programmers.

Furthermore, Wu et al.(Wu, Lee, Chang, & Liang, 2013) indicated that in some AR

systems, the learning content and the teaching sequence are rather ''fixed'' such that

teachers are not able to make changes to accommodate students' learning needs. As

such, an authoring/storytelling tool is highly needed for teachers and students to create

AR applications (Klopfer & Squire, 2008). In addition, learning programming with

activities such as generating prime numbers or sorting a list/array is not of interest for

many young learners. And finally, there is a little guidance when things go wrong or

encourage deeper explorations when things went right in some context of programming.

These limitations create a barrier to fostering computational thinking in STEM

education

The purpose of this dissertation is to help students enhance computational

thinking skills for a successful future career through making an Augmented Reality

application. To tackle the aforementioned issues, we provide students with an

interactive web-based tool that allows them for experimenting, testing, abstracting,