Google Cloud Platform in Action

MANNING

JJ Geewax

Foreword by Urs Hölzle

Google Cloud Platform in Action

Google Cloud

Platform in Action

JJ GEEWAX

MANNING

SHELTER ISLAND

For online information and ordering of this and other Manning books, please visit

www.manning.com. The publisher offers discounts on this book when ordered in quantity.

For more information, please contact

Special Sales Department

Manning Publications Co.

20 Baldwin Road

PO Box 761

Shelter Island, NY 11964

Email: [email protected]

©2018 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in

any form or by means electronic, mechanical, photocopying, or otherwise, without prior written

permission of the publisher.

Many of the designations used by manufacturers and sellers to distinguish their products are

claimed as trademarks. Where those designations appear in the book, and Manning

Publications was aware of a trademark claim, the designations have been printed in initial

caps or all caps.

Recognizing the importance of preserving what has been written, it is Manning’s policy to have

the books we publish printed on acid-free paper, and we exert our best efforts to that end.

Recognizing also our responsibility to conserve the resources of our planet, Manning books

are printed on paper that is at least 15 percent recycled and processed without the use of

elemental chlorine.

The photographs in this book are reproduced under a Creative Commons license.

Manning Publications Co. Development editor: Christina Taylor

20 Baldwin Road Review editor: Aleks Dragosavljevic

PO Box 761 Technical development editor: Francesco Bianchi

Shelter Island, NY 11964 Project manager: Kevin Sullivan

Copy editors: Pamela Hunt and Carl Quesnel

Proofreaders: Melody Dolab and Alyson Brener

Technical proofreader: Romin Irani

Typesetter: Dennis Dalinnik

Illustrator: Jason Alexander

Cover designer: Marija Tudor

ISBN: 9781617293528

Printed in the United States of America

1 2 3 4 5 6 7 8 9 10 – DP – 23 22 21 20 19 18

v

brief contents

PART 1GETTING STARTED ........................................................1

1 ■ What is “cloud”? 3

2 ■ Trying it out: deploying WordPress on Google Cloud 24

3 ■ The cloud data center 38

PART 2STORAGE ....................................................................51

4 ■ Cloud SQL: managed relational storage 53

5 ■ Cloud Datastore: document storage 89

6 ■ Cloud Spanner: large-scale SQL 117

7 ■ Cloud Bigtable: large-scale structured data 158

8 ■ Cloud Storage: object storage 199

PART 3COMPUTING .............................................................241

9 ■ Compute Engine: virtual machines 243

10 ■ Kubernetes Engine: managed Kubernetes clusters 306

11 ■ App Engine: fully managed applications 337

12 ■ Cloud Functions: serverless applications 385

13 ■ Cloud DNS: managed DNS hosting 406

vi BRIEF CONTENTS

PART 4MACHINE LEARNING ..................................................425

14 ■ Cloud Vision: image recognition 427

15 ■ Cloud Natural Language: text analysis 446

16 ■ Cloud Speech: audio-to-text conversion 463

17 ■ Cloud Translation: multilanguage machine

translation 473

18 ■ Cloud Machine Learning Engine: managed

machine learning 485

PART 5DATA PROCESSING AND ANALYTICS.............................519

19 ■ BigQuery: highly scalable data warehouse 521

20 ■ Cloud Dataflow: large-scale data processing 547

21 ■ Cloud Pub/Sub: managed event publishing 568

vii

contents

foreword xvii

preface xix

acknowledgments xxi

about this book xxiii

about the cover illustration xxvii

PART 1GETTING STARTED ..............................................1

1 What is “cloud”? 3

1.1 What is Google Cloud Platform? 4

1.2 Why cloud? 4

Why not cloud? 5

1.3 What to expect from cloud services 6

Computing 6 ■ Storage 7 ■ Analytics (aka, Big Data) 8

Networking 8 ■ Pricing 9

1.4 Building an application for the cloud 9

What is a cloud application? 9 ■ Example: serving photos 10

Example projects 12

1.5 Getting started with Google Cloud Platform 13

Signing up for GCP 13 ■ Exploring the console 14

Understanding projects 15 ■ Installing the SDK 16

viii CONTENTS

1.6 Interacting with GCP 18

In the browser: the Cloud Console 18 ■ On the command line:

gcloud 20 ■ In your own code: google-cloud-* 22

2 Trying it out: deploying WordPress on Google Cloud 24

2.1 System layout overview 25

2.2 Digging into the database 26

Turning on a Cloud SQL instance 27 ■ Securing your Cloud SQL

instance 28 ■ Connecting to your Cloud SQL instance 30

Configuring your Cloud SQL instance for WordPress 30

2.3 Deploying the WordPress VM 31

2.4 Configuring WordPress 33

2.5 Reviewing the system 36

2.6 Turning it off 37

3 The cloud data center 38

3.1 Data center locations 39

3.2 Isolation levels and fault tolerance 42

Zones 42 ■ Regions 42 ■ Designing for fault tolerance 43

Automatic high availability 45

3.3 Safety concerns 45

Security 46 ■ Privacy 47 ■ Special cases 48

3.4 Resource isolation and performance 48

PART 2STORAGE ..........................................................51

4 Cloud SQL: managed relational storage 53

4.1 What’s Cloud SQL? 54

4.2 Interacting with Cloud SQL 54

4.3 Configuring Cloud SQL for production 60

Access control 60 ■ Connecting over SSL 61 ■ Maintenance

windows 66 ■ Extra MySQL options 67

4.4 Scaling up (and down) 68

Computing power 69 ■ Storage 69

4.5 Replication 71

Replica-specific operations 75

CONTENTS ix

4.6 Backup and restore 75

Automated daily backups 76 ■ Manual data export to

Cloud Storage 77

4.7 Understanding pricing 81

4.8 When should I use Cloud SQL? 83

Structure 83 ■ Query complexity 84 ■ Durability 84

Speed (latency) 84 ■ Throughput 84

4.9 Cost 85

Overall 85

4.10 Weighing Cloud SQL against a VM running MySQL 87

5 Cloud Datastore: document storage 89

5.1 What’s Cloud Datastore? 90

Design goals for Cloud Datastore 91 ■ Concepts 92

Consistency and replication 96 ■ Consistency with

data locality 99

5.2 Interacting with Cloud Datastore 101

5.3 Backup and restore 107

5.4 Understanding pricing 110

Storage costs 110 ■ Per-operation costs 110

5.5 When should I use Cloud Datastore? 111

Structure 111 ■ Query complexity 112 ■ Durability 112

Speed (latency) 112 ■ Throughput 113 ■ Cost 113

Overall 113 ■ Other document storage systems 115

6 Cloud Spanner: large-scale SQL 117

6.1 What is NewSQL? 118

6.2 What is Spanner? 118

6.3 Concepts 118

Instances 119 ■ Nodes 120 ■ Databases 120 ■ Tables 120

6.4 Interacting with Cloud Spanner 121

Creating an instance and database 122 ■ Creating a table 125

Adding data 127 ■ Querying data 127 ■ Altering database

schema 131

6.5 Advanced concepts 132

Interleaved tables 133 ■ Primary keys 136 ■ Split points 137

Choosing primary keys 138 ■ Secondary indexes 139

Transactions 145

x CONTENTS

6.6 Understanding pricing 152

6.7 When should I use Cloud Spanner? 153

Structure 154 ■ Query complexity 154 ■ Durability 154

Speed (latency) 154 ■ Throughput 154 ■ Cost 155

Overall 155

7 Cloud Bigtable: large-scale structured data 158

7.1 What is Bigtable? 159

Design goals 159 ■ Design nongoals 161

Design overview 162

7.2 Concepts 162

Data model concepts 163 ■ Infrastructure concepts 168

7.3 Interacting with Cloud Bigtable 173

Creating a Bigtable Instance 173 ■ Creating your schema 175

Managing your data 177 ■ Importing and exporting data 181

7.4 Understanding pricing 184

7.5 When should I use Cloud Bigtable? 185

Structure 185 ■ Query complexity 186 ■ Durability 186

Speed (latency) 186 ■ Throughput 186 ■ Cost 187

Overall 187

7.6 What’s the difference between Bigtable and HBase? 190

7.7 Case study: InstaSnap recommendations 191

Querying needs 191 ■ Tables 192 ■ Users table 192

Recommendations table 195 ■ Processing data 196

7.8 Summary 198

8 Cloud Storage: object storage 199

8.1 Concepts 200

Buckets and objects 200

8.2 Storing data in Cloud Storage 201

8.3 Choosing the right storage class 204

Multiregional storage 204 ■ Regional storage 205

Nearline storage 205 ■ Coldline storage 206

8.4 Access control 207

Limiting access with ACLs 207 ■ Signed URLs 213

Logging access to your data 217

8.5 Object versions 219

CONTENTS xi

8.6 Object lifecycles 223

8.7 Change notifications 225

URL restrictions 227

8.8 Common use cases 228

Hosting user content 228 ■ Data archival 229

8.9 Understanding pricing 230

Amount of data stored 231 ■ Amount of data transferred 232

Number of operations executed 233 ■ Nearline and Coldline

pricing 234

8.10 When should I use Cloud Storage? 236

Structure 236 ■ Query complexity 236 ■ Durability 236

Speed (latency) 237 ■ Throughput 237 ■ Overall 237

To-do list 237 ■ E*Exchange 238 ■ InstaSnap 238

PART 3COMPUTING....................................................241

9 Compute Engine: virtual machines 243

9.1 Launching your first (or second) VM 244

9.2 Block storage with Persistent Disks 245

Disks as resources 246 ■ Attaching and detaching disks 247

Using your disks 250 ■ Resizing disks 252 ■ Snapshots 253

Images 258 ■ Performance 259 ■ Encryption 261

9.3 Instance groups and dynamic resources 264

Changing the size of an instance group 269 ■ Rolling

updates 270 ■ Autoscaling 274

9.4 Ephemeral computing with preemptible VMs 276

Why use preemptible machines? 277 ■ Turning on preemptible

VMs 278 ■ Handling terminations 278 ■ Preemption

selection 279

9.5 Load balancing 280

Backend configuration 282 ■ Host and path rules 285

Frontend configuration 286 ■ Reviewing the configuration 287

9.6 Cloud CDN 289

Enabling Cloud CDN 290 ■ Cache control 293

9.7 Understanding pricing 294

Computing capacity 294 ■ Sustained use discounts 295

Preemptible prices 298 ■ Storage 298 ■ Network traffic 299

xii CONTENTS

9.8 When should I use GCE? 301

Flexibility 301 ■ Complexity 302 ■ Performance 302

Cost 302 ■ Overall 302 ■ To-Do List 303

E*Exchange 303 ■ InstaSnap 304

10 Kubernetes Engine: managed Kubernetes clusters 306

10.1 What are containers? 307

Configuration 307 ■ Standardization 307 ■ Isolation 309

10.2 What is Docker? 310

10.3 What is Kubernetes? 310

Clusters 312 ■ Nodes 312 ■ Pods 313 ■ Services 314

10.4 What is Kubernetes Engine? 315

10.5 Interacting with Kubernetes Engine 315

Defining your application 315 ■ Running your container

locally 317 ■ Deploying to your container registry 319

Setting up your Kubernetes Engine cluster 320 ■ Deploying

your application 321 ■ Replicating your application 323

Using the Kubernetes UI 325

10.6 Maintaining your cluster 327

Upgrading the Kubernetes master node 327 ■ Upgrading

cluster nodes 329 ■ Resizing your cluster 331

10.7 Understanding pricing 332

10.8 When should I use Kubernetes Engine? 332

Flexibility 332 ■ Complexity 333 ■ Performance 333

Cost 334 ■ Overall 334 ■ To-Do List 334

E*Exchange 335 ■ InstaSnap 335

11 App Engine: fully managed applications 337

11.1 Concepts 338

Applications 339 ■ Services 341 ■ Versions 342

Instances 342

11.2 Interacting with App Engine 343

Building an application in App Engine Standard 344

On App Engine Flex 353

11.3 Scaling your application 361

Scaling on App Engine Standard 362 ■ Scaling on App

Engine Flex 367 ■ Choosing instance configurations 368

CONTENTS xiii

11.4 Using App Engine Standard’s managed services 371

Storing data with Cloud Datastore 371 ■ Caching ephemeral

data 372 ■ Deferring tasks 374 ■ Splitting traffic 375

11.5 Understanding pricing 379

11.6 When should I use App Engine? 380

Flexibility 380 ■ Complexity 381 ■ Performance 381

Cost 381 ■ Overall 382 ■ To-Do List 382

E*Exchange 382 ■ InstaSnap 383

12 Cloud Functions: serverless applications 385

12.1 What are microservices? 385

12.2 What is Google Cloud Functions? 386

Concepts 388

12.3 Interacting with Cloud Functions 391

Creating a function 391 ■ Deploying a function 392

Triggering a function 394

12.4 Advanced concepts 395

Updating functions 395 ■ Deleting functions 396

Using dependencies 396 ■ Calling other Cloud APIs 399

Using a Google Source Repository 401

12.5 Understanding pricing 403

13 Cloud DNS: managed DNS hosting 406

13.1 What is Cloud DNS? 407

Example DNS entries 409

13.2 Interacting with Cloud DNS 410

Using the Cloud Console 410 ■ Using the Node.js client 414

13.3 Understanding pricing 418

Personal DNS hosting 418 ■ Startup business DNS hosting 418

13.4 Case study: giving machines DNS names at boot 419

PART 4MACHINE LEARNING........................................425

14 Cloud Vision: image recognition 427

14.1 Annotating images 428

Label annotations 429 ■ Faces 432 ■ Text recognition 435

Logo recognition 437 ■ Safe-for-work detection 440

Combining multiple detection types 441

xiv CONTENTS

14.2 Understanding pricing 443

14.3 Case study: enforcing valid profile photos 443

15 Cloud Natural Language: text analysis 446

15.1 How does the Natural Language API work? 447

15.2 Sentiment analysis 448

15.3 Entity recognition 452

15.4 Syntax analysis 455

15.5 Understanding pricing 457

15.6 Case study: suggesting InstaSnap hash-tags 459

16 Cloud Speech: audio-to-text conversion 463

16.1 Simple speech recognition 465

16.2 Continuous speech recognition 467

16.3 Hinting with custom words and phrases 468

16.4 Understanding pricing 469

16.5 Case study: InstaSnap video captions 469

17 Cloud Translation: multilanguage machine translation 473

17.1 How does the Translation API work? 475

17.2 Language detection 477

17.3 Text translation 479

17.4 Understanding pricing 481

17.5 Case study: translating InstaSnap captions 481

18 Cloud Machine Learning Engine: managed machine

learning 485

18.1 What is machine learning? 485

What are neural networks? 486 ■ What is TensorFlow? 488

18.2 What is Cloud Machine Learning Engine? 491

Concepts 492 ■ Putting it all together 495

18.3 Interacting with Cloud ML Engine 498

Overview of US Census data 498 ■ Creating a model 499

Setting up Cloud Storage 501 ■ Training your model 503

Making predictions 506 ■ Configuring your underlying

resources 509