Tài liệu Linux Shell Scripting Cookbook pdf

www.it-ebooks.info

Linux Shell Scripting

Cookbook

Solve real-world shell scripting problems with over 110

simple but incredibly effective recipes

Sarath Lakshman

BIRMINGHAM - MUMBAI

www.it-ebooks.info

Linux Shell Scripting Cookbook

Copyright © 2011 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system,

or transmitted in any form or by any means, without the prior written permission of the

publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the

information presented. However, the information contained in this book is sold without

warranty, either express or implied. Neither the author, nor Packt Publishing, and its

dealers and distributors will be held liable for any damages caused or alleged to be

caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the

companies and products mentioned in this book by the appropriate use of capitals.

However, Packt Publishing cannot guarantee the accuracy of this information.

First published: January 2011

Production Reference: 1200111

Published by Packt Publishing Ltd.

32 Lincoln Road

Olton

Birmingham, B27 6PA, UK.

ISBN 978-1-849513-76-0

www.packtpub.com

Cover Image by Charwak A ([email protected])

www.it-ebooks.info

Credits

Author

Sarath Lakshman

Reviewers

Aric Pedersen

John L Weatherwax

Acquisition Editor

Steven Wilding

Development Editor

Hyacintha D'Souza

Technical Editor

Hithesh Uchil

Indexer

Tejal Daruwale

Editorial Team Leader

Aanchal Kumar

Project Team Leader

Priya Mukherji

Project Coordinator

Jovita Pinto

Proofreader

Lesley Harrison

Production Coordinator

Kruthika Bangera

Cover Work

Kruthika Bangera

www.it-ebooks.info

About the Author

Sarath Lakshman is a 21 year old who was bitten by the Linux bug during his teenage

years. He is an undergraduate student of Computer Science and Engineering at Model

Engineering College, Cochin, Kerala.

Sarath is a life hacker who loves to explore innovations. He is a GNU/Linux enthusiast

and hactivist of free and open source software. He spends most of his time hacking

with computers and having fun with his great friends. Sarath is well known as the

developer of SLYNUX (2005)—a user friendly GNU/Linux distribution for Linux newbies.

The free and open source software projects he has contributed to are PiTiVi Video editor,

SLYNUX GNU/Linux distro, Swathantra Malayalam Computing, School-Admin, Istanbul,

and the Pardus Project. He has authored many articles for the Linux For You magazine

on various domains of FOSS technologies. He had made a contribution to several

different open source projects during his multiple Google Summer of Code projects.

He currently lives with his parents in Kerala. Sarath can be reached via his website

http://www.sarathlakshman.com.

I would like to thank my friends and family for the great support and

encouragement they have given me for all my endeavors. I would like

to thank my friends Anu Mahadevan and Neenu Jacob for the tireless

enthusiasm and patience to read through the chapter developments

and providing comments during development. I would also like to thank

Mr. Atanu Datta for helping me come up with the chapter titles. I extend

my gratitude to the team at Packt Publishing who helped me in making this

book happen.

www.it-ebooks.info

About the Reviewers

Aric Pedersen is the author of cPanel User Guide and Tutorial (ISBN 978-1-904811-

92-3) and Web Host Manager Administration Guide (ISBN 978-1-904811-50-3), both

written for Packt Publishing. He also served as a reviewer for CUPS Administrative Guide

(ISBN 978-1-84719-258-5) and Linux E-mail Second Edition (ISBN 978-1-847198-64-8)

published by Packt Publishing.

Aric has over nine years of experience working as a System Administrator. He currently

works for Hostdime.com, the world-class web host; and also for Netenberg.com,

makers of Fantastico, the world's most popular web script installer for cPanel servers.

I would like to thank PMUG (http://www.pmug-nj.org/) for years of

interesting content and camaraderie.

John L Weatherwax has been in love with computers ever since he had access to

one. An applied mathematician by trade, he began his study of computer programming

with the Pascal programming language on the computer that his father bought way

before most people had even thought of using a computer for anything truly practical.

As an undergraduate, he obtained dual degrees in physics and mathematics from the

University of Missouri Columbia and as a graduate student he obtained a Ph.D. from M.I.T.

in Applied Mathematics. After graduating, he worked for many years applying pattern

recognition techniques to various problems in electrical engineering before transitioning to

mathematical finance. In school, he was strongly influenced by the UNIX/Linux philosophy

of using simpler component programs together to solve larger problems. Many years later,

while exploring various trading strategies as a quantitative analysis at a Wall Street firm,

he still uses UNIX commands and the UNIX philosophy on a daily basis. Most of the work

he does now is related to using mathematical models to predict prices and trade various

financial instruments. He spends his free time working on applied mathematics and pattern

recognition (much of which can be found on his website at http://waxworksmath.com/)

and spending time with his family.

www.it-ebooks.info

www.PacktPub.com

Support files, eBooks, discount offers, and more

You might want to visit www.PacktPub.com for support files and downloads related to

your book.

Did you know that Packt offers eBook versions of every book published, with PDF and

ePub files available? You can upgrade to the eBook version at www.PacktPub.com and

as a print book customer, you are entitled to a discount on the eBook copy. Get in touch

with us at [email protected] for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up

for a range of free newsletters and receive exclusive discounts and offers on Packt books

and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book

library. Here, you can access, read, and search across Packt's entire library of books.

Why Subscribe?

• Fully searchable across every book published by Packt

• Copy and paste, print, and bookmark content

• On demand and accessible via web browser

Free Access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access

PacktLib today and view nine entirely free books. Simply use your login credentials for

immediate access.

www.it-ebooks.info

www.it-ebooks.info

www.it-ebooks.info

This book is dedicated to my parents who showed me the bright choices in life.

www.it-ebooks.info

www.it-ebooks.info

Table of Contents

Preface 1

Chapter 1: Shell Something Out 7

Introduction 7

Printing in the terminal 9

Playing with variables and environment variables 12

Doing math calculations with the shell 17

Playing with file descriptors and redirection 19

Arrays and associative arrays 25

Visiting aliases 27

Grabbing information about terminal 29

Getting, setting dates, and delays 30

Debugging the script 33

Functions and arguments 35

Reading the output of a sequence of commands 38

Reading "n" characters without pressing Return 40

Field separators and iterators 41

Comparisons and tests 44

Chapter 2: Have a Good Command 49

Introduction 50

Concatenating with cat 50

Recording and playback of terminal sessions 53

Finding files and file listing 55

Playing with xargs 63

Translating with tr 69

Checksum and verification 72

Sorting, unique and duplicates 75

Temporary file naming and random numbers 80

Splitting files and data 81

www.it-ebooks.info

ii

Table of Contents

Slicing file names based on extension 84

Renaming and moving files in bulk 86

Spell checking and dictionary manipulation 89

Automating interactive input 90

Chapter 3: File In, File Out 95

Introduction 96

Generating files of any size 96

Intersection and set difference (A-B) on text files 97

Finding and deleting duplicate files 100

Making directories for a long path 103

File permissions, ownership, and sticky bit 104

Making files immutable 109

Generating blank files in bulk 110

Finding a symbolic link and its target 111

Enumerating file type statistics 113

Loopback files and mounting 115

Creating ISO files, Hybrid ISO 117

Finding difference between files, patching 120

head and tail – printing the last or first 10 lines 122

Listing only directories – alternative methods 125

Fast command-line navigation using pushd and popd 126

Counting number of lines, words, and characters in a file 128

Printing directory tree 129

Chapter 4: Texting and Driving 131

Introduction 132

Basic regular expression primer 132

Searching and mining "text" inside a file with grep 136

Column-wise cutting of a file with cut 142

Frequency of words used in a given file 146

Basic sed primer 147

Basic awk primer 150

Replacing strings from a text or file 156

Compressing or decompressing JavaScript 158

Iterating through lines, words, and characters in a file 161

Merging multiple files as columns 162

Printing the nth word or column in a file or line 163

Printing text between line numbers or patterns 164

Checking palindrome strings with a script 165

Printing lines in the reverse order 169

Parsing e-mail addresses and URLs from text 171

www.it-ebooks.info

iii

Table of Contents

Printing n lines before or after a pattern in a file 172

Removing a sentence in a file containing a word 174

Implementing head, tail, and tac with awk 175

Text slicing and parameter operations 177

Chapter 5: Tangled Web? Not At All! 179

Introduction 180

Downloading from a web page 180

Downloading a web page as formatted plain text 183

A primer on cURL 183

Accessing Gmail from the command line 188

Parsing data from a website 189

Image crawler and downloader 191

Web photo album generator 193

Twitter command-line client 195

define utility with Web backend 197

Finding broken links in a website 199

Tracking changes to a website 200

Posting to a web page and reading response 203

Chapter 6: The Backup Plan 205

Introduction 205

Archiving with tar 206

Archiving with cpio 211

Compressing with gunzip (gzip) 212

Compressing with bunzip (bzip) 215

Compressing with lzma 217

Archiving and compressing with zip 219

squashfs – the heavy compression filesystem 220

Cryptographic tools and hashes 222

Backup snapshots with rsync 224

Version control based backup with Git 227

Cloning hard drive and disks with dd 230

Chapter 7: The Old-boy Network 233

Introduction 233

Basic networking primer 234

Let's ping! 241

Listing all the machines alive on a network 243

Transferring files 247

Setting up an Ethernet and wireless LAN with script 250

Password-less auto-login with SSH 253

Running commands on remote host with SSH 255

www.it-ebooks.info

iv

Table of Contents

Mounting a remote drive at a local mount point 259

Multi-casting window messages on a network 260

Network traffic and port analysis 262

Chapter 8: Put on the Monitor's Cap 265

Introduction 266

Disk usage hacks 266

Calculating execution time for a command 272

Information about logged users, boot logs, and failure boot 274

Printing the 10 most frequently-used commands 276

Listing the top 10 CPU consuming process in a hour 278

Monitoring command outputs with watch 281

Logging access to files and directories 282

Logfile management with logrotate 283

Logging with syslog 285

Monitoring user logins to find intruders 286

Remote disk usage health monitor 289

Finding out active user hours on a system 292

Chapter 9: Administration Calls 295

Introduction 295

Gathering information about processes 296

Killing processes and send or respond to signals 304

which, whereis, file, whatis, and loadavg explained 307

Sending messages to user terminals 309

Gathering system information 311

Using /proc – gathering information 312

Scheduling with cron 313

Writing and reading MySQL database from Bash 316

User administration script 321

Bulk image resizing and format conversion 325

Index 329

www.it-ebooks.info