Automating UNIX and Linux administration

Automating UNIX and Linux Administration

by Kirk Bauer ISBN:1590592123

Apress © 2003

This book focuses on automating the tedious daily tasks of system administration. It provides real￾world examples and explores the tools useful for managing 2 or 5,000 systems.

Table of Contents

Automating UNIX and Linux Administration

Preface

Chapter 1 - Introducing the Basics of Automation

Chapter 2 - Using SSH to Securely Automate System Administration

Chapter 3 - Creating Login Scripts and Shell Scripts

Chapter 4 - Pre-Installation—Network Preparation and Management

Chapter 5 - Automating and Customizing Installation

Chapter 6 - Automatic System Configuration

Chapter 7 - Sharing Data Between Systems

Chapter 8 - Packages and Patches

Chapter 9 - System Maintenance and Changes

Chapter 10 - System Monitoring

Chapter 11 - Improving System Security

Chapter 12 - Backing Up and Restoring Data

Chapter 13 - User Interfaces

Appendix A - Introduction to Basic Tools

Appendix B - Customizing and Automating Red Hat Linux Installation

Appendix C - Building Red Hat Package Manager (RPM) Packages

Index

List of Figures

List of Tables

List of Sidebars

Automating UNIX and Linux Administration

by Kirk Bauer ISBN:1590592123

Apress © 2003

This book focuses on automating the tedious daily tasks of system administration. It provides real￾world examples and explores the tools useful for managing 2 or 5,000 systems.

Table of Contents

Automating UNIX and Linux Administration

Preface

Chapter 1 - Introducing the Basics of Automation

Chapter 2 - Using SSH to Securely Automate System Administration

Chapter 3 - Creating Login Scripts and Shell Scripts

Chapter 4 - Pre-Installation—Network Preparation and Management

Chapter 5 - Automating and Customizing Installation

Chapter 6 - Automatic System Configuration

Chapter 7 - Sharing Data Between Systems

Chapter 8 - Packages and Patches

Chapter 9 - System Maintenance and Changes

Chapter 10 - System Monitoring

Chapter 11 - Improving System Security

Chapter 12 - Backing Up and Restoring Data

Chapter 13 - User Interfaces

Appendix A - Introduction to Basic Tools

Appendix B - Customizing and Automating Red Hat Linux Installation

Appendix C - Building Red Hat Package Manager (RPM) Packages

Index

List of Figures

List of Tables

List of Sidebars

Back Cover

Wouldn’t you like to automate the tedious daily tasks of system administration? Automating UNIX and Linux

Administration will show you how, by exploring existing tools and offering real-world examples. Parts of the book are

Linux-specific, but most of it applies to a UNIX system, including multiple variants of UNIX. Author Kirk Bauer briefly

overviews tools and technologies—and assumes preliminary knowledge about editing a configuration file or mounting a

file system.

The techniques, methods, and tools in this book will help you manage a single system—but will prove especially

powerful across multiple systems. No matter if the systems are desktops, servers, or Beowulf clusters—all of them will

benefit from this automation. And managing five to five thousand systems will become a simpler task!

About the Author

Kirk Bauer has been involved in computer programming since 1985. He has been using and administering UNIX

systems since 1994. Although his personal favorite UNIX variant is Linux, he has administered and developed on

everything from FreeBSD to Solaris, AIX, and IRIX. He is the author of various open-source system administration

programs such as AutoRPM and Logwatch.

Bauer has been involved with software development and system/network administration since his first year at Georgia

Tech. He has done work for the Georgia Tech Residential Network, the Georgia Tech Research Institute, and the Fermi

National Accelerator Laboratory. Bauer was one of the founders and the CTO of TogetherWeb in 2000, which was

purchased in 2003 by Proficient Systems. He is currently a software architect with Proficient Systems, and continues to

support and develop the collaborative browsing software and Linux-based network appliance created by TogetherWeb,

including C++ server software that provides high scalability, high efficiency, and high reliability.

Kirk graduated from Georgia Tech in 2001 with a bachelor’s degree in computer engineering. Shortly thereafter, he

began work on his first book, Automating UNIX and Linux Administration, which was published by Apress in September

2003.

Automating UNIX and Linux Administration

Kirk Bauer

Apress™

Copyright © 2003 Kirk Bauer

All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means,

electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system,

without the prior written permission of the copyright owner and the publisher.

(pbk):

1-59059-212-3

10987654321

Trademarked names may appear in this book. Rather than use a trademark symbol with every occurrence of

a trademarked name, we use the names only in an editorial fashion and to the benefit of the trademark

owner, with no intention of infringement of the trademark.

Technical Reviewers: Nate Campi, Erik Melander, Alf Wachsmann Editorial Board: Dan Appleman, Craig

Berry, Gary Cornell, Tony Davis, Steven Rycroft, Julian Skinner, Martin Streicher, Jim Sumser, Karen

Watterson, Gavin Wray, John Zukowski

Assistant Publisher: Grace Wong

Copy Editor: Rebecca Rider

Production Manager: Kari Brooks

Proofreader: Laura Cheu

Compositor: Susan Glinert Stevens

Indexer: Kevin Broccoli

Cover Designer: Kurt Krames

Manufacturing Manager: Tom Debolski

Distributed to the book trade in the United States by Springer-Verlag New York, Inc., 175 Fifth Avenue, New

York, NY, 10010 and outside the United States by Springer-Verlag GmbH & Co. KG, Tiergartenstr. 17, 69112

Heidelberg, Germany.

In the United States: phone 1-800-SPRINGER, email [email protected], or visit http://www.springer￾ny.com. Outside the United States: fax +49 6221 345229, email [email protected], or visit

http://www.springer.de.

For information on translations, please contact Apress directly at 2560 Ninth Street, Suite 219, Berkeley, CA

94710. Phone 510-549-5930, fax 510-549-5939, email [email protected], or visit http://www.apress.com.

The information in this book is distributed on an "as is" basis, without warranty. Although every precaution has

been taken in the preparation of this work, neither the author(s) nor Apress shall have any liability to any

person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by

the information contained in this work.

The source code for this book is available to readers at http://www.apress.com in the Downloads section.

To my parents, especially my father, who started and supported my interest in computers, and my

loving wife Amber, who has supported me throughout this process.

About the Author

Kirk Bauer has been using computers and programming since 1985. He has been using and administering

UNIX systems since 1994. Although his personal favorite UNIX variant is Linux, he has administered

everything from FreeBSD to Solaris, AIX, and IRIX.

Kirk has been involved with software development and system/network administration since his first year at

Georgia Tech. He has done work for the Georgia Tech Residential Network, the Georgia Tech Research

Institute, and the Fermi National Accelerator Laboratory. Kirk was one of the founders and the CTO of

TogetherWeb in 2000, which was purchased in 2003 by Proficient Systems. Kirk is currently a software

architect with Proficient Systems and continues to support and develop the collaborative browsing software

and Linux-based network appliance created by TogetherWeb.

Kirk's latest development is a fully automated installation, configuration, management, and monitoring system

that is used to deploy Proficient's software on RLX ServerBlades. Saving time through automation has always

been his passion, as evidenced by his collection of open-source software—the most popular being AutoRPM

and Logwatch.

Shortly after graduating from Georgia Tech in 2002 with a bachelor's of science in Computer Engineering,

Kirk married Amber, the love of his life. They currently live in Peoria, AZ with their two dogs and four cats.

When not using a computer, Kirk can be found involved in one of his many hobbies. Kirk enjoys reading,

playing strategy games, taking pictures, and watching movies. He likes to snow ski, water ski, and scuba dive

when he gets the opportunity. Skydiving is his favorite sport— Kirk has made over 1,400 jumps to date.

About the Technical Reviewers

Nate Campi is a UNIX and network administrator in Silicon Valley. He is currently employed at a (Linux￾based) network appliance vendor, running all aspects of their internal IT, helping guide development of

products, and designing DNS architectures for customers.

Past jobs include postmaster, hostmaster, and webmaster duties at Terra Lycos, UNIX and network

administrator at several Silicon Valley web hosting companies, and a tour as a hospital corpsman in the US

Navy. While in the Navy, Nate developed a love of teaching as a instructor for basic and advanced life

support, and also pediatric advanced life support for the American Heart Association.

He is married, has a son and a daughter, lives in the San Francisco East Bay, and dreams of one day owning

a home in the area.

Erik Melander, Managing Architect of Central Systems at Wyndham International, has a decade of

experience with UNIX systems, including time working with the University of Minnesota and IBM Global

Services. Currently, he is evolving Wyndham's IT infrastructure to realize a shared vision of autonomic

computing.

Alf Wachsmann holds a doctorate of natural science in Computer Science from the University of Paderborn

in Germany. He wrote his thesis about parallel and distributed computing.

He then worked at DESY, Germany's national high-energy physics lab, where he learned system software

programming and system administration in a very heterogeneous UNIX environment. His specialty became

automation.

Wachsmann then moved from Germany to the San Francisco Bay area, where he now works at the Stanford

Linear Accelerator Center, a site with fewer UNIX versions but with a lot more computers. His main focus is

again automating system administration and system infrastructure tasks.

Other professional interests include the OpenAFS filesystem and Kerberos 5 authentication.

Acknowledgments

I have used computers since I was very young and have always loved them, thanks in large part to the

support of my parents throughout my childhood. They have always helped me learn and have supported me

in whatever I wanted to do. I have to particularly thank my father who, much to my Mom's chagrin, came

home one day with my very first computer—a Commodore Vic 20. My life was never the same after that.

I also thank my lovely wife, Amber, who has shared me with this book for many, many months. She has been

supportive and understanding, even though I started writing the book just after our honeymoon was over.

In addition, my friend and colleague Moshe Jacobson has been very helpful in this process. He quickly

answered my many questions and even did some of the technical review. He helped make writing this book

the learning experience that it was.

Finally, I must thank everybody at Apress for helping me through my first book—especially my editor, Jim

Sumser, who has stuck by me through this long and arduous process.

Preface

Admit it. You are reading this book because you are lazy. Lazy system administrators are wonderful

people—who else is willing to spend so much time now in order to do nothing later? We all dream of waking

up in the morning, grabbing the laptop from the bedside table, checking our email, and then heading off to

the lake for the day.

Using the techniques in this book, you can get closer to the ideal world of fully automated system

administration. Although unexpected things always go wrong, we can at least delegate all of the mundane

and repetitive tasks to the computer (whose purpose, of course, was to make our lives easier). I will leave it

up to you to convince your boss that you only need to come in to work one day per week.

Benefits of Automation

In most cases, the motivation behind automation is saving time. We are busy people and our time is valuable.

We would rather write a script to add a user than add one manually a few times a day. We can then take that

time we save and spend it doing things that aren't as easy to automate (or things that are much more

entertaining). There are other benefits of automation, however, that are not quite as apparent.

In many cases, automation allows others to do things that they don't have enough direct knowledge to do

themselves. These other people range from inexperienced system administrators working under you to

support staff manning the corporate help desk. Your automation makes everybody's lives much easier. They

don't have to bother you so much, and you don't have to answer the same questions every day.

Equally important is the unintentional documentation that can result from automation. For example, to add a

new account, you have to add it to the passwd, shadow, and group files, as well as create a home

directory on the file server and set up the automount tables. Although you normally thoroughly document and

follow step-by-step procedures for most of your administration tasks, you somehow manage to neglect this

particular task.

This is where automation is very helpful. If you write a script to do all of the tasks required to create a user,

you have effectively written a step-by-step guide explaining how to create a new account. If you put some

good comments in the script, you have documented the process as well. The script comes in handy when you

haven't added a new account for three months. Even if the script is dated and fails to operate correctly, you

still know what was supposed to happen and that it was supposed to work. Instead of having to re-create the

process from scratch, you can just tweak the script so that it will work this time and the next.

Uses for Automation Techniques

Automation can be beneficial even when it is limited to one system. Regardless of how simple the system is

or how little it does, menial tasks still need to be done, logs need to be monitored, and so on. Of course, the

benefits of automation really start to outweigh the costs when it is deployed across several systems.

Managing hundreds or even thousands of machines can be fun. It can also be a nightmare if things are not

done right. To avoid this and reduce your future workload, make sure to do things correctly from the start.

However, there are also many things you can do to make life easier when you are managing existing sets of

machines. So even if you can't start from scratch, you may still find this book very helpful. For example, as we

will discuss later, you can use methods to automatically standardize machines in your existing network.

If you are managing more than one machine, you probably think that it would be nice if all of the machines

had the same hardware, operating system, and software. Sometimes this is possible (if you are using Beowulf

clusters, for instance); usually it is not (especially in a software engineering company with 1,000

programmers). This book deals with both uniform and mixed UNIX environments. Although the examples I

use only directly apply to Linux and UNIX environments, you can apply these ideas to any situation.

Who Will Benefit?

I have written this book for the experienced system administrator. This doesn't mean that you have to be an

expert, it just means that you need to have a little experience before you will find this book valuable. For

instance, if you can't remember how to mount a filesystem without looking it up in a reference manual, then

this book may be too advanced for you.

In addition to assuming that you can perform basic system administration tasks, I assume that you are familiar

with both Perl and/or bash scripting. In addition, I expect you to understand basic regular expression syntax.

If you are a student and are running a Linux server, then this book could be for you. If you have a few UNIX

boxes at work, then you will probably find this book useful. If you administer a few UNIX workstations, a set of

UNIX servers, a web farm, or a Beowulf cluster, then this book could be a lifesaver.

Techniques Covered

You can take one of three approaches when you go to solve the automation problems I present in this book:

Open-source software: Open-source solutions exist for many of the problems I present in this book,

but I do not attempt to cover every open source solution to every problem. Instead, I cover the most

popular and mature software in each category. Some programs I cover include GNU cfengine,

Logwatch, Swatch, AutoRPM, and NetSaint (which is currently being developed as Nagios). This book

is not a complete instruction manual for any of this software, but it does provide enough information for

you to consider and begin to use these programs.

Custom scripts and software: Another focus of this book involves custom solutions. Why am I

advocating custom solutions when open-source solutions already exist? I'm not. I definitely recommend

that you use existing solutions when possible. However, there are many cases in which existing

solutions are too complicated, are not powerful enough, or are too restrictive for your particular needs.

In addition, there are also plenty of areas within automation where existing solutions are hard to come

by. This is usually because the situation is significantly different in each case or the solution is too

simple to release as a product.

Commercial software: I do not cover commercial software in this book— not because there is

anything wrong with the software, it is just too costly for many situations (especially when you consider

the consulting costs typically associated with this type of software). In addition, not enough information

is available on the commercial solutions for me to fully discuss them within this book. However, for

some situations commercial software may be the best solution and I suggest that you consider it if it fits

your needs.

Chapter Summary

The book begins with introductory chapters that you should be very familiar with before you move on to the

meat of the text. It then proceeds with one chapter on each core area of automation. Each chapter fully

discusses the area in question, describes both existing and custom solutions to each problem, and provides

numerous examples.

Chapter 1: Introducing the Basics of Automation covers the methodology behind system

administration. automation.

Chapter 2: Using SSH to Securely Automate System Administration

covers the basics of using Secure Shell (SSH), discusses SSH security concerns, describes how to set

up passwordless SSH, and delves into various other related topics.

Chapter 3: Creating Login Scripts and Shell Scripts discusses some more advanced features of

bash and how you can use them to customize your prompt, create command aliases, enhance tab

completion, and otherwise enhance your shell experience. This chapter also provides scripts that allow

commands to run across multiple systems.

Chapter 4: Pre-Installation: Network Preparation and Management

discusses the tasks you need to complete to add a new system to your network and provides a custom,

modular shell script you can use to automate these tasks.

Chapter 5: Automating and Customizing Installation discusses the options available to you when

you want to automatically perform operating system installations. This chapter also discusses your

options for customizing your operating systems and provides scripts that allow a new system (even

without a custom operating system) to quickly and easily join your automation system.

Chapter 6: Automatic System Configuration covers the automatic configuration all of the systems on

your network, regardless of the operating systems they run or the tasks they perform. This chapter

provides both a custom solution and a comprehensive discussion of <application>GNU

cfengine</application>.

Chapter 7: Sharing Data Between Systems discusses many methods you can use to share data

among your various systems and talks about network filesystems such as the Network File System

(NFS), Network Information Services (NIS/NIS+), GNU cfengine, rsync, Concurrent Versioning System

(CVS), and the Hypertext Transfer Protocol (HTTP) and the File Transfer Protocol (FTP) protocols.

Chapter 8: Packages and Patches discusses the issues involved in updating many systems. This

chapter also talks about both Solaris and custom patches, the Red Hat Package Manager (RPM) and

Debian package formats, and a custom package solution. In addition, it covers automatic package

installation with AutoRPM and introduces the OpenPKG system, which provides packages that can be

installed on many different operating systems.

Chapter 9: System Maintenance and Changes covers the various maintenance tasks inherent in

most modern operating systems. This includes time synchronization, account management, log file

rotation, and general system cleanup.

Chapter 10: System Monitoring introduces techniques for monitoring your systems. This chapter

presents Logwatch and swatch for log monitoring and NetSaint for network monitoring.

Chapter 11: Improving System Security discusses tools that can enhance your system's security with

minimal maintenance on your part. This chapter also shows you how to use and configure Tripwire,

how to create automatic firewall configurations, and how to use GNU cfengine to increase your system

and network security.

Chapter 12: Backing Up and Restoring Data presents simple and inexpensive solutions for

automatically backing up your systems' data.

Chapter 13: User Interfaces shows you how to create both console- and web-based user interfaces

for your automation system using bash, Perl, and Mason.

Appendix A: Introduction to Basic Tools provides a basic introduction to the tools used throughout

the book and provides a good starting point for understanding and utilizing the examples presented in

this text. This appendix covers the following tools: bash, Perl, grep, sed, and awk.

Appendix B: Customizing and Automating Red Hat Linux Installation

shows you how to automate the installation of Red Hat Linux. Also this appendix shows you how to

create your own custom Linux distribution derived from Red Hat Linux.

Appendix C: Building Red Hat Package Manager (RPM) Packages houses comprehensive

instructions on how to build your own RPMs.

Additional Resources

The process of automating system administration covers a wide range of topics. Throughout the book, I will

suggest additional reading material. Also, there are several additional books that you will most likely find

helpful and I would like to mention them here:

UNIX System Administration Handbook (3rd Edition), by Evi Nemeth, Garth Snyder, Scott Seebass, Trent

R. Hein (Prentice Hall, 2000). This book covers almost anything you ever wanted to know about UNIX

system administration.

Principles of Network and System Administration, by Mark Burgess (John Wiley & Sons, 2000). This book

discusses the theory and principles behind good network and system administration.

Unix Storage Management, by Ray A. Kampa and Lydia V. Bell (Apress, 2002). This book covers

everything you ever wanted to know about storing data on UNIX systems.

Learning the bash Shell, by Cameron Newham and Bill Rosenblatt (O'Reilly and Associates, 1998). This

book covers the bash command shell as well as bash shell scripting.

Learning Perl, by Randal L. Schwartz and Tom Phoenix (O'Reilly and Associates, 2001). This is the first

place to start if you want to learn Perl, a very powerful scripting language that is used extensively

throughout this book.

Programming Perl, by Larry Wall et al. (O'Reilly and Associates, 2000). This more advanced guide to Perl

provides extensive reference material.

Perl for System Administration, by David N. Blank-Edelman (O'Reilly and Associates, 2000). You already

know we use Perl extensively in this book. By looking at the title, you also probably know that this book is

about automating system administration. Any questions?

Practical UNIX & Internet Security, by Simson Garfinkel, Alan Schwartz, and Gene Spafford (O'Reilly and

Associates, 2003). This book discusses security, which should always be on your mind, especially when

you automating system administration.

Conventions Used in This Book

I have used several special font and formatting conventions in this book. This section reviews these

conventions and how they are applied.

Here are some of the most common formatting conventions you will come across:

Note Notes call attention to important issues or interesting facts about the given subject.

Tip Tips provide additional information about the subject that you may find useful.

Caution Cautions serve to alert you about potentially dangerous side effects your actions or the

presented scripts may cause.

Warning Warnings provide important information that you must be aware of before proceeding, such as

potentially serious side effects.

Sidebars

Sidebars are used to provide larger amounts of related information that you may or may not be interested

in reading.

This book contains a large number of program listings. Sometimes they are shown as a block of code:

#!/bin/bash

echo "hello, how are you?"

Other times they may be broken into sections. I may first define a function:

say_hello() {

echo "Hello, how are you?"

}

and then provide the code that calls the function:

case $0 in

hello)

say_hello

;;

esac

To try these examples, you can download the code samples from the Downloads section of the Apress web

site (http://www.apress.com). Alternatively, you can create your own script file by combining the separate

parts of the script into one file, with the proper interpreter declaration at the top (such as #!/bin/bash).

Finally, some code listings will have their lines numbered and some of the numbered lines will be bold. These

lines will be discussed in more detail after the code. Here is an example:

1 #!/bin/bash

2

3 echo "Hello, how are you?"

Line 1: Run this using bash as the interpreter.

Line 3: Output the string to stdout using the echo command.

Throughout the book, you will often see examples of commands being run as follows:

% ./hello

echo "Hello, how are you?"

Note that a constant-width font is used, just like on a terminal. Anything that you should type is presented in

bold.

Here are some of the most common font conventions you will come across:

Monospaced: Monospaced font is used for a variety of purposes in this book. I use it to identify literal

strings, such as Hello, how are you?. I also use it for URLs (such as http://www.apress.com),

commands (like grep), filenames (/etc/passwd), and within program listings.

Bold: Bold is used to indicate something the reader or user should type, usually at a command prompt.

Italics: Italics is used to emphasis something or to indicate that a term is being defined.

We'd Like to Hear from You

We have gone through several stages of proofreading and error checking during the production of this book

in an effort to reduce the number of errors. We have also tried to make the examples and the explanations as

clear as possible.

There may, however, still be errors and unclear areas in this book. If you have questions or find any of these

errors, please feel free to contact me at [email protected]. You can also visit the Apress web site at

http://www.apress.com to download code from the book and see any available errata.