Red Hat Linux Security and Optimization ppt

S ECUR I T Y

TO

O L S

ON

CD- ROM

®

® PRESS

®

®

PRESS

and Optimization

Red Hat Linux Security

Linux Solutions from the Experts at Red Hat SECURITY TOOLS ON CD-ROM

Mohammed J. Kabir

Kabir

Your Official Red Hat

® Linux

® Guide to Security and Optimization

MOHAMMED J. KABIR is the founder and CEO of Evoknow, Inc., a company

specializing in customer relationship management software development.

His books include Red Hat Linux 7 Server, Red Hat Linux Administrator’s

Handbook, Red Hat Linux Survival Guide, and Apache Server 2 Bible.

■ Upgrade and configure your hardware to boost performance

■ Customize the kernel and tune the filesystem for optimal results

■ Use JFS and LVM to enhance the filesystem reliability and manageability

■ Tweak Apache, Sendmail, Samba, and NFS servers for increased speed

■ Protect against root compromises by enabling LIDS and Libsafe in the

kernel

■ Use PAM, OpenSSL, shadow passwords, OpenSSH, and xinetd to enhance

network security

■ Set up sensible security on Apache and reduce CGI and SSI risks

■ Secure BIND, Sendmail, ProFTPD, Samba, and NFS servers

■ Create a highly configurable packet filtering firewall to protect your

network

■ Build a secure virtual private network with FreeS/WAN

■ Use port scanners, password crackers, and CGI scanners to locate vulner￾abilities before the hackers do

Reviewed and approved by the experts at Red Hat, this comprehensive

guide delivers the know-how you need to improve the performance of

your Red Hat Linux system—and protect it from attacks and break-ins.

Red Hat Linux expert Mohammed Kabir starts by showing you how to

tune the kernel and filesystems and optimize network services, from

speeding up Web servers to boosting the performance of Samba. He then

explains how to secure your Red Hat Linux system, offering hands-on

techniques for network and Internet security as well as in-depth coverage

of Linux firewalls and virtual private networks.

Complete with security utilities and ready-to-run scripts on CD-ROM,

this official Red Hat Linux guide is an indispensable resource.

9 780764 547546

54999

ISBN 0-7645-4754-2

7 85555 04474 6

Proven Red Hat Linux Performance and Security Solutions

CD-ROM FEATURES

Scripts from the book

Security tools, including

cgichk.pl, gShield, IP

Filter, John the Ripper,

L i d s, L S O F, N e s s u s,

Netcat, Ngrep, Nmap,

OpenSSH, OpenSSL,

Postfix, SAINT trial

version, SARA, Snort,

Swatch, tcpdump,

Tripwire Open Source

Linux Edition, Vetescan,

and Whisker

Plus a searchable

e-version of the book

Reviewed and Approved by the Experts at Red Hat $49.99 USA

$74.99 Canada

£39.99 UK incl.VAT

Shelving Category

Networking

Reader Level

Intermediate to

Advanced

www.redhat.com

www.hungryminds.com

Cover design by

Michael J. Freeland

Cover photo ©

H. Armstrong Roberts

®

®

® ®

™

™

4754-2 cover 10/25/01 1:37 PM Page 1

Red Hat Linux

Security and

Optimization

Mohammed J. Kabir

Hungry Minds, Inc.

New York, NY ● Indianapolis, IN ● Cleveland, OH

014754-2 FM.F 11/5/01 9:03 AM Page i

Trademarks: are trademarks or registered trademarks of Hungry Minds, Inc. All other trademarks are the

property of their respective owners. Hungry Minds, Inc., is not associated with any product or vendor

mentioned in this book.

LIMIT OF LIABILITY/DISCLAIMER OF WARRANTY: THE PUBLISHER AND AUTHOR HAVE USED THEIR

BEST EFFORTS IN PREPARING THIS BOOK. THE PUBLISHER AND AUTHOR MAKE NO REPRESENTATIONS

OR WARRANTIES WITH RESPECT TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THIS

BOOK AND SPECIFICALLY DISCLAIM ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS

FOR A PARTICULAR PURPOSE. THERE ARE NO WARRANTIES WHICH EXTEND BEYOND THE

DESCRIPTIONS CONTAINED IN THIS PARAGRAPH. NO WARRANTY MAY BE CREATED OR EXTENDED BY

SALES REPRESENTATIVES OR WRITTEN SALES MATERIALS. THE ACCURACY AND COMPLETENESS OF

THE INFORMATION PROVIDED HEREIN AND THE OPINIONS STATED HEREIN ARE NOT GUARANTEED OR

WARRANTED TO PRODUCE ANY PARTICULAR RESULTS, AND THE ADVICE AND STRATEGIES

CONTAINED HEREIN MAY NOT BE SUITABLE FOR EVERY INDIVIDUAL. NEITHER THE PUBLISHER NOR

AUTHOR SHALL BE LIABLE FOR ANY LOSS OF PROFIT OR ANY OTHER COMMERCIAL DAMAGES,

INCLUDING BUT NOT LIMITED TO SPECIAL, INCIDENTAL, CONSEQUENTIAL, OR OTHER DAMAGES.

FULFILLMENT OF EACH COUPON OFFER IS THE SOLE RESPONSIBILITY OF THE OFFEROR.

Red Hat Linux Security and Optimization

Published by

Hungry Minds, Inc.

909 Third Avenue

New York, NY 10022

www.hungryminds.com

Copyright © 2002 Hungry Minds, Inc. All rights

reserved. No part of this book, including interior

design, cover design, and icons, may be reproduced

or transmitted in any form, by any means

(electronic, photocopying, recording, or otherwise)

without the prior written permission of the publisher.

Library of Congress Control Number: 2001092938

ISBN: 0-7645-4754-2

Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

1B/SX/RR/QR/IN

Distributed in the United States by Hungry Minds,

Inc.

Distributed by CDG Books Canada Inc. for Canada;

by Transworld Publishers Limited in the United

Kingdom; by IDG Norge Books for Norway; by IDG

Sweden Books for Sweden; by IDG Books Australia

Publishing Corporation Pty. Ltd. for Australia and

New Zealand; by TransQuest Publishers Pte Ltd. for

Singapore, Malaysia, Thailand, Indonesia, and Hong

Kong; by Gotop Information Inc. for Taiwan; by ICG

Muse, Inc. for Japan; by Intersoft for South Africa;

by Eyrolles for France; by International Thomson

Publishing for Germany, Austria, and Switzerland;

by Distribuidora Cuspide for Argentina; by LR

International for Brazil; by Galileo Libros for Chile;

by Ediciones ZETA S.C.R. Ltda. for Peru; by WS

Computer Publishing Corporation, Inc., for the

Philippines; by Contemporanea de Ediciones for

Venezuela; by Express Computer Distributors for the

Caribbean and West Indies; by Micronesia Media

Distributor, Inc. for Micronesia; by Chips

Computadoras S.A. de C.V. for Mexico; by Editorial

Norma de Panama S.A. for Panama; by American

Bookshops for Finland.

For general information on Hungry Minds’ products

and services please contact our Customer Care

department within the U.S. at 800-762-2974, outside

the U.S. at 317-572-3993 or fax 317-572-4002.

For sales inquiries and reseller information,

including discounts, premium and bulk quantity

sales, and foreign-language translations, please

contact our Customer Care department at

800-434-3422, fax 317-572-4002 or write to Hungry

Minds, Inc., Attn: Customer Care Department, 10475

Crosspoint Boulevard, Indianapolis, IN 46256.

For information on licensing foreign or domestic

rights, please contact our Sub-Rights Customer Care

department at 212-884-5000.

For information on using Hungry Minds’ products

and services in the classroom or for ordering

examination copies, please contact our Educational

Sales department at 800-434-2086 or fax

317-572-4005.

For press review copies, author interviews, or other

publicity information, please contact our Public

Relations department at 317-572-3168 or fax

317-572-4168.

For authorization to photocopy items for corporate,

personal, or educational use, please contact

Copyright Clearance Center, 222 Rosewood Drive,

Danvers, MA 01923, or fax 978-750-4470.

is a trademark of Hungry Minds, Inc.

014754-2 FM.F 11/5/01 9:03 AM Page ii

About the Author

Mohammed Kabir is the founder and CEO of Evoknow, Inc. His company specializes

in open-source solutions and customer relationship management software develop￾ment. When he is not busy managing software projects or writing books, he enjoys

traveling around the world. Kabir studied computer engineering at California State

University, Sacramento. He is also the author of Red Hat Linux Server and Apache

Server Bible. He can be reached at [email protected].

Credits

ACQUISITIONS EDITOR

Debra Williams Cauley

PROJECT EDITOR

Pat O’Brien

TECHNICAL EDITORS

Matthew Hayden

Sandra “Sam” Moore

COPY EDITORS

Barry Childs-Helton

Stephanie Provines

EDITORIAL MANAGER

Kyle Looper

RED HAT PRESS LIAISON

Lorien Golaski, Red Hat

Communications Manager

SENIOR VICE PRESIDENT, TECHNICAL

PUBLISHING

Richard Swadley

VICE PRESIDENT AND PUBLISHER

Mary Bednarek

PROJECT COORDINATOR

Maridee Ennis

GRAPHICS AND PRODUCTION

SPECIALISTS

Karl Brandt

Stephanie Jumper

Laurie Petrone

Brian Torwelle

Erin Zeltner

QUALITY CONTROL TECHNICIANS

Laura Albert

Andy Hollandbeck

Carl Pierce

PERMISSIONS EDITOR

Carmen Krikorian

MEDIA DEVELOPMENT SPECIALIST

Marisa Pearman

PROOFREADING AND INDEXING

TECHBOOKS Production Services

014754-2 FM.F 11/5/01 9:03 AM Page iii

014754-2 FM.F 11/5/01 9:03 AM Page iv

This book is dedicated to my wife, who proofs my writing, checks my facts,

and writes my dedications.

014754-2 FM.F 11/5/01 9:03 AM Page v

Preface

This book is focused on two major aspects of Red Hat Linux system administration:

performance tuning and security. The tuning solutions discussed in this book will

help your Red Hat Linux system to have better performance. At the same time, the

practical security solutions discussed in the second half of the book will allow you

to enhance your system security a great deal. If you are looking for time saving,

practical solutions to performance and security issues, read on!

How This Book is Organized

The book has five parts, plus several appendixes.

Part I: System Performance

This part of the book explains the basics of measuring system performance, cus￾tomizing your Red Hat Linux kernel to tune the operating system, tuning your

hard disks, and journaling your filesystem to increase file system reliability and

robustness.

Part II: Network and Service Performance

This part of the book explains how to tune your important network services,

including Apache Web server, Sendmail and postfix mail servers, and Samba and

NFS file and printer sharing services.

Part III: System Security

This part of the book covers how to secure your system using kernel-based Linux

Intrusion Detection System (LIDS) and Libsafe buffer overflow protection mecha￾nisms. Once you have learned to secure your Red Hat Linux kernel, you can secure

your file system using various tools. After securing the kernel and the file system,

you can secure user access to your system using such tools as Pluggable

Authentication Module (PAM), Open Source Secure Socket Layer (OpenSSL), Secure

Remote Password (SRP), and xinetd.

Part IV: Network Service Security

This part of the book shows how to secure your Apache Web server, BIND DNS

server, Sendmail and postfix SMTP server, POP3 mail server, Wu-FTPD and

ProFTPD FTP servers, and Samba and NFS servers.

vi

014754-2 FM.F 11/5/01 9:03 AM Page vi

Part V: Firewalls

This part of the book shows to create packet filtering firewall using iptables, how to

create virtual private networks, and how to use SSL based tunnels to secure access

to system and services. Finally, you will be introduced to an wide array of security

tools such as security assessment (audit) tools, port scanners, log monitoring and

analysis tools, CGI scanners, password crackers, intrusion detection tools, packet

filter tools, and various other security administration utilities.

Appendixes

These elements include important references for Linux network users, plus an

explanation of the attached CD-ROM.

Conventions of This Book

You don’t have to learn any new conventions to read this book. Just remember the

usual rules:

◆ When you are asked to enter a command, you need press the Enter or the

Return key after you type the command at your command prompt.

◆ A monospaced font is used to denote configuration or code segment.

◆ Text in italic needs to be replaced with relevant information.

Watch for these icons that occasionally highlight paragraphs.

The Note icon indicates that something needs a bit more explanation.

The Tip icon tells you something that is likely to save you some time and

effort.

Preface vii

014754-2 FM.F 11/5/01 9:03 AM Page vii

The Caution icon makes you aware of a potential danger.

The cross-reference icon tells you that you can find additional information

in another chapter.

Tell Us What You Think of This Book

Both Hungry Minds and I want to know what you think of this book. Give us your

feedback. If you are interested in communicating with me directly, send e-mail

messages to [email protected]. I will do my best to respond promptly.

viii Red Hat Linux Security and Optimization

014754-2 FM.F 11/5/01 9:03 AM Page viii

Acknowledgments

While writing this book, I often needed to consult with many developers whose

tools I covered in this book. I want to specially thank a few such developers who

have generously helped me present some of their great work.

Huagang Xie is the creator and chief developer of the LIDS project. Special

thanks to him for responding to my email queries and also providing me with a

great deal of information on the topic.

Timothy K. Tsai, Navjot Singh, and Arash Baratloo are the three members of the

Libsafe team who greatly helped in presenting the Libsafe information. Very special

thanks to Tim for taking the time to promptly respond to my emails and providing

me with a great deal of information on the topic.

I thank both the Red Hat Press and Hungry Minds teams who made this book a

reality. It is impossible to list everyone involved but I must mention the following

kind individuals.

Debra Williams Cauley provided me with this book opportunity and made sure I

saw it through to the end. Thanks, Debra.

Terri Varveris, the acquisitions editor, took over in Debra’s absence. She made

sure I had all the help needed to get this done. Thanks, Terri.

Pat O’Brien, the project development editor, kept this project going. I don’t know

how I could have done this book without his generous help and suggestions every

step of the way. Thanks, Pat.

Matt Hayden, the technical reviewer, provided numerous technical suggestions,

tips, and tricks — many of which have been incorporated in the book. Thanks, Matt.

Sheila Kabir, my wife, had to put up with many long work hours during the few

months it took to write this book. Thank you, sweetheart.

ix

014754-2 FM.F 11/5/01 9:03 AM Page ix

014754-2 FM.F 11/5/01 9:03 AM Page x

Contents at a Glance

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . ix

Part I System Performance

Chapter 1 Performance Basics . . . . . . . . . . . . . . . . . . . . . . . . . 3

Chapter 2 Kernel Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Chapter 3 Filesystem Tuning . . . . . . . . . . . . . . . . . . . . . . . . . 39

Part II Network and Service Performance

Chapter 4 Network Performance . . . . . . . . . . . . . . . . . . . . . . 75

Chapter 5 Web Server Performance . . . . . . . . . . . . . . . . . . . . 89

Chapter 6 E-Mail Server Performance . . . . . . . . . . . . . . . . . 125

Chapter 7 NFS and Samba Server Performance . . . . . . . . . . 141

Part III System Security

Chapter 8 Kernel Security . . . . . . . . . . . . . . . . . . . . . . . . . . 155

Chapter 9 Securing Files and Filesystems . . . . . . . . . . . . . . 179

Chapter 10 PAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241

Chapter 11 OpenSSL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

Chapter 12 Shadow Passwords and OpenSSH . . . . . . . . . . . . 277

Chapter 13 Secure Remote Passwords . . . . . . . . . . . . . . . . . . 313

Chapter 14 xinetd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323

Part IV Network Service Security

Chapter 15 Web Server Security . . . . . . . . . . . . . . . . . . . . . . 351

Chapter 16 DNS Server Security . . . . . . . . . . . . . . . . . . . . . . 399

Chapter 17 E-Mail Server Security . . . . . . . . . . . . . . . . . . . . 415

Chapter 18 FTP Server Security . . . . . . . . . . . . . . . . . . . . . . . 443

Chapter 19 Samba and NFS Server Security . . . . . . . . . . . . . 473

014754-2 FM.F 11/5/01 9:03 AM Page xi

Part V Firewalls

Chapter 20 Firewalls, VPNs, and SSL Tunnels . . . . . . . . . . . . 491

Chapter 21 Firewall Security Tools . . . . . . . . . . . . . . . . . . . . 541

Appendix A IP Network Address Classification . . . . . . . . . . . . 589

Appendix B Common Linux Commands . . . . . . . . . . . . . . . . . 593

Appendix C Internet Resources . . . . . . . . . . . . . . . . . . . . . . . . 655

Appendix D Dealing with Compromised Systems . . . . . . . . . . 661

Appendix E What’s On the CD-ROM? . . . . . . . . . . . . . . . . . . . 665

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669

End-User License Agreement . . . . . . . . . . . . . . . . 691

014754-2 FM.F 11/5/01 9:03 AM Page xii

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . ix

Part I System Performance

Chapter 1 Performance Basics . . . . . . . . . . . . . . . . . . . . . . . . . 3

Measuring System Performance . . . . . . . . . . . . . . . . . . . . . . . 4

Monitoring system performance with ps . . . . . . . . . . . . . . . . . . . . . 4

Tracking system activity with top . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Checking memory and I/O with vmstat . . . . . . . . . . . . . . . . . . . . . . 8

Running Vtad to analyze your system . . . . . . . . . . . . . . . . . . . . . . 9

Chapter 2 Kernel Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Compiling and Installing a Custom Kernel . . . . . . . . . . . . . . 11

Downloading kernel source code (latest distribution) . . . . . . . . . . 11

Creating the /usr/src/linux symbolic link . . . . . . . . . . . . . . . . . . . 12

Selecting a kernel-configuration method . . . . . . . . . . . . . . . . . . . 13

Using menuconfig . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Compiling the kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Booting the new kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Running Demanding Applications . . . . . . . . . . . . . . . . . . . . 35

Chapter 3 Filesystem Tuning . . . . . . . . . . . . . . . . . . . . . . . . . 39

Tuning your hard disks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Tuning ext2 Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

Changing the block size of the ext2 filesystem . . . . . . . . . . . . . . . 44

Using e2fsprogs to tune ext2 filesystem . . . . . . . . . . . . . . . . . . . . 45

Using a Journaling Filesystem . . . . . . . . . . . . . . . . . . . . . . . 48

Compiling and installing ReiserFS . . . . . . . . . . . . . . . . . . . . . . . . 50

Using ReiserFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Benchmarking ReiserFS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

Managing Logical Volumes . . . . . . . . . . . . . . . . . . . . . . . . . 54

Compiling and installing the LVM module for kernel . . . . . . . . . . 54

Creating a logical volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Adding a new disk or partition to a logical volume . . . . . . . . . . . 62

Removing a disk or partition from a volume group . . . . . . . . . . . 65

014754-2 FM.F 11/5/01 9:03 AM Page xiii

Using RAID, SAN, or Storage Appliances . . . . . . . . . . . . . . 66

Using Linux Software RAID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

Using Hardware RAID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Using Storage-Area Networks (SANs) . . . . . . . . . . . . . . . . . . . . . . 67

Using Storage Appliances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

Using a RAM-Based Filesystem . . . . . . . . . . . . . . . . . . . . . . 68

Part II Network and Service Performance

Chapter 4 Network Performance . . . . . . . . . . . . . . . . . . . . . . 75

Tuning an Ethernet LAN or WAN . . . . . . . . . . . . . . . . . . . . 75

Using network segmentation technique for performance . . . . . . . 77

Using switches in place of hubs . . . . . . . . . . . . . . . . . . . . . . . . . . 80

Using fast Ethernet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Using a network backbone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

Understanding and controlling network traffic flow . . . . . . . . . . . 83

Balancing the traffic load using the DNS server . . . . . . . . . . . . . . 85

IP Accounting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

IP accounting on a Linux network gateway . . . . . . . . . . . . . . . . . 86

Chapter 5 Web Server Performance . . . . . . . . . . . . . . . . . . . . 89

Compiling a Lean and Mean Apache . . . . . . . . . . . . . . . . . . 89

Tuning Apache Configuration . . . . . . . . . . . . . . . . . . . . . . . 95

Controlling Apache processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

Controlling system resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

Using dynamic modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

Speeding Up Static Web Pages . . . . . . . . . . . . . . . . . . . . . . 103

Reducing disk I/O for faster static page delivery . . . . . . . . . . . . . 104

Using Kernel HTTP daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Speeding Up Web Applications . . . . . . . . . . . . . . . . . . . . . 105

Using mod_perl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

Using FastCGI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

Installing and configuring FastCGI module for Apache . . . . . . . . 115

Using Java servlets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

Using Squid proxy-caching server . . . . . . . . . . . . . . . . . . . . . . . . 118

Chapter 6 E-Mail Server Performance . . . . . . . . . . . . . . . . . 125

Choosing Your MTA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

Tuning Sendmail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126

Controlling the maximum size of messages . . . . . . . . . . . . . . . . 127

Caching Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

Controlling simultaneous connections . . . . . . . . . . . . . . . . . . . . 130

Limiting the load placed by Sendmail . . . . . . . . . . . . . . . . . . . . . 131

xiv Contents

014754-2 FM.F 11/5/01 9:03 AM Page xiv