Building and integrating vitual private networks with openswan

Building and Integrating Virtual

Private Networks with

Openswan

Learn from the developers of Openswan how to build

industry-standard, military-grade VPNs and connect

them with Windows, Mac OS X, and other VPN vendors

Paul Wouters

Ken Bantoft

BIRMINGHAM - MUMBAI

Building and Integrating Virtual Private Networks with

Openswan

Copyright © 2006 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or

transmitted in any form or by any means, without the prior written permission of the publisher,

except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the

information presented. However, the information contained in this book is sold without warranty,

either express or implied. Neither the authors, Packt Publishing, nor its dealers or distributors will

be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all the companies and

products mentioned in this book by the appropriate use of capitals. However, Packt Publishing

cannot guarantee the accuracy of this information.

First published: February 2006

Production Reference: 1010206

Published by Packt Publishing Ltd.

32 Lincoln Road

Olton

Birmingham, B27 6PA, UK.

ISBN 1-904811-25-6

www.packtpub.com

Cover Design by www.visionwt.com

Credits

Authors

Paul Wouters

Ken Bantoft

Reviewers

Michael Stelluti

Tuomo Soini

Nate Carlson

James Eaton-Lee

Technical Editor

Richard Deeson

Editorial Manager

Dipali Chittar

Development Editor

Louay Fatoohi

Indexer

Abhishek Shirodkar

Proofreader

Chris Smith

Production Coordinator

Manjiri Nadkarni

Cover Designer

Helen Wood

About the Authors

Paul Wouters has been involved with Linux networking and security since he co-founded the

Dutch ISP Xtended Internet back in 1996, where he started working with FreeS/WAN IPsec in

1999 and with DNSSEC for the .nl domain in 2001.

He has been writing since 1997, when his first article about network security was published in

Linux Journal in 1997. Since then, he has written mostly for the Dutch spin-off of the German c't

magazine, focusing on Linux, networking, and the impact of the digital world on society.

He has presented papers at SANS, OSA, CCC, HAL, BlackHat, and Defcon, and several other

smaller conferences.

He started working for Xelerance in 2003, focusing on IPsec, DNSSEC, Radius, and training delivery.

Over a year ago, we wrote a proposal for an Openswan book. Without knowing about this

proposal, Louay Fatoohi of Packt Publishing asked us if we were interested in publishing

just such a book. We are very happy with the result of that collaboration.

We would like to thank everyone who is or has been part of the Linux IPsec and Openswan

communities, without whom neither Openswan nor this book would have been possible.

Many thanks to John Gilmore for founding the FreeS/WAN Project, and to XS4ALL for

hosting it. Many people contributed to FreeS/WAN, but we would like to especially thank

Hugh Daniel, Michael Richardson, Hugh Redelmeier, and Richard Guy Briggs.

The FreeS/WAN and Openswan community contributed some important features. Thanks

to Andreas Steffen of StrongSec for the X.509 patches, JuanJo Ciarlante for the original

ALG patches that included AES, Mattieu Lafon of Arkoon Systems for the NAT-Traversal

patches, and Hendrik Nordstrom of MARA Systems for the Aggressive Mode patches.

Further thanks are due to Rene Mayrhofer of Debian and Robert-Jan Cornelissen of

Xtended Internet as early adopters of Openswan. Xtended Internet also graciously hosted

the Openswan servers for two years.

We are especially grateful to Herbert Xu for his tremendous work on integrating Openswan

with the Linux 2.6 NETKEY stack, and Michael Richardson for maintaining and

enhancing tcpdump.

Thanks also to Jacco de Leeuw for his excellent work on documenting L2TP, and Nate

Carlson for his elaborate X.509 configuration guide. They have invested a large amount of

time in helping the community with Openswan configuration.

Everyone knows how important a cute logo is, but the logo that Nana Manojlovic

spontaneously gave us surpasses even the penguin. Thank you Nana!

And of course, thanks to all the Linux distributions that have included Openswan in their

packages. You have truly caused the widespread use and acceptance of Openswan.

Over the course of a year, quite a few people have helped to create this book. Many thanks

to Louay Fatoohi and Richard Deeson of Packt Publishing. This book would have been filled

with errors, had it not been for our reviewers, Tuomo Soini, Nate Carlson, and James Eaton￾Lee. Extra praise goes to Mike Stelluti who, without ever having touched a Linux computer,

went through the book verifying every single command, which included setting up and

testing entire X.509, L2TP, and UML setups from scratch. And a special thanks goes to

Michael Richardson for writing the section on debugging Openswan using tcpdump.

Ken Bantoft started programming in 1988, and successfully avoided it as a full-time job until

2002. Before that, he opted instead to focus on Unix, Networking, and Linux integration.

Beginning at OLS2002, he started working alongside the FreeS/WAN project, integrating various

patches into his own fork of its code—Super FreeS/WAN, which is now known as Openswan.

He currently lives in Oakville, ON, Canada, with his wife Van, two cats, and too many computers.

Ken started working for Xelerance in 2003 where he works mostly on IPsec, BGP/OSPF, Asterisk,

LDAP, and Radius.

I'd like to thank: My father, who put a computer in front of me 20 years ago, and who has

supported my digital addiction for all those years; My wife Van, who puts up with the large

amount of hardware in the basement, and the power bills it generates; Kyle Schustyk, with

whom I set up my first IPsec tunnel; Jim Alton, Alex Bichuch, and Rob Rankin who kept

me busy building VPNs for various people; Michael Richardson—without his ROT13-

encrypted party invitation I'd have never starting hacking IPsec code; Sam Sgro, with

whom a bet started Super FreeS/WAN, which in turn begat Openswan; D. Hugh

Reidelmier, who still answers any C question I have.

About the Reviewers

Michael Stelluti is completing his studies in Computer Science and has been an intern at

Xelerance Corporation since 2005. As part of the Xelerance support group, Michael reproduces

client environments in the labs and also moderates the Openswan mailing lists. To relax, he enjoys

watching Battlestar Gallactica with a pint of Guinness well in hand. Michael currently resides in

Kelowna, British-Columbia, in Canada.

Nate Carlson is currently a full time systems administrator for Internet Broadcasting, and

also does occasional Linux consulting on the side. He's been using IPSec under Linux since

the early FreeS/WAN days, and has written a popular guide on using Windows XP in a

RoadWarrior configuration.

He lives near Minneapolis, Minnesota with his wonderful wife Tiffany. He can be reached via his

website, www.natecarlson.com

James Eaton-Lee works as an Infrastructure Security Consultant for a firm whose clients range

from small businesses with a handful of employees to multinational banks. He has formerly

worked for an Internet Service Provider and at a call center, as well as providing independent

consultancy in the areas of forensics and security.

James has extensive experience of traditional and IP telephony, as well as how these technologies

can be integrated into existing IT infrastructure. He has been involved in a variety of work in his

present role, ranging from simple IT and infrastructure work for small clients to security work

across infrastructure comprising thousands of servers for a large bank. He is a strong advocate of

the relevancy of open-source and free software, and—wherever appropriate—uses it for himself

and his clients.

Table of Contents

Preface 1

Chapter 1: Introduction 5

The Need for Cryptography 5

Privacy 5

Security 6

A History of the Internet 6

Holding the Internet Together 7

The Creation of ICANN 7

ICANN Bypassed 8

The Root Name Servers 8

Running the Top-Level Domains 8

History of Internet Engineering 9

The Internet Engineering Task Force (IETF) 9

RFCs—Requests For Comments 10

IETF and Crypto 11

The War on Crypto 12

Dual Use 12

Public Cryptography 12

The Escrowed Encryption Standard 13

Export Laws 13

The Summer of '97 14

The EFF DES Cracker 14

Echelon 14

The End of the Export Restrictions 15

Free Software 15

The GPL 15

Free as in Verifiable 16

The Open Source Movement 16

The History of Openswan 17

IETF Troubles over DNS 17

Super FreeS/WAN 17

The Arrival of Openswan 18

NETKEY 18

Table of Contents

Further Reading 19

Using Openswan 19

Copyright and License Conditions 20

Writing and Contributing Code 20

Legality of Using Openswan 21

International Agreements 21

International Law and Hosting Openswan 22

Unrecognized International Claims 22

Patent Law 23

Expired and Bogus Patents 23

Useful Legal Links 24

Summary 25

Chapter 2: Practical Overview of the IPsec Protocol 27

A Very Brief Overview of Cryptography 27

Valid Packet Rewriting 28

Ciphers 28

DES, 3DES, and AES 29

Algorithms 29

Uniqueness 30

Public-Key Algorithms 30

Exchanging Public Keys 30

Digital Signatures 30

Diffie-Hellman Key Exchange 30

Avoiding the Man in the Middle 31

Session Keys 31

Crypto Requirements for IPsec 32

IPsec: A Suite of Protocols 32

Kernel Mode: Packet Handling 32

Authentication Header (AH) 33

Encapsulated Security Payload (ESP) 34

Transport and Tunnel Mode 34

Choosing the IPsec Mode and Type 35

The Kernel State 35

Encryption Details 36

Manual Keying 36

Final Note on Protocols and Ports 37

Usermode: Handling the Trust Relationships 37

The IKE Protocol 37

Phase 1: Creating the ISAKMP SA 37

ii

Table of Contents

Phase 2: Quick Mode 39

The NAT Problem 41

Summary 44

Chapter 3: Building and Installing Openswan 45

Linux Distributions 45

Red Hat 46

Debian 46

SuSE 46

Slackware 47

Gentoo 47

Linux 'Router' Distributions 48

Deciding on the Userland 48

Pluto 48

Racoon 49

Isakmpd 50

More Reasons to Pick Pluto 50

Choosing the Kernel IPsec Stack 50

KLIPS, the Openswan Stack 50

ipsecX Interfaces 51

First Packet Caching 51

Path MTU Discovery 51

KLIPS' Downside 52

NETKEY, the 2.6 IPsec Stack 53

The USAGI / SuSE IPsec Stack 53

Making the Choice 54

GPL Compliance and KLIPS 54

Binary Installation of the Openswan Userland 54

Checking for Old Versions 55

Installing the Binary Package for Openswan 55

Building from Source 56

Using RPM-based Distributions 57

Rebuilding the Openswan Userland 58

Building src.rpm from Scratch 58

Openswan Options 59

Building the Openswan Userland from Source 59

Downloading the Source Code 59

Configuring the Userland Tools 59

Optional Features 60

Compile Flags 61

iii

Table of Contents

File Path Options 62

Obscure Pluto Options 62

Compiling and Installing 63

Binary Installation of KLIPS 63

Building KLIPS from Source 64

Kernel Prerequisites 64

Identifying your Kernel's Abilities 65

Using Both KLIPS and NETKEY 65

The Kernel Build Options 65

Required Kernel Options 66

Desired Options 66

NETKEY Stack Options 66

KLIPS Stack Options 67

L2TP Options 68

Patching the Kernel 69

NAT-Traversal Patch 69

KLIPS Compile Shortcut 69

Activating KLIPS 70

Determining the Stack in Use 70

Building KLIPS into the Linux Kernel Source Tree 71

Building a Standard Kernel 71

NAT Traversal 72

Patching KLIPS into the Linux Kernel 72

Verifying the Installation 74

Summary 74

Chapter 4: Configuring IPsec 75

Manual versus Automatic 75

PSK versus RSA 76

Pitfalls of Debugging IPsec 76

Pre-Flight Check 77

The ipsec verify Command 77

NAT and Masquerading 78

Checking External Commands 79

Opportunistic Encryption 79

The ipsec livetest Command 79

Configuration of Openswan 80

The ipsec.conf File 81

iv

Table of Contents

Host-to-Host Tunnel 82

Left and Right 82

The type Options 83

The auto Option 83

The rsasigkey Options 84

Bringing Up the IPsec Tunnels 84

Listing IPsec Connections 85

Testing the IPsec Tunnel 85

Connecting Subnets Through an IPsec Connection 86

Testing Subnet Connections 87

Testing Properly 87

Encrypting the Host and the Network Behind It 88

Employing Advanced Routing 88

Creating More Tunnels 88

Avoiding Duplication 89

The Also Keyword 89

KLIPS and the ipsecX Interfaces 89

Pre-Shared Keys (PSKs) 90

Proper Secrets 90

Dynamic IP Addresses 90

Hostnames 91

Roadwarriors 91

Multiple Roadwarrior Connections 92

Dynamic IP and PSKs 92

PSK and NAT 93

Mixing PSK and RSA 93

Connection Management 93

Subnet Extrusion 94

NAT Traversal 96

Deprecated Syntax 97

Confirming a Functional NAT-T 97

Dead Peer Detection 98

DPD Works Both Ways 99

Configuring DPD 99

Buggy Cisco Routers 100

Ciphers and Algorithms 101

Using ike= to Specify Phase 1 Parameters 101

Using esp= to Specify Phase 2 Parameters 102

v

Table of Contents

Defaults and Strictness 102

Unsupported Ciphers and Algorithms 103

Aggressive Mode 103

XAUTH 104

XAUTH Gateway (Server Side) 105

XAUTH Client (Supplicant Side) 105

Fine Tuning 106

Perfect Forward Secrecy 106

Rekeying 106

Key Rollover 107

Summary 107

Chapter 5: X.509 Certificates 109

X.509 Certificates Explained 109

X.509 Objects 110

X.509 Packing 112

Types of Certificates 112

Passphrases, PIN Codes, and Interactivity 113

IKE and Certificates 113

Using the Certificate DN as ID for Openswan 113

Generating Certificates with OpenSSL 114

Setting the Time 114

Configuring OpenSSL 114

Be Consistent with All Certificates 115

OpenSSL Commands for Common Certificate Actions 115

Configuring Apache for IPsec X.509 Files 116

Creating X.509-based Connections 117

Using a Certificate Authority 120

Using Multiple CAs 121

Sending and Receiving Certificate Information 122

Creating your own CA using OpenSSL 122

Creating Host Certificates with Your Own CA 123

Host Certificates for Microsoft Windows (PKCS#12) 124

Certificate Revocation 125

Dynamic CRL Fetching 126

Configuring CRL 127

Online Certificate Status Protocol (OCSP) 128

Summary 129

vi

Table of Contents

Chapter 6: Opportunistic Encryption 131

History of Opportunistic Encryption 132

Trusting Third Parties 132

Trusting the DNS? 133

OE in a Nutshell 133

An OE Security Gateway 134

DNS Key Records 135

Forward and Reverse Zones 135

The OE DNS Records 136

Different Types of OE 136

Policy Groups 137

Internal States 138

Configuring OE 138

Configuring Policies 139

Full OE or Initiate-Only 139

Generating Correct DNS Records 139

Name Server Updates 140

Verifying Your OE Setup 141

Testing Your OE Setup 142

The trap eroute 143

The pass eroute 143

The hold eroute 143

Manipulating OE Connections Manually 143

Advanced OE Setups 144

Caveats 144

Summary 145

Chapter 7: Dealing with Firewalls 147

Where to Firewall? 147

Allowing IPsec Traffic 148

NAT and IPsec Passthrough 149

Configuring the Firewall on the Openswan Host 150

Firewalling and KLIPS 151

Firewalling and NETKEY 151

Packet Size 152

Summary 153

vii