Ethernet

www.it-ebooks.info

www.it-ebooks.info

Charles E. Spurgeon and Joann Zimmerman

SECOND EDITION

Ethernet: The Definitive Guide

www.it-ebooks.info

Ethernet: The Definitive Guide, Second Edition

by Charles E. Spurgeon and Joann Zimmerman

Copyright © 2014 Charles E. Spurgeon and Joann Zimmerman. All rights reserved.

Printed in the United States of America.

Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.

O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are

also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/

institutional sales department: 800-998-9938 or [email protected].

Editor: Meghan Blanchette

Production Editor: Nicole Shelby

Copyeditor: Rachel Head

Proofreader: Jasmine Kwityn

Indexer: Judy McConville

Cover Designer: Randy Comer

Interior Designer: David Futato

Illustrator: Rebecca Demarest

March 2014: Second Edition

Revision History for the Second Edition:

2014-03-11: First release

See http://oreilly.com/catalog/errata.csp?isbn=9781449361846 for release details.

Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly

Media, Inc. Ethernet: The Definitive Guide, the image of an octopus, and related trade dress are trademarks

of O’Reilly Media, Inc.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as

trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark

claim, the designations have been printed in caps or initial caps.

While every precaution has been taken in the preparation of this book, the publisher and authors assume

no responsibility for errors or omissions, or for damages resulting from the use of the information contained

herein.

ISBN: 978-1-449-36184-6

[LSI]

www.it-ebooks.info

Table of Contents

Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv

Part I. Introduction to Ethernet

1. The Evolution of Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

History of Ethernet 3

The Aloha Network 4

The Invention of Ethernet 4

Reinventing Ethernet 6

Reinventing Ethernet for Twisted-Pair Media 7

Reinventing Ethernet for 100 Mb/s 8

Reinventing Ethernet for 1000 Mb/s 8

Reinventing Ethernet for 10, 40, and 100 Gb/s 9

Reinventing Ethernet for New Capabilities 9

Ethernet Switches 10

The Future of Ethernet 10

2. IEEE Ethernet Standards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Evolution of the Ethernet Standard 11

Ethernet Media Standards 13

IEEE Supplements 13

Draft Standards 14

Differences Between DIX and IEEE Standards 15

Organization of IEEE Standards 16

The Seven Layers of OSI 16

IEEE Sublayers Within the OSI Model 18

Levels of Compliance 20

The Effect of Standards Compliance 20

IEEE Media System Identifiers 21

iii

www.it-ebooks.info

10 Megabit per Second (Mb/s) Media Systems 21

100 Mb/s Media Systems 23

1000 Mb/s Media Systems 24

10 Gb/s Media Systems 24

40 Gb/s Media Systems 25

100 Gb/s Media Systems 25

3. The Ethernet System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

The Four Basic Elements of Ethernet 27

The Ethernet Frame 28

The Media Access Control Protocol 30

Hardware 33

Network Protocols and Ethernet 36

Best-Effort Delivery 36

Design of Network Protocols 37

Protocol Encapsulation 38

Internet Protocol and Ethernet Addresses 39

Looking Ahead 41

4. The Ethernet Frame and Full-Duplex Mode. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

The Ethernet Frame 44

Preamble 46

Destination Address 46

Source Address 48

Q-Tag 48

Envelope Prefix and Suffix 49

Type or Length Field 50

Data Field 51

FCS Field 52

End of Frame Detection 52

Full-Duplex Media Access Control 53

Full-Duplex Operation 53

Effects of Full-Duplex Operation 55

Configuring Full-Duplex Operation 55

Full-Duplex Media Support 56

Full-Duplex Media Segment Distances 56

Ethernet Flow Control 57

PAUSE Operation 58

High-Level Protocols and the Ethernet Frame 60

Multiplexing Data in Frames 60

IEEE Logical Link Control 61

iv | Table of Contents

www.it-ebooks.info

The LLC Sub-Network Access Protocol 62

5. Auto-Negotiation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Development of Auto-Negotiation 64

Auto-Negotiation for Fiber Optic Media 65

Basic Concepts of Auto-Negotiation 65

Auto-Negotiation Signaling 67

FLP Burst Operation 68

Auto-Negotiation Operation 72

Parallel Detection 74

Operation of Parallel Detection 74

Parallel Detection and Duplex Mismatch 75

Auto-Negotiation Completion Timing 76

Auto-Negotiation and Cabling Issues 77

Limiting Ethernet Speed over Category 3 Cable 78

Cable Issues and Gigabit Ethernet Auto-Negotiation 79

Crossover Cables and Auto-Negotiation 79

1000BASE-X Auto-Negotiation 80

Auto-Negotiation Commands 81

Disabling Auto-Negotiation 82

Auto-Negotiation Debugging 82

General Debugging Information 83

Debugging Tools and Commands 84

Developing a Link Configuration Policy 86

Link Configuration Policies for Enterprise Networks 87

Issues with Manual Configuration 87

6. Power Over Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

Power Over Ethernet Standards 89

Goals of the PoE Standard 90

Devices That May Be Powered Over Ethernet 91

Benefits of PoE 91

PoE Device Roles 92

PoE Type Parameters 93

PoE Operation 94

Power Detection 94

Power Classification 95

Link Power Maintenance 97

Power Fault Monitoring 97

PoE and Cable Pairs 98

PoE and Ethernet Cabling 101

PoE Power Management 102

Table of Contents | v

www.it-ebooks.info

PoE Power Requirements 102

PoE Port Management 103

PoE Monitoring and Power Policing 103

Vendor Extensions to the Standard 105

Cisco UPoE 105

Microsemi EEPoE 105

Power over HDBaseT (POH) 105

Part II. Ethernet Media Systems

7. Ethernet Media Signaling and Energy Efficient Ethernet. . . . . . . . . . . . . . . . . . . . . . . . 109

Media Independent Interfaces 111

Ethernet PHY Components 112

Ethernet Signal Encoding 113

Baseband Signaling Issues 113

Baseline Wander and Signal Encoding 114

Advanced Signaling Techniques 115

Ethernet Interface 115

Higher-Speed Ethernet Interfaces 116

Energy Efficient Ethernet 117

IEEE EEE Standard 118

EEE Operation 119

Impact of EEE Operation on Latency 121

EEE Power Savings 122

8. 10 Mb/s Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

10BASE-T Media System 125

10BASE-T Ethernet Interface 126

Signal Polarity and Polarity Reversal 126

10BASE-T Signal Encoding 126

10BASE-T Media Components 128

Connecting a Station to 10BASE-T Ethernet 130

10BASE-T Link Integrity Test 130

10BASE-T Configuration Guidelines 131

Fiber Optic Media Systems (10BASE-F) 131

Old and New Fiber Link Segments 132

10BASE-FL Signaling Components 133

10BASE-FL Ethernet Interface 133

10BASE-FL Signal Encoding 133

10BASE-FL Media Components 134

10BASE-FL Fiber Optic Characteristics 134

vi | Table of Contents

www.it-ebooks.info

Alternate 10BASE-FL Fiber Optic Cables 135

Fiber Optic Connectors 135

Connecting a 10BASE-FL Ethernet Segment 136

10BASE-FL Link Integrity Test 136

10BASE-FL Configuration Guidelines 137

9. 100 Mb/s Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

100BASE-X Media Systems 139

Fast Ethernet Twisted-Pair Media Systems (100BASE-TX) 140

100BASE-TX Signaling Components 140

100BASE-TX Ethernet Interface 140

100BASE-TX Signal Encoding 141

100BASE-TX Media Components 145

100BASE-TX Link Integrity Test 146

100BASE-TX Configuration Guidelines 146

Fast Ethernet Fiber Optic Media Systems (100BASE-FX) 146

100BASE-FX Signaling Components 147

100BASE-FX Signal Encoding 147

100BASE-FX Media Components 147

100BASE-FX Fiber Optic Characteristics 150

Alternate 100BASE-FX Fiber Optic Cables 150

100BASE-FX Link Integrity Test 150

100BASE-FX Configuration Guidelines 150

Long Fiber Segments 151

10. Gigabit Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153

Gigabit Ethernet Twisted-Pair Media Systems (1000BASE-T) 153

1000BASE-T Signaling Components 154

1000BASE-T Signal Encoding 155

1000BASE-T Media Components 158

1000BASE-T Link Integrity Test 159

1000BASE-T Configuration Guidelines 159

Gigabit Ethernet Fiber Optic Media Systems (1000BASE-X) 159

1000BASE-X Signaling Components 160

1000BASE-X Link Integrity Test 160

1000BASE-X Signal Encoding 160

1000BASE-X Media Components 161

1000BASE-X Fiber Optic Specifications 164

1000BASE-SX Loss Budget 164

1000BASE-LX Loss Budget 166

1000BASE-LX/LH Long Haul Loss Budget 166

1000BASE-SX and 1000BASE-LX Configuration Guidelines 167

Table of Contents | vii

www.it-ebooks.info

Differential Mode Delay 167

Mode-Conditioning Patch Cord 168

11. 10 Gigabit Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

10 Gigabit Standards Architecture 172

10 Gigabit Ethernet Twisted-Pair Media Systems (10GBASE-T) 173

10GBASE-T Signaling Components 174

10GBASE-T Signal Encoding 175

10GBASE-T Media Components 177

10GBASE-T Link Integrity Test 180

10GBASE-T Configuration Guidelines 180

10GBASE-T Short-Reach Mode 181

10GBASE-T Signal Latency 181

10 Gigabit Ethernet Short Copper Cable Media Systems (10GBASE-CX4) 182

10 Gigabit Ethernet Short Copper Direct Attach Cable Media Systems

(10GSFP+Cu) 183

10GSFP+Cu Signaling Components 184

10GSFP+Cu Signal Encoding 186

10GSFP+Cu Link Integrity Test 187

10GSFP+Cu Configuration Guidelines 187

10 Gigabit Ethernet Fiber Optic Media Systems 187

10 Gigabit LAN PHYs 189

10 Gb/s Fiber Optic Media Specifications 191

10 Gigabit WAN PHYs 193

12. 40 Gigabit Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

Architecture of 40 Gb/s Ethernet 196

PCS Lanes 196

40 Gigabit Ethernet Twisted-Pair Media Systems (40GBASE-T) 201

40 Gigabit Ethernet Short Copper Cable Media Systems (40GBASE-CR4) 202

40GBASE-CR4 Signaling Components 204

40GBASE-CR4 Signal Encoding 205

QSFP+ Connectors and Multiple 10 Gb/s Interfaces 206

40 Gigabit Ethernet Fiber Optic Media Systems 207

40 Gb/s Fiber Optic Media Specifications 211

40GBASE-LR4 Wavelengths 213

40 Gigabit Extended Range 214

13. 100 Gigabit Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

Architecture of 100 Gb/s Ethernet 215

PCS Lanes 216

100 Gigabit Ethernet Twisted-Pair Media Systems 219

viii | Table of Contents

www.it-ebooks.info

100 Gigabit Ethernet Short Copper Cable Media Systems (100GBASE-CR10) 219

100GBASE-CR10 Signal Encoding 222

100 Gigabit Ethernet Fiber Optic Media Systems 223

Cisco CPAK Module for 100 Gigabit Ethernet 224

100 Gb/s Fiber Optic Media Specifications 225

14. 400 Gigabit Ethernet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231

400 Gb/s Ethernet Study Group 232

400 Gb/s Standardization 232

Proposed 400 Gb/s Operation 232

Part III. Building an Ethernet System

15. Structured Cabling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

Structured Cabling Systems 238

The ANSI/TIA/EIA Cabling Standards 239

Solving the Problems of Proprietary Cabling Systems 239

ISO and TIA Standards 240

The ANSI/TIA Structured Cabling Documents 240

Elements of the Structured Cabling Standards 241

Star Topology 242

Twisted-Pair Categories 244

Minimum Cabling Recommendation 246

Ethernet and the Category System 246

Horizontal Cabling 247

Horizontal Channel and Basic Link 248

Cabling and Component Specifications 249

Category 5 and 5e Cable Testing and Mitigation 250

Cable Administration 250

Identifying Cables and Components 251

Class 1 Labeling Scheme 251

Documenting the Cabling System 253

Building the Cabling System 253

Cabling System Challenges 254

16. Twisted-Pair Cables and Connectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

Horizontal Cable Segment Components 257

Twisted-Pair Cables 258

Twisted-Pair Cable Signal Crosstalk 260

Twisted-Pair Cable Construction 260

Twisted-Pair Installation Practices 263

Table of Contents | ix

www.it-ebooks.info

Eight-Position (RJ45-Style) Jack Connectors 264

Four-Pair Wiring Schemes 265

Tip and Ring 265

Color Codes 265

Wiring Sequence 266

Modular Patch Panels 269

Work Area Outlets 270

Twisted-Pair Patch Cables 270

Twisted-Pair Patch Cable Quality 270

Telephone-Grade Patch Cables 271

Twisted-Pair Ethernet and Telephone Signals 272

Equipment Cables 272

50-Pin Connectors and 25-Pair Cables 273

25-Pair Cable Harmonica Connectors 273

Building a Twisted-Pair Patch Cable 273

Installing an RJ45 Plug 274

Ethernet Signal Crossover 278

10BASE-T and 100BASE-T Crossover Cables 279

Four-Pair Crossover Cables 280

Auto-Negotiation and MDIX Failures 281

Identifying a Crossover Cable 282

17. Fiber Optic Cables and Connectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283

Fiber Optic Cable 283

Fiber Optic Core Diameters 284

Fiber Optic Modes 285

Fiber Optic Bandwidth 286

Fiber Optic Loss Budget 287

Fiber Optic Connectors 289

ST Connectors 289

SC Connectors 290

LC Connectors 290

MPO Connectors 291

Building Fiber Optic Cables 292

Fiber Optic Color Codes 293

Signal Crossover in Fiber Optic Systems 294

Signal Crossover in MPO Cables 294

Part IV. Ethernet Switches and Network Design

18. Ethernet Switches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299

x | Table of Contents

www.it-ebooks.info

Basic Switch Functions 300

Bridges and Switches 300

What Is a Switch? 301

Operation of Ethernet Switches 301

Address Learning 303

Traffic Filtering 305

Frame Flooding 306

Broadcast and Multicast Traffic 306

Combining Switches 308

Forwarding Loops 308

The Spanning Tree Protocol 309

Switch Performance Issues 316

Packet Forwarding Performance 316

Switch Port Memory 317

Switch CPU and RAM 317

Switch Specifications 317

Basic Switch Features 321

Switch Management 321

Packet Mirror Ports 322

Switch Traffic Filters 322

Virtual LANs 323

802.1Q Multiple Spanning Tree Protocol 325

Quality of Service (QoS) 326

19. Network Design with Ethernet Switches. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327

Advantages of Switches in Network Designs 327

Improved Network Performance 327

Switch Hierarchy and Uplink Speeds 329

Uplink Speeds and Traffic Congestion 330

Multiple Conversations 331

Switch Traffic Bottlenecks 332

Hierarchical Network Design 333

Network Resiliency with Switches 336

Spanning Tree and Network Resiliency 337

Routers 339

Operation and Use of Routers 339

Routers or Bridges? 340

Special-Purpose Switches 342

Multilayer Switches 342

Access Switches 343

Stacking Switches 343

Industrial Ethernet Switches 344

Table of Contents | xi

www.it-ebooks.info

Wireless Access Point Switches 344

Internet Service Provider Switches 345

Metro Ethernet 345

Data Center Switches 346

Advanced Switch Features 349

Traffic Flow Monitoring 349

sFlow and NetFlow 349

Power over Ethernet 350

Part V. Performance and Troubleshooting

20. Ethernet Performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353

Performance of an Ethernet Channel 354

Performance of Half-Duplex Ethernet Channels 354

Persistent Myths About Half-Duplex Ethernet Performance 354

Simulations of Half-Duplex Ethernet Channel Performance 357

Measuring Ethernet Performance 360

Measurement Time Scale 361

Data Throughput Versus Bandwidth 364

Network Design for Best Performance 367

Switches and Network Bandwidth 367

Growth of Network Bandwidth 368

Changes in Application Requirements 368

Designing for the Future 369

21. Network Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371

Reliable Network Design 372

Network Documentation 373

Equipment Manuals 374

System Monitoring and Baselines 374

The Troubleshooting Model 375

Fault Detection 377

Gathering Information 378

Fault Isolation 378

Determining the Network Path 379

Duplicating the Symptom 379

Binary Search Isolation 380

Troubleshooting Twisted-Pair Systems 381

Twisted-Pair Troubleshooting Tools 381

Common Twisted-Pair Problems 381

Troubleshooting Fiber Optic Systems 385

xii | Table of Contents

www.it-ebooks.info

Fiber Optic Troubleshooting Tools 385

Common Fiber Optic Problems 386

Data Link Troubleshooting 387

Collecting Data Link Information 387

Collecting Information with Probes 388

Network-Layer Troubleshooting 388

Part VI. Appendixes

A. Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393

B. Half-Duplex Operation with CSMA/CD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403

C. External Transceivers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427

Glossary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 449

Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463

Table of Contents | xiii

www.it-ebooks.info