In preparation of your CCNP exam, we want to make sure we cover the various concepts that we could see on your Cisco CCNP exam. So to assist you, below we will discuss on of the more difficult CCNP concepts; Configuring EtherChannel Switch-to-Switch Connections on Catalyst 4000, 5000, an. As you progress through your CCNP exam studies, I am sure with repetition you will find this topic becomes easier. So even though it may be a difficult concept and confusing at first, keep at it as no one said getting your Cisco certification would be easy!

Introduction

EtherChannel allows multiple physical Fast Ethernet or Gigabit Ethernet links to be combined into one logical channel. A logical channel allows load sharing of traffic among the links in the channel as well as redundancy in the event that one or more links in the channel should fail. EtherChannel can be used to interconnect LAN switches, routers, servers, and clients with unshielded twisted pair (UTP) wiring or single-mode and multimode fiber.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

• A console cable1 suitable for the Supervisor Engine in the switch
• Two Catalyst 5505 switches in a lab environment with cleared configurations2
• A Fast Ethernet module capable of EtherChannel in each Catalyst 5505
• Four RJ-45 Ethernet crossover cables3 to connect the EtherChannel

1. For more information, refer to the Identifying a Rollover Cable, Straight-through, and Crossover Cable section of the document Connecting a Terminal to the Console Port on Catalyst Switches.


2. The clear config all command was entered on the switch to ensure that it has a default configuration.


3. For a pinout of an Ethernet crossover cable, see Appendix A.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Background Information

EtherChannel is an easy way to aggregate bandwidth between critical networking devices. On the Catalyst 5500/5000, a channel can be created from two ports, making it a 200 Mbps link (400 Mbps, full-duplex), or four ports, making it a 400 Mbps link (800 Mbps, full-duplex). Some cards and platforms also support Gigabit EtherChannel and have the ability to use from two to eight ports in an EtherChannel. The concept is the same no matter the speeds or number of links that are involved. Normally, the Spanning Tree Protocol (STP) considers these redundant links between two devices to be loops and causes the redundant links to be in blocking mode. This effectively makes the links inactive (providing only backup capabilities if the main link fails). When using a Catalyst OS (CatOS) software version 3.1(1) or later, STP treats the channel as one large link, so all the ports in the channel can be active at the same time.

This document takes you through the steps to configure EtherChannel between two Catalyst 5500/5000 switches and shows you the results of the commands as they are issued. Catalyst 4500/4000 and 6500/6000 switches that run CatOS can be used in the scenarios presented in this document to obtain the same results. For the Catalyst 2900XL and Catalyst 1900/2820, the command syntax differs, but the EtherChannel concepts are the same. For EtherChannel guidelines and configuration information for the Catalyst 6500/6000 series switches that run Cisco IOS® System Software, refer to the document:

• Sample Configuration: EtherChannel Between Catalyst Switches Running CatOS and Native IOS

For an overview and comparison of the Catalyst 6500 Cat0S and Cisco IOS Software platforms, refer to the document:

• Comparison of the Cisco Catalyst and Cisco IOS Operating Systems for the Cisco Catalyst 6500 Series Switch

EtherChannel may be configured manually by executing the appropriate commands, or it may be configured automatically by having the switch negotiate the channel with the other side using Port Aggregation Protocol (PAgP). It is best to use the PAgP desirable mode to configure EtherChannel whenever possible because manually configuring EtherChannel sometimes creates complications. This document provides examples of configuring EtherChannel manually and examples of configuring EtherChannel using PAgP. Also included is how to troubleshoot EtherChannel and how to use trunking with EtherChannel. In this document, the terms EtherChannel, Fast EtherChannel, Gigabit EtherChannel, or channel all refer to EtherChannel.

Network Diagram

The network setup shown in this section illustrates the test environment.

After the configuration of the switches was cleared with the clear config all command, the prompt was changed with the set system name command. An IP address and mask were assigned to the switch for management purposes using the set interface sc0 172.16.84.6 255.255.255.0 command for Switch A and the set interface sc0 172.16.84.17 255.255.255.0 command for Switch B. A default gateway was assigned to both switches with the set ip route default 172.16.84.1 command.

The switch configurations were cleared to start from the default conditions. The switches were given names for identification from the prompt on the command line. To ping between the switches for testing, the IP addresses were assigned. The default gateway was not used.

Many of the commands display more output than is needed for this discussion. Extraneous output is suppressed in this document.

Manually Configure EtherChannel

Step-by-Step

Complete these steps to manually configure EtherChannel:

The show version command displays the software version the switch is running. The show module command lists which modules are installed in the switch.

Note: The show port capabilities command is available in CatOS software versions 4.x and later. If you have a software version earlier than 4.x, you must skip this step. Not every Fast Ethernet module supports EtherChannel. Some of the original EtherChannel modules have "Fast EtherChannel" printed on the bottom left corner of the module (as you face it in the switch), which tells you that the feature is supported. But this convention was abandoned on later modules. The modules in this test do not have "Fast EtherChannel" printed on them, but they do support the feature.

A port that does not support EtherChannel looks like this:

Before connecting the cables, the port status is:

After connecting the cables between the two switches, the status is:

Since the switch configurations were cleared before starting this test, the ports are in their default conditions. They are all in VLAN 1, and their speed and duplex are set to auto. After connecting the cables, they negotiate to a speed of 100 Mbps and full-duplex. The status is connected. You are now able to ping the other switch.

In your network, you may want to to set the speeds manually to 100 Mbps and full-duplex instead of relying on autonegotiation because you probably want your ports to always run at the fastest speed. For a discussion of autonegotiation, refer to the document:

• Configuring and Troubleshooting Ethernet 10/100/1000Mb Half/Full Duplex Auto-Negotiation.

This is an important point that is covered in more detail in the Troubleshoot EtherChannel section. If the command to set up EtherChannel does not work, it is usually because the ports involved in the channel have configurations that differ from each other. This includes the ports on the other side of the link as well as the local ports. In this case, since the switch configurations were cleared before this test, the ports are in their default conditions. They are all in VLAN 1, their speed and duplex are set to auto, and all spanning tree parameters for each port are set to be the same. You saw from the output above that, after connecting the cables, the ports negotiate to a speed of 100 Mbps and full-duplex. Since STP runs for each VLAN, it is easier to simply configure the channel and respond to error messages than to attempt to check every STP field for consistency for each port and VLAN in the channel.

On the Catalyst 5500/5000, only certain ports can be put together into a channel. These restrictive dependencies do not apply to all platforms. The ports in a channel on a Catalyst 5500/5000 must be contiguous. The show port capabilities command, issued for port 2/1, shows the possible combinations:

Notice that this port can be a part of a group of two (2/1-2) or part of a group of four (2/1-4). An Ethernet Bundling Controller (EBC) on the module causes these configuration limitations. Here, show port capabilities is issued for another port:

This port can be grouped into a group of two ports (2/3-4) or into a group of four ports (2/1-4).

Note: Depending on the hardware, there may be additional restrictions. On certain modules (WS-X5201 and WS-X5203), you cannot form an EtherChannel with the last two ports in a port group unless the first two ports in the group already form an EtherChannel. A port group is a group of ports that are allowed to form an EtherChannel. (In the example above, 2/1-4 is a port group.)

For example, if you are creating separate EtherChannels with only two ports in a channel, you cannot assign ports 2/3-4 to a channel until you have first configured ports 2/1-2 to a channel. This is true only for the modules that have this restriction. Similarly, before you configure ports 2/6-7, you must configure ports 2/5-6. This restriction does not occur on the modules used for this document (WS-X5225R and WSX5234).

Since you are configuring a group of four ports (2/1-4), this is within the approved grouping. You are not able to assign a group of four to ports 2/3-6. This is a group of contiguous ports, but they do not start on the approved boundary, as shown by the show port capabilities command. (Valid groups are ports 1-4, 5-8, 9-12, 13-16, 17-20, and 21-24.)

To create the channel manually, use the set port channel mod/port on command for each switch. It is best to turn the ports off on one side of the channel using the set port disable command before turning EtherChannel on manually. This avoids possible problems with STP during the configuration process.

STP can shut down some ports (with a port status of errdisable) if one side is configured as a channel before the other side can be configured as a channel. Because of this possibility, it is much easier to create EtherChannels using PAgP, which is covered in the Using PAgP to Configure EtherChannel (Recommended) section of this document. To avoid this situation when configuring EtherChannel manually, you disable the ports on Switch A, configure the channel on Switch A, configure the channel on Switch B, then reenable the ports on Switch A.

Now, STP does not generate errors and shut down the ports.

Note: In this case, ports 2/1 to 2/4 are configured for Etherchannel with a single command. If you configure the etherchannel for every port independently without using the port range, remember to mention the same admin-group for all the ports that need to be part of same Etherchannel. If the admin-group is not specified, then every port will belong to different Etherchannel groups and the desired Etherchannel bundle will never be formed.

Notice that the channel mode has been set to on, but the status of the ports is disabled (because you disabled them earlier). The channel is not operational at this point, but it becomes operational when the ports are enabled.

Because Switch A ports were (temporarily) disabled, Switch B ports no longer have a connection.

This message is displayed on the Switch B console when Switch A ports are disabled:

Notice that the channel mode for Switch B is on, but the status of the ports is notconnect. This is the case because Switch A ports are still disabled.

Verify the EtherChannel Configuration

To verify that the channel is set up properly, issue the show port channel command.

If you have the output of a show port channel command from your Cisco device, you can use the Output Interpreter Tool ( registered customers only) , which allows you to view an analysis of show command output.

STP is shown to treat the ports as one logical port in the show spantree command. In this output, when the port is listed as 2/1-4, it means that STP is treating ports 2/1, 2/2, 2/3 and 2/4 as one port.

If you have the output of a show spantree command from your Cisco device, you can use the Output Interpreter Tool ( registered customers only) , which allows you to view an analysis of show command output.

EtherChannel can be implemented with different ways of distributing the traffic across the ports in a channel.

The EtherChannel specification does not dictate how the traffic should be distributed across the links in a channel. The Catalyst 5500/5000 uses the last bit or the last two bits (depending on how many links are in the channel) of the source and destination MAC addresses in the frame to determine which port in the channel to use. You should see a similar amount of traffic on each of the ports in the channel, assuming that traffic is generated by a normal distribution of MAC addresses on one side of the channel or the other. To verify that traffic is going over all the ports in the channel, you can use the show mac command. If your ports were active before configuring EtherChannel, you can reset the traffic counters to 0 with the clear counters command. The traffic values then represent how EtherChannel has distributed the traffic.

In this test environment, a real-world distribution is not achieved because there are no workstations, servers, or routers generating traffic. The only devices generating traffic are the switches themselves. Pings were issued from Switch A to Switch B; the unicast traffic is using the first port in the channel. (See the output below.) The receive information (Rcv-Unicast) in this case shows how Switch B distributed the traffic across the channel to Switch A. Also in the output, the transmit information (Xmit-Unicast) shows how Switch A distributed the traffic across the channel to Switch B. You also see that a small amount of switch-generated multicast traffic (Dynamic Inter-Switch Link Protocol [ISL], Cisco Discovery Protocol [CDP]) goes out all four ports. The broadcast packets are Address Resolution Protocol (ARP) queries (for the default gateway which does not exist in this lab). If you had workstations sending packets through the switch to a destination on the other side of the channel, you would expect to see traffic going over each of the four links in the channel. You can monitor the packet distribution in your network using the show mac command.

If you have the output of a show mac command from your Cisco device, you can use the Output Interpreter Tool ( registered customers only) , which allows you to view an analysis of show command output.

Using PAgP to Configure EtherChannel (Recommended)

PAgP facilitates the automatic creation of EtherChannel links by exchanging packets between channel-capable ports. The protocol learns the capabilities of port groups dynamically and informs the neighboring ports.

After PAgP identifies correctly paired channel-capable links, it groups the ports into a channel. The channel is then added to the spanning tree as a single bridge port. A given outbound broadcast or multicast packet is transmitted out one port in the channel only, not out every port in the channel. In addition, outbound broadcast and multicast packets transmitted on one port in a channel are blocked from returning on any other port of the channel.

There are four user-configurable channel modes: on, off, auto, and desirable. PAgP packets are exchanged only between ports in auto and desirable mode. Ports configured in on or off mode do not exchange PAgP packets.

For switches to which you want to form an EtherChannel, it is best to have both switches set to desirable mode.

This gives the most robust behavior if one side or the other encounters error situations or is reset. The default mode of the channel is auto.

Both the auto and desirable modes allow ports to negotiate with connected ports to determine if they can form a channel. The determination is based on criteria such as port speed, trunking state, and native VLAN.

Ports can form an EtherChannel when they are in different channel modes as long as the modes are compatible.

This list provides examples:

• A port in desirable mode can successfully form an EtherChannel with another port that is in desirable or auto mode.
• A port in auto mode can form an EtherChannel with another port in desirable mode.
• A port in auto mode cannot form an EtherChannel with another port that is also in auto mode, since neither port initiates negotiation.
• A port in on mode can form a channel only with a port in on mode because ports in on mode do not exchange PAgP packets.
• A port in off mode cannot form a channel with any port.

When using EtherChannel, if this message (or a similar syslog message) is displayed, it indicates a mismatch of EtherChannel modes on the connected ports:

Correct the configuration and reenable the ports by issuing the set port enable command. Valid EtherChannel configurations include:

The next table provides a summary of all the possible channelling mode scenarios. Some of these combinations may cause STP to put the ports on the channelling side in errdisable state. (In other words, some of the combinations shut down the ports on the channelling side.)

The channel from the previous example (Step 6b in the section Manually Configure EtherChannel) is turned off using this command on Switch A and Switch B:

The default channel mode for a port that is able to channel is auto. To verify this, issue this command:

The show port channel port command also shows that the ports currently are not channelling. This is another way to verify the channel state:

It is simple to make the channel work with PAgP. At this point, both switches are set to auto mode, which means that they channel if a connected port sends a PAgP request to channel. Setting Switch A to desirable causes Switch A to send PAgP packets to the other switch, asking it to channel.

To view the channel, issue this command:

Since Switch B is in auto mode, it responds to the PAgP packets and creates a channel with Switch A.

Note: It is best to set both sides of the channel to desirable so that both sides try to initiate the channel if one side drops out. Setting the EtherChannel ports on Switch B to desirable mode, even though the channel is currently active and in auto mode, poses no problem. The command is:

Note: In this case, all ports 2/1 to 2/4 are configured for Etherchannel with a single command. If you configure the etherchannel for every port independently without using the port range, remember to mention the same admin-group for all the ports that need to be part of same Etherchannel. If the admin-group is not specified, then every port will belong to different Etherchannel groups and the desired Etherchannel bundle will never be formed.

If Switch A drops out for some reason, or if new hardware replaces Switch A, Switch B tries to reestablish the channel. If the new equipment cannot channel, Switch B treats its ports 2/1-4 as normal nonchannelling ports.

This is one of the benefits of using the desirable mode. If the channel was configured using the PAgP on mode and one side of the connection has an error of some kind or a reset, it could cause an errdisable state (shutdown) on the other side. With PAgP set in desirable mode on each side, the channel stabilizes and renegotiates the EtherChannel connection.

Silent/Non-Silent Mode

When dealing with fiber connections, it is possible that, even if a receive (Rx) transceiver dies, the transmit (Tx) transceiver on the other end is still up. During a similar scenario, packets can get black holed.

It is important for the switch that is transmitting to remove this port from the EtherChannel bundle. To do so on the Catalyst 5500/5000, you set PAgP in non-silent mode. This means that if the Rx does not receive traffic, the port is not put into the channel. However, this is not enough because this detection happens only when the channel is formed.

To prevent the black holing of traffic when the channel is already formed, this occurs: PAgP detects that the Rx port is not receiving any traffic, so it resets the Tx transceiver of the port that is not receiving. It is reset for 1.6 seconds so that the switch on the other end also resets the port. The faulty port does not join the channel anymore because no traffic is received on that port.

On the Catalyst 5500/5000, it is recommended (default) that non-silent mode is set on fiber strands and silent mode is set on copper strands. The reason for this is that, on fiber connection on the Catalyst 5500/5000, the negotiation is usually not available, so there is no way to detect the problem at a physical layer.

Default PAgP Settings on the Catalyst 4500/4000 and 5500/5000

By default, PAgP is auto for a plug-and-play implementation. It is best to disable PAgP manually from the ports where there is no need to have it.

By default, the silent mode is on. (Non-silent is acceptable as well.) However, since a port can be connected to a device that does not send traffic (for example, a sniffer), it is more general to have silent enabled.

Recommendations

• Use the non-silent keyword when you are connecting to a device that transmits bridge protocol data units (BPDUs) or other traffic. Use this keyword with the auto or desirable mode. PAgP non-silent adds an extra level of link state detection by listening for BPDUs or other traffic to determine if the link is functioning properly. This adds a form of unidirectional link detection capability not available when you use the default silent PAgP mode.
• Use the silent keyword when you are connecting to a silent partner (a device that is not generating BPDUs or other traffic). An example of a silent partner is a traffic generator that is not transmitting packets. Use this keyword with the auto or desirable mode. If you do not specify silent or non-silent, silent is assumed.
• The silent mode does not disable the PAgP capability of detecting unidirectional links. However, if you are configuring a channel, non-silent prevents a unidirectional port from even joining the link.
• PAgP configuration (the set port channel {desirable | auto} command) is safer than non-PAgP configuration (the set port channel on command) because it provides protection for unidirectional links. It also avoids misconfigurations that can arise from having ports channeling on one side of the link and not on the other side.
• Refer to Understanding and Configuring the Unidirectional Link Detection Protocol Feature for more information on UniDirectional Link Detection (UDLD).

Trunking and EtherChannel

EtherChannel is independent of trunking. You can turn trunking on or you can leave trunking off. Also, you can turn trunking on for all the ports before creating the channel, or you can turn it on after creating the channel (as in this example). As far as EtherChannel is concerned, it does not matter; trunking and EtherChannel are completely separate features. What does matter is that all the ports involved are in the same mode: either they are all trunking before you configure the channel, or they are all not trunking before you configure the channel.

All the ports must be in the same trunking state before creating the channel.

Once a channel is formed, whatever is changed on one port is also changed for the other ports in the channel.

The modules used in this test bed can do ISL or IEEE 802.1Q trunking. By default, the modules are set to auto trunking and negotiate mode. This means that they trunk if the other side asks them to trunk, and they negotiate whether to use the ISL or 802.1Q method for trunking. If they are not asked to trunk, they work as normal nontrunking ports.

There are a number of different ways to turn on trunking. For this example, Switch A is set to desirable. Switch A is already set to negotiate. The combination of desirable/negotiate causes Switch A to ask Switch B to trunk and to negotiate the type of trunking to perform (ISL or 802.1Q). Since Switch B defaults to autonegotiate, Switch B responds to the Switch A request. These are the results:

The trunk mode was set to desirable. The result was that trunking mode was negotiated with the neighbor switch, and they decided on ISL (n-isl). The current status now is trunking. This output shows what happened on Switch B because of the command issued on Switch A:

Notice that all four ports (2/1-4) became trunking, even though you only specifically changed one port (2/1) to desirable. This is an example of how changing one port in the channel affects all the ports.

Troubleshoot EtherChannel

The challenges for EtherChannel can be divided into two main areas: troubleshooting during the configuration phase and troubleshooting during the execution phase. Configuration errors usually occur because of mismatched parameters on the ports involved (for example, different speeds, different duplex, or different STP port values) However, you can also generate errors during the configuration by setting the channel on one side to on and waiting too long before configuring the channel on the other side. This causes STP loops which generate an error and shut down the port.

When an error is encountered while configuring EtherChannel, be sure to check the status of the ports after correcting the EtherChannel error situation. If the port status is errdisable, it means that the ports have been shut down by the software. They do not come on again until you issue the set port enable command.

Note: If the port status becomes errdisable, you must specifically enable the ports using the set port enable command for the ports to become active. Currently, you can correct all the EtherChannel issues, but the ports do not come up or form a channel until they are enabled again. Later versions of the operating system may periodically check to determine if errdisable ports should be enabled.

These tests are covered in this section. For the tests, trunking and EtherChannel are turned off:

• Mismatched Parameters
• Waiting Too Long Before You Configure the Other Side
• Correcting errdisable State
• Showing What Happens When a Link Breaks and Is Restored

Mismatched Parameters

Here is an example of mismatched parameters. Port 2/4 is set in VLAN 2 while the other ports are still in VLAN 1. To create a new VLAN, you must assign a VLAN Trunk Protocol (VTP) domain for the switch and then create the VLAN.

I hope you found this article to be of use and it helps you prepare for your Cisco CCNP certification. I am sure you will quickly find out that hands-on real world experience is the best way to cement the CCNP concepts in your head to help you pass your CCNP exam!