Troubleshooting Guide
Page 3
... and Safety Information i-xxx Hardware Installation i-xxx Cisco Fabric Manager i-xxxi Command-Line Interface i-xxxi Troubleshooting and Reference i-xxxi Installation and Configuration Note i-xxxi Obtaining Documentation, Obtaining Support, and Security Guidelines i-xxxii Troubleshooting Overview 1-1 Overview of the Troubleshooting Process 1-1 Best Practices 1-2 Troubleshooting Basics 1-2 General Steps 1-2 Gathering Information Using Common Fabric Manager Tools and CLI Commands 1-3 Common Fabric Manager Tools 1-3 Common CLI Commands 1-4 Verifying Basic Connectivity 1-4 Verifying SAN Element...
... and Safety Information i-xxx Hardware Installation i-xxx Cisco Fabric Manager i-xxxi Command-Line Interface i-xxxi Troubleshooting and Reference i-xxxi Installation and Configuration Note i-xxxi Obtaining Documentation, Obtaining Support, and Security Guidelines i-xxxii Troubleshooting Overview 1-1 Overview of the Troubleshooting Process 1-1 Best Practices 1-2 Troubleshooting Basics 1-2 General Steps 1-2 Gathering Information Using Common Fabric Manager Tools and CLI Commands 1-3 Common Fabric Manager Tools 1-3 Common CLI Commands 1-4 Verifying Basic Connectivity 1-4 Verifying SAN Element...
Troubleshooting Guide
Page 26
..., "Troubleshooting Users and Roles" xxvi Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Appendix B, "Troubleshooting Tools and Methodology" Added information about troubleshooting 3.1(2) nondisruptive upgrades on Cisco MDS 9124 Fabric Switch SAN Device Virtualization SNMP test traps Digital Certificates iSCSI Load Balancing Mixed Generation Hardware AAA Call Home FC-SP, port security, fabric binding IP Access Lists Users and Roles Description Changed in Release Where Documented Updated FICON port numbering...
..., "Troubleshooting Users and Roles" xxvi Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 Appendix B, "Troubleshooting Tools and Methodology" Added information about troubleshooting 3.1(2) nondisruptive upgrades on Cisco MDS 9124 Fabric Switch SAN Device Virtualization SNMP test traps Digital Certificates iSCSI Load Balancing Mixed Generation Hardware AAA Call Home FC-SP, port security, fabric binding IP Access Lists Users and Roles Description Changed in Release Where Documented Updated FICON port numbering...
Troubleshooting Guide
Page 30
... IBM TotalStorage SAN Volume Controller Storage Software for the Cisco MDS 9000 Family, refer to mdsfeedback-doc@cisco.com Related Documentation The documentation set for the Cisco MDS 9000 Family Hardware Installation • Cisco MDS 9124 Multilayer Fabric Switch Quick Start Guide • Cisco MDS 9500 Series Hardware Installation Guide • Cisco MDS 9200 Series Hardware Installation Guide • Cisco MDS 9100 Series Hardware Installation Guide • Cisco MDS 9020 Fabric Switch Hardware Installation Guide Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxx OL-9285...
... IBM TotalStorage SAN Volume Controller Storage Software for the Cisco MDS 9000 Family, refer to mdsfeedback-doc@cisco.com Related Documentation The documentation set for the Cisco MDS 9000 Family Hardware Installation • Cisco MDS 9124 Multilayer Fabric Switch Quick Start Guide • Cisco MDS 9500 Series Hardware Installation Guide • Cisco MDS 9200 Series Hardware Installation Guide • Cisco MDS 9100 Series Hardware Installation Guide • Cisco MDS 9020 Fabric Switch Hardware Installation Guide Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x xxx OL-9285...
Troubleshooting Guide
Page 31
... Command-Line Interface • Cisco MDS 9000 Family Software Upgrade and Downgrade Guide • Cisco MDS 9000 Family Storage Services Module Software Installation and Upgrade Guide • Cisco MDS 9000 Family CLI Quick Configuration Guide • Cisco MDS 9000 Family CLI Configuration Guide • Cisco MDS 9000 Family Command Reference • Cisco MDS 9000 Family Quick Command Reference • Cisco MDS 9020 Fabric Switch Configuration Guide and Command Reference • Cisco MDS 9000 Family SAN Volume Controller Configuration Guide Troubleshooting and Reference • Cisco...
... Command-Line Interface • Cisco MDS 9000 Family Software Upgrade and Downgrade Guide • Cisco MDS 9000 Family Storage Services Module Software Installation and Upgrade Guide • Cisco MDS 9000 Family CLI Quick Configuration Guide • Cisco MDS 9000 Family CLI Configuration Guide • Cisco MDS 9000 Family Command Reference • Cisco MDS 9000 Family Quick Command Reference • Cisco MDS 9020 Fabric Switch Configuration Guide and Command Reference • Cisco MDS 9000 Family SAN Volume Controller Configuration Guide Troubleshooting and Reference • Cisco...
Troubleshooting Guide
Page 34
... can only access some, but not all your Cisco MDS switches. • Refer to be investigated. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 1-2 OL-9285-05 We recommend the following subsections, you should take to ensure the proper operation of the problem. For example, if a host can determine the paths you trying to determine the status of the problem. Use the answers...
... can only access some, but not all your Cisco MDS switches. • Refer to be investigated. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 1-2 OL-9285-05 We recommend the following subsections, you should take to ensure the proper operation of the problem. For example, if a host can determine the paths you trying to determine the status of the problem. Use the answers...
Troubleshooting Guide
Page 51
... just one command. - For example, the command verifies that a wrong image is not used inadvertently to run the setup script after issuing a write erase CLI command, you can view the progress of this question (the default is automatically checked, including the running kickstart and system images. - Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Guidelines Send documentation comments to mdsfeedback-doc@cisco.com • Copy the startup-config to cancel...
... just one command. - For example, the command verifies that a wrong image is not used inadvertently to run the setup script after issuing a write erase CLI command, you can view the progress of this question (the default is automatically checked, including the running kickstart and system images. - Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Guidelines Send documentation comments to mdsfeedback-doc@cisco.com • Copy the startup-config to cancel...
Troubleshooting Guide
Page 57
... the installation within the 10-second span, the command is already in progress. See the "Installing SAN-OS Software Using Fabric Manager" section on page 2-10 or the "Installing Cisco SAN-OS Software from the CLI" section on page 2-11. The install all command is complete before updating the system image. Restart the installation. Or, use the delete command. In Device Manager, choose Admin > Flash Files and delete unnecessary files. Wait until the upgrade is...
... the installation within the 10-second span, the command is already in progress. See the "Installing SAN-OS Software Using Fabric Manager" section on page 2-10 or the "Installing Cisco SAN-OS Software from the CLI" section on page 2-11. The install all command is complete before updating the system image. Restart the installation. Or, use the delete command. In Device Manager, choose Admin > Flash Files and delete unnecessary files. Wait until the upgrade is...
Troubleshooting Guide
Page 61
... switch reboot hangs. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-13 Please check the standby supervisor. -- If the configuration meets all guidelines when the install all modules (supervisor and switching) are upgraded. See the "Recovery Using BIOS Setup for single supervisor configuration. Interrupt the boot process and reconfigure the BIOS through the console port to load a new kickstart image that updates to view the upgraded supervisor module using the show module command...
... switch reboot hangs. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-13 Please check the standby supervisor. -- If the configuration meets all guidelines when the install all modules (supervisor and switching) are upgraded. See the "Recovery Using BIOS Setup for single supervisor configuration. Interrupt the boot process and reconfigure the BIOS through the console port to load a new kickstart image that updates to view the upgraded supervisor module using the show module command...
Troubleshooting Guide
Page 65
... using the exact uppercase characters and file extensions as it is displayed on your TFTP server. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-17 Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software System Reboots Send documentation comments to mdsfeedback-doc@cisco.com Step 4 Press the Tab key to TFTP. Enter the subnet mask for the switch, and press the Tab key. Enter the IP address of the TFTP server...
... using the exact uppercase characters and file extensions as it is displayed on your TFTP server. OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 2-17 Chapter 2 Troubleshooting Installs, Upgrades, and Reboots Troubleshooting Cisco SAN-OS Software System Reboots Send documentation comments to mdsfeedback-doc@cisco.com Step 4 Press the Tab key to TFTP. Enter the subnet mask for the switch, and press the Tab key. Enter the IP address of the TFTP server...
Troubleshooting Guide
Page 75
...: Interface fc1/1 is down (Link failure o r not-connected) switch# Enter the following command to check the syslog file to see which process restarted and why it restarted: switch# show processes The following example: Sep 10 23:31:31 dot-6 % LOG_SYSMGR-3-SERVICE_TERMINATED: Service "sensor" (PID 704) has finished with error code SYSMGR_EXITCODE_SY. switch# show log logfile | include error For information about the meaning of each message, refer...
...: Interface fc1/1 is down (Link failure o r not-connected) switch# Enter the following command to check the syslog file to see which process restarted and why it restarted: switch# show processes The following example: Sep 10 23:31:31 dot-6 % LOG_SYSMGR-3-SERVICE_TERMINATED: Service "sensor" (PID 704) has finished with error code SYSMGR_EXITCODE_SY. switch# show log logfile | include error For information about the meaning of each message, refer...
Troubleshooting Guide
Page 99
... mode, the Active LED is detected, the Link LED turns off. If one or more of the individual interfaces on the supervisor module front panel are green when the supervisor module completes initialization. If no signal is orange. If a Status LED is operational (online). Chapter 4 Troubleshooting Hardware Troubleshooting Power Supply Issues Send documentation comments to the supervisor module console port if the boot information and system banner are not displayed. Each Link LED flashes...
... mode, the Active LED is detected, the Link LED turns off. If one or more of the individual interfaces on the supervisor module front panel are green when the supervisor module completes initialization. If no signal is orange. If a Status LED is operational (online). Chapter 4 Troubleshooting Hardware Troubleshooting Power Supply Issues Send documentation comments to the supervisor module console port if the boot information and system banner are not displayed. Each Link LED flashes...
Troubleshooting Guide
Page 144
... 5 Troubleshooting Mixed Generation Hardware Send documentation comments to mdsfeedback-doc@cisco.com Step 3 Step 4 Use the show port index-allocation" for Generation-1 modules Check "show module command to display the reason why a module does not power up. Power-on . Do "copy running-config startup-config" to correct the problem. Table 5-9 Cannot Configure Port in Dedicated Mode Symptom Cannot configure a port in dedicated mode. Cannot Configure Port in Dedicated Mode Symptom Cannot configure port in dedicated mode. sw# show module recovery-steps command to...
... 5 Troubleshooting Mixed Generation Hardware Send documentation comments to mdsfeedback-doc@cisco.com Step 3 Step 4 Use the show port index-allocation" for Generation-1 modules Check "show module command to display the reason why a module does not power up. Power-on . Do "copy running-config startup-config" to correct the problem. Table 5-9 Cannot Configure Port in Dedicated Mode Symptom Cannot configure a port in dedicated mode. Cannot Configure Port in Dedicated Mode Symptom Cannot configure port in dedicated mode. sw# show module recovery-steps command to...
Troubleshooting Guide
Page 161
...: • IVR • NTP • DPVM • user roles • AAA server addresses, Radius and TACACS daemons • SFM • SDV • syslog • port-security • call home OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-1 Send documentation comments to mdsfeedback-doc@cisco.com 7 C H A P T E R Troubleshooting Cisco Fabric Services This chapter describes procedures used to maintain configuration synchronization across a fabric for automatic configuration synchronization in the...
...: • IVR • NTP • DPVM • user roles • AAA server addresses, Radius and TACACS daemons • SFM • SDV • syslog • port-security • call home OL-9285-05 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 7-1 Send documentation comments to mdsfeedback-doc@cisco.com 7 C H A P T E R Troubleshooting Cisco Fabric Services This chapter describes procedures used to maintain configuration synchronization across a fabric for automatic configuration synchronization in the...
Troubleshooting Guide
Page 210
... failed uplink that are logged in again, the login requests are logged out. Common Problems with NPV Chapter 9 Troubleshooting N-Port Virtualization Send documentation comments to mdsfeedback-doc@cisco.com Total number of npv-attached entries = 2 If you need to contact support, enter the show npv flogi-table command. To display a list of flogi = 4. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-4 OL-9285-05 switch# show npv flogi-table SERVER EXTERNAL INTERFACE VSAN FCID PORT...
... failed uplink that are logged in again, the login requests are logged out. Common Problems with NPV Chapter 9 Troubleshooting N-Port Virtualization Send documentation comments to mdsfeedback-doc@cisco.com Total number of npv-attached entries = 2 If you need to contact support, enter the show npv flogi-table command. To display a list of flogi = 4. Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x 9-4 OL-9285-05 switch# show npv flogi-table SERVER EXTERNAL INTERFACE VSAN FCID PORT...
Troubleshooting Guide
Page 230
... enabled, you must commit the changes. By default, this feature is checked and click Apply Changes. 11-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 If not, set the Admin drop-down menu to enable and the Global drop-down . FL ports do not support DPVM and no entries will be learned through which the dynamic device connects to the Cisco MDS switch...
... enabled, you must commit the changes. By default, this feature is checked and click Apply Changes. 11-10 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x OL-9285-05 If not, set the Admin drop-down menu to enable and the Global drop-down . FL ports do not support DPVM and no entries will be learned through which the dynamic device connects to the Cisco MDS switch...
Troubleshooting Guide
Page 266
... Troubleshooting IVR Send documentation comments to mdsfeedback-doc@cisco.com Table 13-2 IVR Enhancements by Cisco SAN-OS Release Cisco SAN-OS Release (continued) Release 2.1(1a) Release 2.0(1) Release 1.3(4a) Release 1.3(1) IVR Enhancement • IVR NAT • AFIDs • Auto-topology • Virtual domains added to remote domain lists • IVR LUN zoning • IVR QoS zoning • Service group IVR with CFS support...
... Troubleshooting IVR Send documentation comments to mdsfeedback-doc@cisco.com Table 13-2 IVR Enhancements by Cisco SAN-OS Release Cisco SAN-OS Release (continued) Release 2.1(1a) Release 2.0(1) Release 1.3(4a) Release 1.3(1) IVR Enhancement • IVR NAT • AFIDs • Auto-topology • Virtual domains added to remote domain lists • IVR LUN zoning • IVR QoS zoning • Service group IVR with CFS support...
Troubleshooting Guide
Page 499
... resynchronize the SNMP and CLI user name and password. The workaround is installed or upgraded from a CD-ROM or from Cisco.com. Cannot Log Into Fabric Manager Fabric Manager uses the SNMP user name/password combination to assign links incorrectly between these errant switches. If you use the administrator password recovery procedure, you must manually reset the administrative password on page 2-32. You should open Java Web Start and choose File > Preferences to access the HTTP...
... resynchronize the SNMP and CLI user name and password. The workaround is installed or upgraded from a CD-ROM or from Cisco.com. Cannot Log Into Fabric Manager Fabric Manager uses the SNMP user name/password combination to assign links incorrectly between these errant switches. If you use the administrator password recovery procedure, you must manually reset the administrative password on page 2-32. You should open Java Web Start and choose File > Preferences to access the HTTP...
Troubleshooting Guide
Page 508
... service using a web browser. Click Admin > Configure > Web Users > Local Database. Set authentication.mode=local. Restart Fabric Manager. In this way, Performance Manager maintains significant historical data without the cost of users in the real-time round-robin database. You see the list of an ever-increasing database size. The Performance Manager has three operational stages: • Definition-The Flow Wizard sets up flows in the collection configuration file. Performance Manager creates a series of archived data...
... service using a web browser. Click Admin > Configure > Web Users > Local Database. Set authentication.mode=local. Restart Fabric Manager. In this way, Performance Manager maintains significant historical data without the cost of users in the real-time round-robin database. You see the list of an ever-increasing database size. The Performance Manager has three operational stages: • Definition-The Flow Wizard sets up flows in the collection configuration file. Performance Manager creates a series of archived data...
Troubleshooting Guide
Page 539
...config terminal switch2(config)# span session 1 Appendix B Troubleshooting Tools and Methodology Using Cisco MDS 9000 Family Tools Send documentation comments to mdsfeedback-doc@cisco.com The no logging console command shown in Example B-7: • Disables console logging • Enabled by default Example B-7 no logging console Command switch(config)# no logging console The terminal monitor command shown in Example B-8: • Enables logging for telnet or SSH • Disabled by fixing the device configuration, you typically need to focus on what devices are monitoring user traffic...
...config terminal switch2(config)# span session 1 Appendix B Troubleshooting Tools and Methodology Using Cisco MDS 9000 Family Tools Send documentation comments to mdsfeedback-doc@cisco.com The no logging console command shown in Example B-7: • Disables console logging • Enabled by default Example B-7 no logging console Command switch(config)# no logging console The terminal monitor command shown in Example B-8: • Enables logging for telnet or SSH • Disabled by fixing the device configuration, you typically need to focus on what devices are monitoring user traffic...
Troubleshooting Guide
Page 556
... best practices 2-2 install error 2-9 overview 2-1 power on or reboot fails 2-13 recognizing errors 2-26 recoverable restart 2-27 resets 2-26 unrecoverable restart 2-31 upgrading best practices 2-2 verifying installation 2-6 software images, selecting for supervisor modules 5-13 IN-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x Software Installation Wizard (procedure) 2-10 SSI boot images configuring with install ssi command 3-6 verifying 3-5 SSI boot variables verifying configuration 3-7 SSM description 3-1 Features per release (table) 3-2 initial checklist 3-3 modflash...
... best practices 2-2 install error 2-9 overview 2-1 power on or reboot fails 2-13 recognizing errors 2-26 recoverable restart 2-27 resets 2-26 unrecoverable restart 2-31 upgrading best practices 2-2 verifying installation 2-6 software images, selecting for supervisor modules 5-13 IN-8 Cisco MDS 9000 Family Troubleshooting Guide, Release 3.x Software Installation Wizard (procedure) 2-10 SSI boot images configuring with install ssi command 3-6 verifying 3-5 SSI boot variables verifying configuration 3-7 SSM description 3-1 Features per release (table) 3-2 initial checklist 3-3 modflash...