“Swaptronics” an Inexpensive and Quick Method to Isolate Problems in Redundant Components and/or Paths

In our example, you have an InfiniBand network down on the blue path. There are redundant InfiniBand host channel adapters, cables, InfiniBand switch ports, and storage controllers. As you use this technique you are looking to answer the question, “does the problem follow the changed path or stay with the original path?”

Step 1:

• Select a simple location to swap cables between the down blue path and the green path. In figure 1, I chose the cables connected to the InfiniBand switch ports.
• Swap the cables between the two ports. This swap is illustrated above in figure 2.
• Did the issue stay at the original blue port or follow the blue cable to the green port? If the issue remained at the same blue port, the issue is at this port or further down the path toward the storage. If the issue has followed the blue cable to the new green port, the issue is with the cable or back down the path toward the server. Everything in the green path is will always be good.
• Note: This step eliminated 50% of the possible bad components.

Step 2:

• Perform the same swap between the same 2 cables at the server side as illustrated in figure 3.
• Envision again the issue followed the blue cable, rather than staying with the HCA port. Magic! We now know we have a bad blue cable that needs to be replaced.
• Best practice is to return swaps you have made back to their original places. Again, this method can be very quick without involving special tools.

Be sure to check back often for new tips.