When you are configuring the IP settings for the two InfiniBand NICs, you set the default gateway even though you were already connected to your primary network with another NIC.
Windows gave a warning, and its an important warning! This could cause weird and unusual network connectivity for that computer, as it will have two "defaults" to choose from.
The way to fix this is just leave default gateway blank on the InfiniBand NICs. The two computers should still communicate fine without a gateway, just as long as they're on the same subnet.
This is at least the case with Ethernet based networking. I doubt its much different for InfiniBand (its still TCP/IP).
