Question:  I have moms with infiniband and ethernet interfaces.  How can I verify that the communication between moms is happening on the right interface?

Answer:  You would want to run a command on one of the compute nodes to watch for traffic on the infiniband interface, on the port used for mom to mom communications, coming from the ip associated with the infiniband interface of another compute node:
tcpdump -i ib0 port 15003 -n src host <ip>

You'd want to decide which two compute nodes you're going to use for your test. On one of them you'd run that command, replacing <ip> with the ip address of the infiniband card of the other computer node.

Then you can submit a job that will run on those two nodes like this:
echo sleep 60 | qsub -l walltime=60,nodes=node1+node2

If you don't get data showing up in tcpdump when that job runs then you would want to re-do the test, removing the '-i ib0' to see if the traffic is coming on the ethernet card.