I can do that, but if the relay is bad, would there be power going to the pump?
No, if the relay is bad, then you won't have power to the pump. However, you could have a good relay, and a bad inertia switch, which would still mean no power to the pump. The inertia switch is on the firewall between the washer reservoir and the motor. (First check that the plunger button is depressed! If it isn't, then the inertia switch has triggered, and resetting it might solve your issue. After that, pull the plug, and test for voltage into the Inertia Switch on selection of position II. (I forget which side of the plug is power in, and which side is power out.) If you get power there, then your relay is closing.
Alternatively, as Levi says, you can pull the rear carpet, and do the same voltage check at the pump. (That's the way I started my bug hunt.) But, if you don't have volts at the pump, then you'll work "backwards" up the wiring harness to the power source. It's really dealers' choice, as you'll end up checking all the same stuff, but if you want to rule out the relay first, then I'd personally start near the relay, and the inertia switch plug is the least invasive place to do that.
Wherever you test, the wire you're looking for is white/purple.