I'm going to jump in here for a minute or three.
First a few definitions:
"VSD" (variable speed device) is a generic term that we use for anything used to control the spindle speed, including VFDs (variable frequency devices).
For these other terms, Google/Bing/DuckDuckGo/Yahoo/InsertYourFavoriteSearchEngineHere is your friend. A quick summary:
PWM (pulse width modulation) is one type of signal used to control spindle speed. The duty cycle of the pulse (the "pulse width" divided by the "pulse period") tells what percent of full speed to run.
Step/Dir is another method that *can* be used by *some* spindle controllers to control spindle speed. This type of interface makes the spindle look like a stepper motor, and the spindle speed is proportional to the number of step pulses per second.
PID (proportional, integral and derivative) is a method used in control loops to provide an accurate and stable system. This is probably used inside your VFD as it translates the incoming control voltage into an actual motor RPM.
Run/Direction and Forward/Reverse are two styles of control inputs to spindle controllers. Some VFDs can be configured to use either type of interface. Your VFD uses only the Forward/Reverse style of inputs (i.e. one input connected to COM runs the spindle forward, the other input connected to COM runs the spindle backwards).
Now on to some of your questions:
When I click the 'Spindle CW F5' button in Mach3 the spindle at first does nothing. It isn't until I also hit the 'reset' button that it comes on.
What did the "reset" button look like before you clicked on it (and made the spindle run)? In Mach3, the "reset" button must be NOT flashing, with a solid green border, before anything will run. That is like an EStop switch on the PC.
It revs a crazy high speed and says 833% on the spindle override. I can tweak its settings and get the number down and get the spindle turning at a slower speed but the seems no method to its madness. Sometimes I think it even goes backwards. Even when I uncheck 'reversing'.
Two things to try. First, did you try setting the VFD to local control as Steve requested earlier, and can you control the spindle speed from the VFD panel?
Second, if you can control the spindle speed from the VFD panel, re-configure the VFD to use its control inputs again. Then go back and try using the PMDX-107's test button and see if the VFD/spindle respond rationally (i.e. close to full speed on the first press, then approx 30% of full speed on the 2nd press). If this doesn't work (and I haven't seen you say that it does actually make the spindle move), then we need to address that before we get into Mach3 issues.
Bob