A few things first on your configuration and wiring diagram:
Input Signals:
You have Mach4 input names "Input #1" through "Input #4" mapped to PMDX-411 pins 11, 12, 13 and 15. First off, unless you have a Lua script specifically reading "Input #1", you do not need these signals. Mach4 normally doesn'tuse them, though they don't cause any harm. But may just confuse things. And second, if you ARE trying to keep track of G540 "Input 1" pin number, they should be assigned to pins 10, 11, 12 and 13.
You have "Index" mapped to Pin 13, this should be Pin 10 (which is G540 Input 1, terminal pin 1, blue wire to your spindle sensor).
You have "Limit Override" mapped to pin 12, which is G540 Input 3, terminal number 3 which you show not connected to anything. That is OK, I guess.
You have E-Stop mapped to Pin 10. For a G540 this needs to be Pin 15 (Fault). I think it is active high (red X in the "Active Low" column), but I could be wrong. The G540 manual does not mention the polarity, but their sample Mach3 XML file looks to show it as active high.
Output Signals:
You have "Output #0" mapped to pin17. Again, unless you have a Lua script specifically controlling this Mach4 output signal name (I suspect you don't), you do not need this. Yes, pin 17 controls the G540 "Output 1" terminal, but Mach4 normally does nothing with signals "Output #0" through "Output #63" (unlike Mach3). If you want to use the G540 "Output 1" to run the spindle forward, see below.
You have "Spindle On" mapped to pin 1, which is G450 "Output 2" on terminal pin 6 which you show wired to spindle controller "X2" (rev run).
In this configuration, you can only run your spindle in the 'reverse" direction (unless you have a Lua script controlling "Output #0" as mentioned above). If you want to be able to run the spindle forward and backward, change your Mach4 output signal configuration like this:
1) Disable the "Output #0" signal and un-map from the SmartBOBUSB Pin 17.
2) Disable the "Spindle On" output and un-map from the SmartBOBUSB Pin 1.
3) Enable "Spindle Fwd", map to SmartBOBUSB pin 17. I presume active low since you had the "Spindle On" set that way, but I don't know.
4) Enable "Spindle Rev" and map to SmartBOBUSB pin 1. Same presumption about active low as the "Spindle Fwd" signal.
Now, finally, to your inability to control the spindle speed. I have no idea :) On your spindle controller, are the two "com" terminals electrically connected? Sometimes the common for the VFD speed control is isolated from the other command or ground terminals. I notice you have the G540 VFD "ground" tied to the "com". If so, try the changes above and see what happens. If you have a DC volt meter, try various "S" commands and measure the voltage between G540 terminals 7 and 8 (8 is positive) and let us know what you measure.