The A, B, C inputs on a SmartBOB are designed for "TTL" signals in the 0 to 5 volt range. There is protection provided so a 10 volt signal from the Masso encoder will not harm the input, but I am not sure the input will respond.
In order for the A, B, C, input to respond the signal must be capable of producing a valid logic low signal of less than 1.5 volts and the ground reference of the encoder power supply must be connected to the GND terminal of the encoder connector of the SmartBOB. Be careful that there is no conflict in the ground systems.
You can use the Real-Time Input/Output Status panel provided in the SmartBOB plugin configuration tab in Mach4 to verify that the SmartBOB is able to read these signals.