SSB OUT LEVEL only affects the receive output level at the RTTY/DATA jack.
RPORT GAIN should work as long as you have selected rear panel input. I use DATA-U for WSJT operating, but I agree that the 3 kHz maximum receive bandwidth can be a bit limiting. RPORT GAIN certainly works in DATA-U.
There appears to be no attenuator between the CODEC and the mic input circuitry, so it is necessary to run the computer audio at quite a low level to avoid clipping. It may be this is not the case and therefore the tx audio is at too high a level before reaching the RPORT GAIN control and you can't turn it down enough to see an effect.
