Those photos of the radio and duplexer inside the computer will become legendary.
On UHF your radio output is 29.3 watts and it should be closer to 35 watts based on their claim for UHF frequencies. With that in mind I really do think the power supply may not be providing enough volts and amps to the radio. It could be the power supply itself, and or the small wires from the computer power supply not letting the power flow. (Probably the small wires)
I would say turn the radio down to "Mid" 1 at 19 watts, so that it is not starving for power. Mfg. claims 20 and you're getting 19 watts. It is probably still starving for volts/amps but it'll work at that setting and probably not get wonky. (Well, you should actually
purchase a switching power supply like this one and use the wires that came with the radio to connect it.) Until then...Mid 1.
As for coaxial cable in your vehicle, measure the run and add a few feet to be sure. Really thin coax is RG316 and RG174, but they have very unfriendly losses even for short runs. There is also LMR240 and LMR240 UF (Ultra Flex). One has a solid core, the UF is stranded. I believe either is the same size as the RG8X, but offers improved performance. Others will probably chime in.