This may be a silly question, but when you ordered the duplexer, they did actually tune it for you? Normally they email you and ask you to supply the input and output frequency. If they did not do this, then it cannot be tuned. A 4ft fibreglass antenna suggests a zero gain design. One thing that is country specific. Here in the UK, to get the licence you have to supply OFCOM, our Government Agency with location, height, power at the antenna, antenna gain and other details to get a licence - did you not have to do the same thing? 60ft of RG58 and a 0dB gain antenna mean that you're losing a not tiny amount in the cable.
A few things you can test. With the duplexer in and out of circuit, inhibit repeat on the Kenwood and just see what the range OUT is. Even with 58, 30 ft up in the air should give reasonable coverage of the terrain is reasonably flat. If you can use cell phones, transmit on the system and see how far it reaches. Then, again with and without the duplexer, try the reverse - inhibit TX, and see what you can hear. If these tow tests give pretty good range, that simply vanishes when you switch to repeat, then it's desense - with the TX output desensitising the receiver - badly tuned duplexer being the normal cause. Report back.