Let me point out that horizontal separation, in addition to vertical separation is most likely needed. It's all in the charts in the above referenced link.
But, once you start investing in high quality coax, connectors, antenna mount arms and such, you will instantaneously surpass the cost of a starter duplexer. And you would need two runs of coax, one from each radio to it's respective antenna.
While good quality coax is a must, I suspect that you'd get much better performance with a $150 duplexer and the antenna at 20 ft than you are now, even using the coax you currently have. The low antenna height means a shorter, single run of coax which will negate at least some of the loss due to your current setup. Depending on what coax you are currently using (you didn't say BTW), you might even see a drastic improvement in performance.
In a perfect world, you'd have a $2,000 repeater with a built-in duplexer, super high grade hard line, and a 7 db gain antenna on the top of that 60' mast. However, I'm wondering if the mast you have would handle the weight of the hard line and gain antenna.
For starters, I think you'd be much better served with the antenna much lower and a duplexer in place particularly if you want to keep it on the inexpensive side for now.
Using even a mobile duplexer(ASSuming an at home repeater) in an environment that is not saturated with adjacent channel RF, using 1/2" hardline, will run circles around dual antennas with LMR400.
Never going to work well... The problem with the above is the two antennas will have different 'ranges' with the one on top covering a much larger area. If the receive antenna is on top you will have a 'mouse'-hearing much further than it talks and the other way you will have an 'alligator' that can be heard much farther than it can hear. Both are poor in practice.
If the OP already has HAAT advantage, vertical spaced antennas will have little difference in 'range'. Having done coverage analysis of a site, already at a HAAT advantage site, there was very little difference on predicted coverage with an antenna at 40m and and an antenna at 140m. The major difference, with 100m(328') height difference was shadows at 40m were filled in at 140m. Increase in footprint was less than 6mile radius.
What will be significantly different will be antenna pattern. If one antenna is above the top of the tower, it will have more of an omni pattern than an antenna side mounted. The side mounted antenna will become somewhat directional due to tower shielding.
The reason most installations use a single antenna is to make the transmit and receive 'patterns' nearly the same.
Actually, many systems use a single antenna because it is cheaper than buying two runs of feed line, and two antennas. It is also, at a low RF site, much easier to engineer a repeater to use one antenna than two.
You can improve your operation by the use of some cavities in both the RX and TX lines at the radios. The use of some pass and notch cavities will greatly improve the performance.
Your actually creating a poor mans duplexer in this fashion. But if you only use one cavity in each line, you should not get into problems with the cable lengths. If you put multiple cavities in each line, then the cable lengths will be critical.
At this point he may as well just use a duplexer. Also, critical length coax is required only between the pass/reject cans.
why are not the cellular companies saving money and using this type of cable on all their towers?... They all use Heliax type cable or this Comscope aluminum, solid outside shielded cable... Even Motorola is using it when they install Public Safety antennas these days.
They use hardline because of attenuation at 700/800/1700/1900/2100/2500MHz, that and it is a full duplex application. 100m of LMR hardline has extremely high losses at these frequencies.
Figure that a rural cell site is 100m tall, minimum loss(NOT including connectors) of LMR400 just for on the tower is:
700MHz 10.3dB
800MHz 11.0dB
1700MHz 18.0dB
2100MHz 20.2dB
1-5/8" hardline losses for the same run
700MHz 1.87dB
800MHz 2.01dB
1700MHz 3.07dB
2100MHz 3.53dB
It is simply an attenuation issue with regards to cellular.
The PIM issues with LMR400 are a completely different issue all together.