Discones are pretty simple antennas, so I'd recommend doing some more troubleshooting before replacing it. Since you haven't completely ruled out other issues, replacing the antenna might not fix the issue.
So, from what i gather above, you get some reception of the system with a RS800. That should mean there is something useable there. Ideally, an outdoor antenna should work better.
There are cases on simulcast systems where too much reception can work against you. If you could tell us exactly what system you are listening to, that would help.
Not sure what "5GR8X" is, I think you might be referring to RG-8X.
RG-8x isn't going to work well at 800MHz. At those frequencies, and at 75 feet long, you are losing almost 90% of your received signal just in cable losses. Considering that discone antennas exhibit zero gain, that means what little signal you are picking up, only 10% is making it to the radio.
10% isn't a lot to work with. It isn't surprising that the RS-800 is working better.
Since coax cable losses decrease as frequency drops, that would explain why the antenna appears to work better on VHF, much more signal is making it through the coaxial cable to your radio. Using one of the many on line coaxial cable calculators, at 154MHz, almost half the signal is making it to your radio.
So, since the antenna probably isn't the issue, replacing it would only work if you had a big yagi directional antenna that picked up the signal so strongly that it was enough to get a usable signal to the radio.
That would work, but you'd sacrifice everything else in the process. A directional antenna like a Yagi is good only on a narrow slice of the spectrum and really only in one direction. Probably not a good solution for a scanner listener.
What you need to do is replace the coax cable with something shorter and/or of higher quality.
Reducing the coax cable length would help, if that is an option. If it isn't, you'll need to replace the coaxial cable.
For 800MHz, you really need to be using much better grade cable.
Minimum I'd recommend for a 75 foot long run would be Times Microwave LMR-400. That'll have less than 3dB loss at 75 feet and 854MHz.
It's a bit stiff, so it'll be harder to run. It won't bend as sharp as the smaller cable. It's also going to be more costly.
Depending on what your budget is, you might even want to go to a higher grade cable:
Times Microwave LMR-600 would be a better choice. It'll have 1.8dB of loss at 854MHz/75 feet. That would mean 65% of the signal would make it to your radio.
Drawback of LMR-600 is it's much stiffer than LMR-400, needs a big bend radius and is going to cost more.
A step even farther up would be LDF4-50A Heliax cable. Expensive, unwieldy, but 1.5dB of loss will mean 70% of the signal will make it to the radio. Overkill, yes, but options are nice to have.
Another option to consider would be to leave your discone as is with the existing coax and use it for VHF use.
Install a new dedicated 800MHz vertical antenna with a few dB of gain and feed it with one of the cables listed above. That will really improve reception, but will require switching antennas depending on what you want to listen to, or using a diplexer.
I think your first step would be to replace the coaxial cable. Use the LMR-400 at minimum, better if you can afford it. Use that with your discone and you'll see an improvement.
Get the cable with the connectors already installed. Unless you've done it before, have the right tools, it'll save you a lot of headaches.
Get the antenna end with the right connector to match your antenna. Don't use adapters. Make sure you waterproof the connection at the antenna properly.
At the radio end, don't connect these heavier coax cables directly to the radio. These cables are stiff enough to cause damage.
Instead, terminate the cable with a female type "N" connector. These are good, low loss connectors.
Use a short jumper with a male type "N" connector on one end and a connector to match the radio on the other. Use something like RG-58 or similar. For a short run, there won't be enough loss to worry about. The flexible jumper will prevent a lot of strain on the radio.