Did you program the system you want to monitor and use CC only mode (where it does all the "programming" for you by just pulling info from the control channel itself)? If it's a trunked 800 MHz system all you should be required to do is create a system and make sure it's in CC only mode and program it with the necessary CC frequency or frequencies if there's more than one on that system (the RR database should be able to help with that info), and start scanning it.
Also, is the scanner's attenuator currently on? If it is, that'll reduce the effective signal the scanner works with by about 20 dB which is a huge amount and can basically kill it entirely in some situations making the signal unusable to any degree - it almost sounds like that might be part of the problem here so definitely check on that. If you see ATT on the display (I don't own a 996XT or any physical scanner anymore but I do know from owning a BC246T years ago it would display ATT for attenuator when that was active) then you need to disable it using the proper menu entry. By default it should be
off for all systems and needs to be manually enabled (which could have happened during programming hence me mentioning it).
You can also look up exactly where the given tower(s) are located for that system with the RR database info - in the section where it lists the frequencies for the system, look at the System Name entry and it should be a link that'll take you to a site-specific entry where it'll show you on a map as well as provide latitude and longitude coordinates.
If you're 2-3 miles away from such a site, hell, you should be able to get a rock solid signal even with no antenna attached because of that proximity. If the tower(s) are actually 20 miles away then yes you'll need to ensure you're getting a good signal so that would indeed need a decent antenna. The RS 800 is still one of the best antennas ever made IMO, and it's never failed me to pulled in signals anywhere I've used it (in basically every continental US state so far) but your actual location will obviously make a difference in the reception. You could always get an extension cable for the RS 800, maybe one of those suction cup window mount style extensions and run it away from the scanner itself to a window facing the direction of the towers - again the point here is that changing
anything can sometimes end up making all the difference, and just like with cell phones where you can be standing in one place and get almost no signal but simply turning your head 15 degrees one way or the other magically gets a solid signal is the same with any antennas, even those for scanners. Moving the antenna up, down, right, left, or away from the scanner itself using an extension cable could be what makes the most difference.
Before considering a Yagi you need to do that research and figure out precisely where the actual site tower(s) are located so you know for sure where the signal(s) should be coming from. Once you do that, it's entirely possible that just moving around in your location (home, apartment, whatever) might help or using an external antenna mounted by a window in the direction of the signal(s) might help. You could also build one of these for under a few bucks (the cost of the SO-239 chassis mount itself and a coat hanger):
The top (main) element should be 3.45" long at 855 MHz which is the middle of the 800 MHz public service band (it'll insert into the top connector on the SO-239 mount), and the ground plane elements should be slightly longer but not much, maybe 3.75" or so. Takes like 5-10 mins to make one and it can make a rather dramatic difference with 800 MHz reception - also good for 750 to 950 MHz operation to but, since they're so cheap you could make ones for those bands specifically as well if you wanted using antenna wavelength calculations from the
Frequency Wavelength Calculator.
The fun part is the experimentation to find something that works better than what you've currently got. A Yagi would be excellent for pulling in signals from a long distance because that's what it's designed to do: focus the gain and reception in a specific direction (with the Yagi pointed in the direction you want to grab signals from - think of a Yagi like a big arrow pointed in that direction).
There are about a hundred factors involved here. Tons of information is needed. Analog vs digital, P25, Phase 1 or 2, Trunking, encrypted, transmitter beamwidth, etc, etc, etc. A yagi would be the worst thing you could do. Try the attenuator. Try using a VHF rubber duck antenna--or no antenna. Remember there is nothing on the market that is good at receiving the digital P25 signals on a trunking system--poor, but not good.
You're making it sound like an analog radio signal and a digital radio signal are something unique and different at the same time: they're the same as there's no such thing as an "analog" radio signal or a "digital" radio signal - there are only radio signals, period. The difference comes from the type of modulation (AM, FM, SSB, etc) and perhaps the actual content being transmitted by the radio signal itself: analog or digital in that respect.
As for the Yagi comment, in the OP's situation if the site(s) he or she is actively trying to monitor do happen to be 20+ miles away, and potentially over a slight hill or curvature of some kind then yes, a Yagi would indeed help in some degrees to get them pulled in with much better reception of the weak(er) signals.
And if there's nothing out there but poor antennas for signals from trunked P25 systems, then the millions of us using scanners and SDR sticks to monitor such signals and systems with antennas of all shapes, sizes, and construction materials from empty 2L bottles laced with coat hangers and others like the pictured above to the higher end commercial discones that sell for $1500 must be doing something wrong.
(if worse comes to worse you can always factory reset the 996XT and start totally from scratch - it can be a pain the keister but sometimes a good fresh start can make all the difference)