"Birdie"
Yeah, that's a problem! I tested it on my 500 and same deal. I did note that it seems to be a bit impedance dependent. In other words, characteristic impedance that the antenna presents to the radio's RF amp has a definite effect on the self-quieter's apparent level. In the RF engineering arena that's what a "birdie" is technically called: "self-quieter". All radio receivers have them but with good internal shielding and layout plus paying attention to the receiver's internal frequency plan you can keep them to a minimum at least in the band you are designing for. Unfortunately for receivers designed for a very wide frequency range this is very difficult especially when you have a low price point to hit; internal shielding costs money which a typical scanner manufacturer might forgo in order to remain cost competitive.
Anyway, getting back to the antenna impedance effect - I am not sure which antenna you are using but if you are using the one that came with the radio (I am assuming it is a PSR-500 as opposed to a PSR-600) then that is worst case. That antenna most likely presents a pretty bad load at 857MHz to the receiver's RF amp. I noted that the "birdie" was strongest with no antenna (worst mismatch), slightly less strong with the stock ducky (really bad but slightly better mismatch than no antenna) and weakest with my Radio Shack 800MHz ducky (best of the three apparently in terms of antenna characteristic impedance at 857MHz. The best result I got was from using the 20dB attenuator. This seems to confirm my suspicions that this is an impedance dependent self-quieter. Possibly the RF amp or (depending on the interstage coupling characteristics, port-to-port impedance isolation, etc.) some succeeding stage is going into oscillation when a non-ideal load is present at the antenna port. In such a case, putting a resistive attenuation "pad" in line can often cure or greatly reduce the severity of the problem. I noticed that when I kicked in the attenuator the self-quieter went away completely when no antenna was present. Unfortunately, with the stock antenna attached kicking the attenuator in helped but not completely - not sure why this should be; possibly there is a radiating component to this self-quieter that is still picked up by the antenna (in which case as GRE said, using an external antenna may help). A resistive pad of 20dB should provide more than enough impedance isolation port-to-port. It depends on the internal design and component layout so I really can't speculate further. Using the Radio Shack 800MHz duck with the attenuator kicked in was almost as good as attenuator + no antenna though I still needed to increase the squelch a bit.
So, I would suggest experimenting with different antennas - try the Radio Shack 800MHz ducky as that seems to work well overall on 800MHz. Also, if you are in a strong signal area use the attenuator.You'll probably still have to raise the squelch more than normal but it's better than nothing. If you use it alot in the car then try a mobile whip with a good impedance at 857MHz and see if that helps. If you can find a 6dB to 10dB external 50 ohm pad with BNC connectors (try Mini-Circuits) you could try putting one of those in line between the antenna and the radio to see if that helps - it might be a little mechanically unwieldy but it might help and wouldn't be as lossy as the internal 20dB attenuator. Otherwise, there isn't much else you can do. I really sympathize!
One other thing to try - in the super expert settings of the radio there is an EDACS disconnect tone setting or settings that may help in keeping the radio from hanging on the problem frequency after a transmission ends. I can't help you much here so you should try and find someone more familiar with EDACS disconnect tone settings to help you adjust these. I am having a similar problem on a Motorola system I monitor (though it seems more like external interference in my case rather than an internal self-quieter) and have been playing with the Motorola disconnect tone settings - so far to no avail but I'm running blind as I do not have sufficient knowledge of the finer points of Motorola trunking protocols to intelligently drive these. If you could work this out it wouldn't get rid of the "birdie" but at least (in theory) it might keep the scanner from hanging on the frequency after the transmission ends (you'd still get the interference during transmission though which will vary depending on the strength of the desired signal relative to the strength of the self-quieter).
I've often wished that manufacturers consider a "detune" feature for the receiver which would allow one to shift the internal local oscillator slightly in tandem with the rest of the receive chain's frequency plan to allow one to shift a problem self-quieter to another non-used frequency. Unfortunately, that would take some planning as you need to maintain center slot for all of the internal IF filters and interstage coupling components. Also, such a feature would likely complicate FCC acceptance testing. But it would be nice!
Good luck!
-Mike
