It is a balancing act
Ext Sp is amped audio and not good for a computer.
Rec Out is the best you can get and use the computer Recording Settings for Line-In to control the audio.
Most scanners don't have REC OUT and people have to make do with what is available. Also, many don't have EXT SP either, which means you're stuck with the headphone jack (also 'amped out' - lol).
The key is to not overdrive the sound card input. But you don't want to set the volume level too low either; I find an 11 o'clock setting usually works fine.
Whether to use MIC or LINE input on the sound card depends on the sound card. Some of the sound cards I am using only provide a MIC input. Sometimes there are extra settings in the Properties for those inputs. Mine have an "auto level" setting, or something to that effect which I turn OFF.
Another problem is that most headphone jacks do not provide a true ground. They're wired so that a user can plug in a mono or stereo headphone and get audio in both ears. When that's what you're using, get a ground from somewhere like power or antenna connections. I also run my audio through isolation transformers. Radio Shack sells one for ~$16, I made my own.
I am using a couple different version of the DarkWood meters,
found here. I set my radio to NOAA weather (they are always broadcasting) while watching the meter and listening to the feed on the other end and made initial level settings. Once I select the channel(s) I will be streaming, I usually have to adjust the level up just slightly. You'll hear lots of distortion if you're overdriving the sound card input; in which case turn the radio volume down slightly.
But the problem here is too little sound level. So I suspect you either need a slightly higher setting on the radio volume control or perhaps use MIC input rather than LINE input on the sound card.