While MixW is very good software, the vast majority of sigs outside of hams (and some maritime/aero stations) are in modes that it simply isn't designed to decode. SkySweeper is a much more comprehensive package for such modes, and it is upgradeable.
Now as to how to identify what you are hearing - the first way is
listen. Some modes, such as Pactor and SITOR, have a repeating, rythmic sound to them, and you can identify things like that easily. ALE has a scratchy sound all its own (and is easily decodeable with Freeware- PC ALE is what you need there), and 75 baud RTTY also has its own sound.
The other way is to view a waveform, something MixW will allow you to do (at least, in the old version I have....). Each signal has a characteristic waveform which is likely to be even more accurate than your ear. For example, MT63 has a very wide (almost 1khz) bandwidth, as does PACTOR 3 (which is very controversial to find in the ham bands...). Baudot RTTY has a 2 tone system, which are seperated by a fixed amount.
The utility monitoring wiki (link below) has 3 or 4 links where you can check out audio samples of common - and not so common- signals found on HF and beyond...
http://wiki.radioreference.com/index.php/Utility_Monitoring
The other ways involve a little legwork. Checking references like the Klingenfuss guides are a good start, but a much more up to date way to find these things out is to join a Yahoo group which specializes in utilities - the UDXF. The link should be at the bottom of that article above. Keep in mind that such groups are worldwide; consider propagation characteristics before jumping to conclusions.
73 Mike
