I have a friend that does exactly this. Balint, VK2FUNK presented this at a computer security conference in Melbourne last year:-My received signal contains a set of sample points... How do I detect which modulation technique is being used....For ex- how to detect whether the modulation technique used is BPSK or QPSK..... On basis of what parameters can I detect the modulation technique used....
One old simple trick for determining baud rate was to pass the signal through a band-pass filter whose BW is equal to that of the unknown signal. That is then fed into an envelope (AM style) detector. The result is then plotted using an FFT. There will be a peak at the baud rate of the signal (of course this doesn't work for all classes of digitally modulated signals; I've used it on m-PSK though). [It may be interesting to note that the resulting signal is often in the audio range]...Another point - GNUradio is a great way to do that. The inbuilt GNUradio Companion lets you use a GUI to connect up different modulation/demodulation blocks and experiment around with different things until you figure out whats going on with a specific protocol.