Radios now-a-days sense the mic parked in the clip by grounding a wire that runs back to the radio through the mic cord. Either the metal clip has to be grounded, or the very act of inserting the mic into the clip shorts contacts at the button. Either way, you'd have to tap into this wire to sense it going to ground indicating it's on-hook. I guess it could be done but I'd be leery about doing this as it is almost always a 5 volt logic line and you'd need some kind of buffer circuit to protect the inner workings. Hanging the low side of a relay coil on this line is a definite no-no.
Best thing to do is hunt around for a hang up box. Older radios, Motorola among others, had a mic "box". Inside the box was a switch or contacts. Coming out of the box were a couple of wires that plugged into the control head. You can instead connect the wires to ground and the low side of a relay that would switch whenever the mic is picked up.
The best one would have a little micro switch in it and you can swap wires to the normally closed side, that way the relay would only energize with the mic off-hook.That is the safest method. Other styles have the wires going to the case and a single contact inside. With these you have to pay attention that the ground is on the box and the relay low goes to the contact. Also the relay will stay energized while the mic is on-hook so you need to have a way to remove power from the relay when the radio is off to keep from running down the battery after a period. Less desirable option but can work.
Then hooking the relay up to mute the stereo is up to you. Depends on the unit and it's options.
