I bought about ten cheap Chinese USB sound cards a while back and some had this whine and some did not. The tone produced wasn't consistent either.
If you are using a card with multiple inputs, you may want to open up your mixer and disable or minimize the inputs you are not using. Some internal cards have inputs for CD, telephony, etc, that aren't visible from outside the PC case.
I don't think it will eliminate the tone, buy you could try changing your sample rate from 22.050 KHz to 11.025 KHz.