A first thought:
If the cable you got from RS also included the extra "pigtail" that converts a stereo connector into a mono connector, you must _not_ use it. Use only the cable that has the USB connector on one end, the little brick with the blinking light in the middle and the stereo connector on the other end. The mono adapter messes up the connection between the computer and the radio.
A second thought:
I don't know if this will make any difference or not, but have you tried the sequence of physically connecting the cable to the computer and the radio _before_ starting your software? If not, you might try this. The software might be checking the connection as its loads and might be getting confused about the connection.
A third thought:
Just in case all you have done so far is a basic driver-installation, it will not, by itself, let the computer communicate with the radio in the way most people expect. You will need one of the major software programs used to program/control the scanner--like: ARC..., Free..., Win..., Pro..., etc. as have been mentioned in various RR threads. As a premium subscriber, these software packages will let you connect directly to the RR database (using your RR login) and easily collect the frequencies from the RR DB that you wish to upload to your scanner.
A fourth thought:
Some questions to help narrow the other possible factors:
What software are you using?
What type of computer?
What computer OS?
What other details can you share?
A fifth thought:
No, the scanner does not need to be set to a "specific mode" to communicate with the computer. The programming cable simply needs to be plugged into the jack marked "PC/IF" (not the headphone jack) and connected to the computer at the other end. The programming software and the driver-software together do the rest of the work.
Hope this helps,