It's not hard at all to run directly to the fuse panel. I have a dashboard camera in my car wired up in that fashion.
What you need is something called a fuse panel tap. Pull a fuse from the panel(preferably something that is ~10 amps), plug in the fuse tap, and then you'll have 2 fuse ports. Add the fuse you pulled out to the first slot, and then add a fuse for your radio to the second slot. The wire coming from the fuse tap is where you attach your positive wire. Then just attach the ground wire from your radio to the metal chassis of the car.
Here's a pic of a fuse tap:
Fuse Tap
The fuse you use needs to control a circuit that is off when the car is off, since you're worried about the radio running the battery down. Usually this will be the fuse for the power windows, locks, or the car's stereo. Just use your car's manual to find which fuse is which.
I will say that you shouldn't be too worried about running the battery down. I have a 2m Azden installed in my car and wired directly to the battery. I've been in a hurry before and left it on over night. The next morning the car started with no problems at all. Most radios have very low current draw when they are just sitting there, turned on, and receiving. Transmit is what really runs the battery down but if you just forget to turn the radio off it usually takes a wile for it to run the battery down.
For radios, especially CB's, it's always a good idea to go straight to the battery. Among other reasons, it helps cut down on interference from other electronics/circuits in the car.