Wirelessly posted (Mozilla/5.0 (iPhone; CPU iPhone OS 6_0_1 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A523 Safari/8536.25)
Someone reading will reply with an answer explaining the instruments you'll need to test antenna gain and other performance metrics used to measure antennas. And there is software, but you also must have the test instrument the software uses.
Assuming you are not looking at spending that kind of money, you may want to try an approach that will work with the scanner you already have.
Find the weakest signal that you can barely hear with the squelch set all the way open/off. We're talking something with a lot of hissing and flutter, but is a known signal with regard to location of the transmitter, distance, and direction from you (approximate). NOAA weather radio broadcasts are good for this in the VHF HI band. Pick a NOAA transmitter that is not local to you... or the weakest one you can find. For other bands you want to optimize your antenna for you will need to find a known weak signal in that band.
Switch to the antenna you want to know works better or not. If the weak signal disappears.... Or gets stronger... you'll know the best antenna of the ones you test. Important thing here is to test only with weak signals.
What antennas are you comparing to each other, and what are you trying to hear (frequency and how far away from you)? Answering those two questions will help us suggest better antennas. And here's a "worst to best" list of generic antenna types for handheld scanners:
WORST - The rubberized stubby ("duck") that came with the scanner
So-So- A telescoping antenna that attaches to the handheld scanner
BETTER - An external indoor antenna in your attic/crawlspace or hanging in a window and connected via coaxial cable to your scanner
BEST- An external antenna connected via coaxial cable and mounted on your roof
... and there are multiple types within each of the general categories listed above.