Phooof! Alot of funky info here!
I have a background in RF engineering and have built and tested a lot of antennas. Firstly, mag mount antennas use capacitive coupling through the magnetic base to couple to the metal underneath and use as a ground plane effectively creating the other half of a dipole and definitely do NOT count on a connection to ground from a coax shield further back in the transmission line - that would seriously make for a very strange antenna consisting of the coax lead as well as the outside antenna element! And most vehicle antennas, mag mount or otherwise, are quarter wave so using the vehicle's body for the other half of the dipole or "ground plane" is essential for proper performance. There are half wave designs which do not require ground planes, however, but they are less commonly used relative to quarter wave whips (for non metallic vehicle body mounts such as fiberglass they are the preferred antenna type). Most mag mount mobile scanner antennas that I have seen are based on the quarter wave design and require a ground plane to work optimally.
Secondly, thru-glass antennas are usually half wave designs because of the lack of a groundplane (pure glass cannot work as such, of course). They capacitively couple through the glass as another poster correctly stated. In theory, there is no reason that a properly designed, constructed, and installed thru-glass antenna could not equal the performance of any other similarly placed end-fed or coaxial center fed half wave dipole mobile whip with direct mechanical-electrical connection to the feed line; in fact, in the horizontal plane, a half wave antenna should ideally outperform a quarter wave with a ground plane. But there are many issues to this. Size-wise, half wave mobile antennas for VHF-low band are basically impractical. Also, for best operation and least interaction with the vehicle's metal body the antenna should be mounted as high as possible and as close to the middle of the rear window horizontally as possible. And the mounting point should be free of any in-glass metal such as rear window heating elements.
Finally there is the issue of capacitive coupling. Ideally, in theory, there is nothing wrong with it provided the elements of the capacitor are designed correctly. Herein lies the problem - the further away from the "ideal" model the more issues you encounter. A real world capacitor, even a very expensive high quality low tolerance one, is not ideal and contains internal resistances and inductances which affect its performance especially in very broadband frequency cases. An "ideal" capacitor has a low frequency limit beyond which its capacitive reactance raises with decreasing frequency but above which it should approach near zero all the way to infinity frequency-wise. In the real world, however, any capacitor will also have an upper frequency limit, usually where it will start to look like an inductor and exhibit inductive reactance. The point is that capacitive coupling wherein random elements are involved, like the metal in the roof plus the paint and whatever other contaminants are present for mag mounts and the glass thickness and type for thru-glass mounts (these elements are essentially forming the dielectric of the capacitor), can become unpredictable especially over a wide frequency range. It's always easier to "tune" an antenna system for a narrow range of frequencies, such as just the cellular 800MHz band, than it is to do so for a very wide range of frequencies, such as needed for typical scanner usage. Even with "multi-band" and "broadband" antennas the system will "detune" outside a certain relatively narrow range of frequencies in some way. Capacitive coupling for attaching the feedline to the antenna (i.e. thru-glass half wave mounts) or for attaching one half of the antenna to its other "half" (i.e. magnetic quarter wave mounts) simply presents another random (given the potentially wide variance in materials forming part of the capacitor) element to the design of the system. Practically speaking, for some cars with certain types and thickness of glass and for certain mounting positions and for certain frequencies, thru-glass mounted antennas work fine - as good as or better than many directly connected antennas. But for others, not so much. The same can be said for magnetic mounted antennas though one can argue that capacitively coupling to half the antenna (in the case of a quarter wave magnetic mounted whip to a metal body) might be less dicey than doing so to the whole antenna (in the case of a half-wave thru-glass mounted whip).
For scanner receive-only usage, wherein reflected energy is not an issue, a cheap and simple quarter wave whip tuned for VHF-high band in the 160MHz range will perform quite well on that band and decently on everything else for local reception. Direct mechanical mounting is, for the above reasons, the preferred mounting method but magnetic is usually fine. Multi-band and wideband antennas may improve reception especially on 800MHz and low band (if designed for those). If your primarily interested in receiving a certain band then simply get an antenna tuned for that band and your reception would be best; for example, if you are mostly interested in listening to the 800MHz band then a 3.5inch whip would be a fine quarter wave antenna with little or no wind resistance meaning that "walking issues" with magnetic mounts at high speed and high winds would be minimal if present at all.
If you decide on thru-glass designs make sure that they are either half-wave designs with center fed coaxial vertical dipole designs or correct end-fed designs with matching for 50 ohms or that a provision for some form of external ground plane is made if they are quarter wave designs.
-Mike
