I actually have more trouble understanding people on SSB/AM in the HF bands because, despite all of the noise, lightning crashes, random QRM, and the general non-uniform paths of signals coming in from all over, lots of hams tend to use the phonetic alphabet not as often as you would think; primarily they're used during nets, much like they do on FM on 2 meters and up and you'll also get it on new contacts, but if you listen in on a coversation between some people, they'll just throw their callsigns out there, and things can get quite as informal as in the Technician bands.
If I have trouble hearing someone on 2m or 440, it is because people talk too fast. I have heard people slaughter their own call signs, example: The prefix K-E-4 often to my ears becomes K-4 or K-C-8 might become K-E-8, K-Z-8, or something else ridiculous.
It gets really annoying when running nets and I have specifically have asked people to use STANDARD PHONETICS to give their callsign, and yet they still say the typical alphabet pronunciation or use some other phonetics (such as King Elliot Four or Kind Elmer Four, blah blah blah).
Learning to understand people's callsigns is like anything else we end up doing naturally in life, like adding two numbers or reading a paragraph of words, working the controls on your radio without reading the manual; it will improve with time and interacting with others. Just listening will get you used to hearing it but knowing at some point that you're going to have to say someone's callsign back will force your brain to figure it out and it is easier than you think so try not to think about it too much.