I think it's a bit of everything...the equipment used by the dispatcher, the equipment used by the person listening, the skill level of the dispatcher, and the environment both parties are in.
I'm in my 31st year of dispatching. It all started for me when I got a scanner as gift when I was in high school. I most definitely had trouble understanding some dispatchers, and PD/FD/EMS units, while others I had no problem understanding. Over the years I've worked with several dispatchers who insisted that the faster you talked on the radio the better you were as a dispatcher. Must have been coincidence that those dispatchers were the ones most often asked to repeat what they just said!
When I'm at work I have no problem hearing and understanding the units I'm dispatching, and I hope they have no problem with me. While our desk is set up for headset use we don't use them. The Motorola speakers and desk mic have excellent audio. When I'm off-duty and listening to my scanners at home I do prefer to use either earbuds or headphones. Not so much for the privacy factor, or that I have any hearing issues (none that I'm aware of...yet), but they just make listening a lot easier.
Out of all the variables in this listening equation I believe that training dispatchers, and field units, to speak clearly on the radio is key.