Yes it matters. Yes you should be concerned.
While most scanners don't have an IP rating, there's some good guidelines that can really help.
Looking at some of the Kenwood portables that have the 2 pin audio connectors, they say:
"
Note: 1. To meet IP grades and MIL, the 2-pin connector cap or accessory must be installed."
Getting crud inside those jacks will screw things up. It really gets important if you are using a radio where there may be a lot of metal shavings/dust, like in a shop.
This is also a common problem with some cell phones. I know my wife's iPhone connectors get gummed up with -who knows what- sort of crap and will result in it not charging. There seems to be growing a whole industry/marketing scheme trying to get phone users to buy trickery to keep the sockets clean. (that's what she said….)
Anyway, yeah, if you are in a high dust environment, keeping it covered would be a good plan.
And cleaning them out with compressed air isn't always a good idea. Pushing that dust deeper into the radio can cause more issues.
Amazon.com: uxcell Rubber Cell Phone Anti Dust 3.5mm Earphone Jack Ear Cover Stopper Cap 15pcs Black : Cell Phones & Accessories
www.amazon.com
On the other hand, you'll probably get a lot of people telling you they've never had an issue and regularly bury their scanner in a grain silo and all this is silly.
It's up to you. Each user is going to have a different experience. It's your radio, do what you want.