If you are using a consumer grade receiver, low end coax and an "all band" scanner antenna, I doubt you'd notice a difference. If you were using a high end receiver, 1/2 inch or better coax over a long run, and a good high gain antenna, yeah, you'd probably notice a difference.
It really comes down to keeping this a hobby or not. You can chase 0.1dB of loss around in circles, but the truth is most hobbyists with consumer gear wouldn't know a decibel if it came up and bit them on the rear.
If you are building out a new system, absolutely, go with "N" or "BNC" connectors on everything and ditch the UHF connectors. However if you've got 100 feet of RG-58, a rusty old 1970 scanner antenna duct taped to your chimney and a consumer grade radio, swapping out all your UHF connectors for N probably isn't going to result in anything noticeable.
In other words, swapping out your connectors without changing anything else is going to be pretty much pointless. Use what you've got and what you can afford. If you've got a wad of cash burning a hole in your pocket, then go for it.