If you think that the talkgroups might be of interest at certain times (but not 'right now'), then just park then in a Department that you set as avoided. You can always clear the avoid on that department if you want to put them back in your scanning sequence at some other time. If these are ones that you actually have no need for, then delete them.
But either way, you won't notice much, if any, change in scan speed. The scanner stops on a site for a system, then tries to receive & decode the control channel data. If it can, then it looks for any programmed talkgroups that are active, If none are, it moves on very quickly. You lose much more time, reducing scan speed, if you have multiple sites that either you don't need, or that (at your current location) you can't receive.