Scanners are like mini computers and with computers there are several ways to achieve the end result.
If I'm absolutely, never-ever, be interested in scanning something, I'll delete the sections applicable.
If there is something I might be interested in monitoring occasionally, I'll assign the "stuff" to a different Department Quick Key and set the DQK to be off. I could also do as you mentioned, set the "stuff" to Avoid. I like the DQK over Avoid because it is a little easier to toggle a DQK than it is to un-Avoid one single thing.
You won't see any impact on scan speed by avoiding, deleting or "Quick keying". The scanner won't attempt to scan such things and thus won't be slowed.