Department have nothing to do with sites, it's only a group of talk groups.
You can set a quick key for each site and I believe you have those somehow common with your department/talk group quick keys that confuses you when you turn them on and off. Try deleting the QKs for the sites in your system, there's normally no need for them, and it will become much easier and understandable to you.
/Ubbe
I agree mostly in the "Departments" are clusters of talkgroup IDs (trunked systems). But with respect to Quick Keys, sites are at the Department level.
I usually use Department Quick Keys 0-5 for Fire, Police, EMS, State Police and Emergency Management, regardless of a specific system. That's so there is a common means of Quick Key layout.
Site Quick Keys are assigned 61 on upward.
So when I was describing what I experienced, I turned of Department Quick Key 61 which was associated with a site, meaning that the site should not be monitored. However, when I did a Channel Hold, the site that was associated with DQK 61 was monitored (but only that site and not the three others).