Depending on your programming, Monitor will do one of two things, both of which involve checking a channel to see if it's busy or not.
If you have Monitor set to Open Squelch (I don't remember the exact term in the programming software), pushing whatever button you have set to Monitor will open your squelch all the way and allow you to hear anything that is going on. If nothing is going on, you will hear static.
If you have Monitor set to PL Disable (again, may not be the precise term), pushing Monitor will temporarily let your radio receive all (or no) PL/CTCSS tones instead of the one you have programmed in to the current receive channel. This is useful if you have a channel where several different groups are using it, with different PL codes; you hold Monitor for a moment to make sure somebody with a different PL code than yours isn't talking, so you don't interrupt them when you start.
Busy Channel Lockout, if I recall correctly, is more of a scan mode thing. While you're scanning multiple channels, the radio will usually stop on any channel that has activity on it, so you can hear what is going on. However, if the activity on that channel doesn't match the PL code you have programmed, you don't want the radio to stop on it, so "Busy Channel Lockout" tells the radio to ignore that channel in the scan list until the next time the radio doesn't detect any activity on it. That way it's not constantly having to stop and check if the PL code is the right one.
As for your question about preventing someone else from talking over you, I don't think any of those will do it. What you may be looking for, if your radio has the feature, is "Tx Inhibit on Busy". What this does is, if you are receiving a signal and you try to push the PTT (push to talk), it will "bonk" at you (send an error tone) and not enable the PTT.