In my experience, when people complain about "motorboating", they're referring to a scanner receiving a digital voice transmission (e.g. P25 CAI voice) but not decoding it as "digital" - instead playing the undecoded digital noise as if it was an analog voice transmission.
For example, a Motorola trunked system (3600bps control channel) on which some or all of the voice transmissions are digital. Unless explicitly programmed otherwise, the scanner has to wait for a short time at the beginning of each voice transmission, in order to determine if that transmission is digital or analog. If it's really a digital transmission, but the scanner "misses" that fact, you'll hear the undecoded digital noise: "motorboating".
Another example: a non-trunked (conventional) P25 CAI voice frequency that isn't explicitly forced to digital mode (e.g. by telling the scanner, in some way, that it's digital - like programming a NAC value), or which is erroneously programmed as "analog" (e.g. by programming a DCS or CTCSS value). In the first case, the behavior would be like the trunking example above; in the second case, the scanner will always give you the digital noise, since you've told it it's "analog" (presuming the CTCSS / DCS programming doesn't "gate" the audio).
You'll always get this behavior on an analog scanner programmed with such systems or frequencies. It's possible to get it on a digital scanner, as well, depending on signal quality, scanner settings, etc.