I believe mask in this context refers to the overall footprint or appearance of the waveform amplitude with respect to frequency. This would be what one sees on a spectrum analyzer. The mask might convey requirements regarding maximum power at the center frequency as well as signal bandwidth and power of lobes or sidebands other than the center frequency.