Looks like CableTech was typing too fast...
No, the antenna on top of your mast is likely a half wave VHF antenna designed to be resonate in the high 150 to low 160MHz range, so it would not be a good antenna for the FM broadcast band. It might radiate some, but your SWR will likely be pretty high.
Those antennas are usually fed with pretty lightweight RG-58 cable, and it won't carry a whole lot of power, and what power you do put into it will be mostly lost along the cable path. Even with a properly resonate antenna, you'd probably loose 50% or more of your signal to cable losses.
Theoretically speaking, yes, a small FM transmitter could be run from a boat with 12 volts. You would need a properly tuned antenna and would want to use much better coaxial cable.
As CableTech said, you would need an FCC license, because at only 1 mile off shore, you are still within the United States, and the FCC has authority. I can't remember the exact distance you need to be offshore to be outside the US. I want to say 12 miles, but I could be totally wrong. At 12 miles, you are not going to cover very much dry land with a 30 foot tall antenna.
But, yeah, sure, it could be done. Not legally, but it could be done. Personally I think AM broadcast would be better, you have an excellent salt water ground plane, and you have a tall enough support to put up a loaded or base tuned wire antenna.