When I first got into ham radio, I had one mast on a vent pipe and another mast anchored to the chimney with straps. This was up in NJ, which was relatively safe, but on occasion we'd get a nor'easter or hurricane come through. So, I had a TV rotor (very light duty) with an 11 element 2 meter beam on top (that's very close to the marine frequencies). Never any problems with that. Then I added a 10 meter monobander (it was pretty light). No problems with that. Then I changed the rotor to a Ham 3, with schedule 40 pipe. I put a Moseley tri-bander up (along with a 2 meter antenna and a 440 yagi). It was fine for a while. Then it blew over in a windstorm and took the chimney with it.
So, I lived that "chimneys are made to be chimneys" thing. The other consequence was that the fall damaged the roof and also sent some bricks down to the driveway. Not a good thing, as until I could afford to fix the roof, I had water coming in every time it rained. It was just a disaster.
If your chimney is painted over and maybe even tarred, you have no idea if the mortar is spalled, if the bricks are cracked, etc. All of that affects mechanical integrity. If the chimney is in good condition and you put a light weight antenna on it (note that most TV antennas are extremely flimsy), you'd probably be fine. The more you hang off that bracket, the more force it will transfer to the chimney. The guys are right. Chimneys aren't made to deal with that. With enough crap up there, you'd eventually end up like I did when I was a kid.