I don't think you can delete attachments from the manage attachments window when you make a reply, but you can do it from your forum
User CP in the upper left of the forum.
This link goes directly to that menu. You can get to it manually by clicking on
User CP and looking on the menu on the left side. There's a link under
Miscellaneous called
Attachments where you can delete attachments you have uploaded.