People fish better when they are confident in a particular bait, in the same way if you feel more confident wearing a bright color to be safe then I say you should take your own guts advice. You certainly could always tone the color down on the front of the vest with a few fishing patches, camo duct tape, fabric approved spray paint etc., my thought there is you will still be seen well from behind and you will not be so flashy facing forward to the fish you are presenting your lure to.