For me, it's definitely the wind. Especially during the spawning season, there's often too much wind on the days I have off to get out and safely fish. Second would be loading and unloading my kayak. That takes a long time, both at the lake and at home, and wears me out.
Lots of other things will annoy me, like wake boarders in the cove, party boats with their loud music, bass boats that think they own the water and try to push me off or beat me to spots, angry strangers that want to confront you over stupid things, and of course, gear failures on the water. But I can take all of that in stride. Freezing temps and 100° days aren't much fun either, but I know how to handle them. Even the crowds can have a positive side. Sometimes you'll see someone fishing a spot you never thought of, or you'll have to get creative on an old lake to find new spots. It keeps you learning.