I see this a lot and I am guilty myself. So this just speculation on your specific scenario...
Assuming your right handed, you set the hook somewhere between 2 and 3 o'clock on your right side, the hook is in the left side of the fishes mouth, the fish jumps, you lower your rod and it puts slack in the line, and then you turn the rod to your left between 9 and 10 fighting the fish. If you had stayed to your right, the hook should potentially be facing up. If turned to your left, the fish is no longer pinned and the hook is on the bottom. The fish shakes its head and the bait pops out.
I hope I'm explaining this theory correctly. You see it more on a boat when the fish gets close and goes passed the angler when they reach down to lip it.