Yeah my problem is that when I'm on the side view they swim down and don't slow down, then escape. It seems to happen all the time.
This is exactly what happened to me, almost every single fish would do exactly this. Immediately swim straight down and within 2 seconds ditch the line at the bottom of the river regardless of whether I pressed the button or not.
About fifteen minutes after posting I figured it out. You have to wait until they just flat out stop; you'll know you did it right when they do this weird "turning around" animation and then face towards you. Otherwise they'll just not get hooked and they'll swim away, or it'll catch and they'll speed up. Anyway I'm getting the hang of it, but I'm wondering why there are giant spiders inside the village.