Answering with yes shouldn't imply that they will go fishing with me tomorrow.
Yes=If I have free time tomorrow, I can go to fishing with you.
That person might think that he doesn't have free time tomorrow. But if it unexpectedly happens to be the case that he finds some free time, he agrees to go fishing.
They can still answer yes, and lie later on that they didn't have any free time.
No=Even if I have free time tomorrow, I won't go fishing with you(because i don't want to fish when it's cold or w/e)