I like this quest because, as everyone has said, it's easy once you get the 3 clues. The names and characteristics of the suspects never change from town to town, either, so you can eliminate suspects as you go. (This is all for version 1.4.1, just to be clear.)
The Dignified Contender has a full head of hair, a beard, facial tattoos, and a sword.
The Hardy Contender has a full head of hair, a clean-shaven face, no facial marks, and a sword.
The Confident Contender has a full head of hair, a beard, no facial marks, and carries an axe.
The Bold Contender is bald, has a beard, no facial marks, and a sword.
The most efficient way to deal with this quest is to ask around in the tavern. Most of the time, you can get all three clues there, and there are a bunch of people in close proximity so there's less walking around involved. If the first clue mentions that the spy has a sword, you can immediately eliminate the Confident Contender. Then if the next one mentions a beard, you can immediately eliminate the Hardy Contender. And so forth. If you're lucky and get all three clues in the tavern, you can walk out and straight up to the spy and challenge him without having to actually look at him. I usually spend more of my time during this quest trying to find my way around the twisting streets than anything else!