You could try the most time consuming thing which is to save in a town that has plenty of those with rumors and just reload the save everytime.
But this is assuming its completely random, if the above person is right and you need to attack their parties, it may be just a waste of time, but its an option to consider after attacking them for a while I suppose.