Ok so not a bug, I just figure after all this time of playing when I run across something... unusual it is a bug.
No enemies to be had in town (in terms of enemy lords) as we are at peace with everyone... There is however a Boyer something or nother (I forget who it is) who down right loathes me... Through some quirk with a fight for a lady he got a -20 bonus three times. Once when I first approached him, again when I started the fight, and finally at t he end leaving me with a -63 relation with him.
I believe that if you walk into a town where a lord with -50 relation is present, there is a chance that there will be a hired assassin waiting for you in the tavern.