I haven't experienced this particular scenario, but what I have experienced was the following: I joined the Aserai as a Mercenary at first and made my money by obliterating Gang Hideouts, as well as trading. After making quite a good amount of money for myself, I engaged in battles with enemy lords. By taking them Prisoners and executing them, I made quite a few ewnemies. After becoming a Vasal for the Aserai, I have besieged a Castle and looky there, an Army of about 500 strong engages me. So.. you could say you might want to make some enemis, in order for this to happen.