Hello, looters are spawned from the hideouts, and hideouts are cleared either by lords or by the player. If lords are somehow weakened, then the looters might take the lead and suppress the lords for some time.
We will take a look at the reason why they cannot recruit men easily, and balance it.
In the meantime, you can help the lords by clearing some hideouts.
Thank you for reporting!