Well, the lack of cavalry surely increase the difficulty of the challenge but still.
Since I gave my oath to the king of Swadia, I have the impression to repeat the same process all the time: build up my army and gain money by attacking caravans and enemy lords them taking a castle or a city. I think that the condition of lord lack of diversity. You could have to quell other lords rebellion, have tournament, etc...
But this topic is only about occasional locations and bandits, so I will stay focused on these subjects.