While I do generally build a mill in my villages first thing, I've noticed that it's never going to be able to raise the prosperity of a Very Poor village all that high, especially since at some point, the village is going to get raided. It's just inevitable, unless you sit on that village night and day.
Far better to trade, invest in enterprises, fight enemies and sell their loot and ransom their prisoners and, most of all, take more territories so you can get more and better fiefs!