I'm not really sure what makes the castles increase in prosperity, but villagefarmers make trips to castles, so make sure that you wipe out banditparties near your castle every once in a while..
But.. As you most surely know, you can not do quests for the castle, and you cant build a mill..
Maybe it have something to do with how many soldiers you have in it.. I mean, a lot of soldiers would generate the need of more people, and you could imagine that the castle itself sort of manages the "city in the castle", housing the civilians, and having some traders and outlaying farms and settlements to grow food and such, and you could also imagine the castle to have some form of weapon production or other production. So, you could imagine that the more soldiers you have there, the more the castle will grow..
However, I can not prove this in any way, and I have not tested it in any way. I have 1 castle, it have around 120 soldiers, I've had it for several hundred days.. And it took.. oh... I can make a vague guess.. about 100 days before it became "Average".
I've also asked earlier how high the prosperity of a castle can go, and the answer from the experienced guys was that a castle can get to Average, but never higher.
As for the village/town prosperity, when you say "High" you mean Rich, right? And as for that matter, I also asked how high the prosperity of a village/town could get, and the answer I got was that it could go to "Rich" but not higher.. My villages are only on Average since I don't do much quests for them, to much war going on.. Feel a bit bad about that.. I feel that I neglect my nice little villages
For how long have the castle been "Very Poor"?