Its a superb idea but, lets not kid ourselves, it would take a hell of a lot of coding to make something like this possible.
However, if it was made using sets of buildings...
For example having say four difference types of tower and walls, gates and houses, so four models per set and say 2 or 2 sets depending on which faction you currently work for. As you gain higher ranks the sets could open themselves up to you.
Adding a garrison would be fairly easy, just have soldiers patrolling the walls, and if a seige was encountered, the landscape could be a castle courtyard or the like, maybe inside a great hall.
Just a thought..