Overall a great idea!
I think this is a great idea, but in my opinion this can be taken 1 step further.
You are saying that you are able to construct something at very high price. I would say let's start with no price at all.
Just give you the option to settle in the ruins and make some kind of a leveling system.
Level 1:
You can have a small garrison of 20, you have no protection (if attacked it's just like a bandit camp) and if you are in enemy borders lords will send parties to remove you. Also bandit party's will try to raid your encampment.
Level 2:
You can have a normal garrison 40, some basic wooden walls with gapes (no gates) also no platforms for archers to stand on. (for the rest same as lvl 1)
Level 3:
You can have a garrison of 60, some basic wooden walls with platforms for archers (still no gates).
You can assign a companion to run the camp. This opens a small recruitment pool. Also Quest can be implemented to give certain boost to your encampment.
Level 4 (max lvl):
You have a garrison of 80, wooden walls, platforms and gates (easy destructible as you suggested). recruitment pool is a little bit larger.
------
Nice idea BTW to not allow elite men. Maybe take it's a good idea to only allow bandits and/or mercenary's.