I found that all of them only apply on Town, but not castle, except for the one from Steward -5% and other one skill -20% which said only apply to castle.
If the skill from one hand, two hand, polearm ,bow, etc. which saying reduce garrison wages. It needed to be clearly saying it only apply to Town garrsion.