My character is usually lucky enough to be Marshall for my faction and thus is leading the campaign so these comments assume that. If you are not Marshall your options involve attempting to influence whoever is or your liege to behave more rationally.
Also this is all Native, I am not familiar with any mods. As Woopzilla observed, it is the number of troops not the quality that has the greatest effect on the attack vulnerability computation for a fief.
Also NEVER raid villages, which should not be a problem if you are playing as an honorable character. Furthermore I visit every one I have a chance to in my travels and chase off bandits when they are infested. As a result, I can recruit from 80%+ of the villages on the map even if I am at war with their faction.
Whenever practical, I will go to one of my towns and assemble powerful, quick field armies to whittle-down the enemy forces wherever they may be found. It's always nice to take down an enemy ruler as this reduces the chance of a massive attack by that faction. Rulers are the one type of captive that I may be willing to keep in a prison tower despite a ransom offer or two, the others I always accept. If I capture lords with a high relationship I will always let them go with the aim of having a better chance of recruiting them later.
After a successful siege, if I have enough "disposable" troops (rescued prisoners, recruits from the tavern if a town, recruits from nearby villages, other undesirables in my party, and etc.) to garrison the captured fief on a short-term basis, I won't request (assuming that I don't even want it) the fief and garrison it with those and call that well enough. If you do not request the fief after capture you only have one time to load the garrison.
If I do not have enough of such troops and/or do not have the excess troop capacity to shop around and acquire them from towns/villages for a single add -- and would be willing to have the fief awarded to you, I would request it. This will give you the opportunity to continue adding/subtracting from the garrison until the fief is awarded to someone. So you can run around and recruit and build the garrison over a more extended period. Once again, the caveat -- you may end up with the fief so there is that consideration. But if you would be willing to take it this would be the default.
In either case, if the resultant garrison does not give me sufficient confidence, I will ask various lords participating in the current campaign to "go to" the fief. Most of them may even hang around and possibly defend it for a while. For this I give preference to lords without any fief or only hold fiefs which are not endangered by current enemies, or whose fiefs are pretty close-by. Unless planning to end the campaign, I will stash those who are beaten-up with low healthy troop counts here.
If I am planning to besiege another town/castle I will prefer something close by and a ladder-siege if there is any worry about the recent acquisition. Otherwise I will call off the campaign and let the assembled host run around and harass any enemies they encounter and/or protect their holdings.