When I play, I roam around killing lords with an all-faction-troops army, and keep all rescued soldiers in my town.
When I want to besiege something, I take out all the huscarls, sergeants, knights, mercs, etc. from my castle and just set them loose. Easy.
Villianese, Zerrikanians and Antarians are all fairly rubbish at sieges by themselves (haven't played the other factions yet). (Antarians are good if you have the battle-sizer set low, and vills are good if you retreat a few times, but I have 250 battle size and never retreat if I can avoid it so they both suck for me. Zerriks aren't too bad, the axemen are surprisingly good, but the reapers are annoying only knocking people out. If you're forced to retreat, all their soldiers heal and you have to fight them again).
So basically I just find native troops much more effective for taking castles. If you really need some, though it's sorta cheating, set recruitment to native and build up an army of huscarls. Certainly a lot cheaper than black army guys.
Also, Cyclohexans, I've been using Conquistadors in my current game. They're not too bad at all in the field, but it seems people only use mercs for sieges, and in that respect the black army is definitely the best.