Definately mix up with infarty, otherwise when the enemies finally reach the wall, then your garrison will be slaughtered. I usually use Nord Huscarls, Rhodok Crossbowmen and Swadian Knights/Sarranid Mamlukes for my garrison. Nord Huscarls and Rhodok Crossbowmen are good alone too.
By the way, if you ever get a chance to, then get Almerra Castle as your fief (by capturing it I guess), it's just so easy to defend in sieges as the enemy not only has to walk up a hill while being fired at, but also has to walk INSIDE the castle where the enemy is met by archer/crossbowmen fire from 4 different sides (walls) and met by melee fighters at a ladder which is deeper in the castle.