I'd advise beefing up your castle and town garrisons a little bit more. You want castles to have about 100 to 150 troops and towns to have about 250-400 troops, depending. I usually put in about 25-50 top-tier infantry at the "top" of the garrison (so they'll spawn at the gap), and about 25-50 top-tier archers or crossbowmen after that. Then I fill out the rest of the garrison with cheap tier-2 troops, either Swadian militia (who, even though they're considered infantry, usually spawn with crossbows) or Rhodok crossbowmen.