The trick to getting the special troops is to find "powerful" Notables and increase your rep with them. They're the only ones who get the special troops types in their rolls. You can see their influence level when you mouse over them in the city/village window.
Increasing your rep lets you access more of their troops obviously so you can get more, but since Lords generally don't increase rep with Notables, the higher slots will basically be exclusively yours so you won't have to worry about random Lords sniping all the recruits.
Just gotta find the Powerful guys, butter them up, remember who and where they are and check back periodically. Especially good if there are Powerful Notables in your fiefs because you'll get the passive daily rep gain from high security with them and probably be in the area relatively often when you stop in to your city or castle, and you can buy all their troops even if you don't need them and stick them in your garrison for later.
Should even buy their standard troops if they have them, because getting them out of there will free that slot up for a potential special troops later, otherwise the standard guy will just chill there forever until you or somebody else recruits him.and hog the slot.