The AI does take your party size, garrison size and troop levels into consideration when making attack/wait/flee decisions, but I've noticed some parties don't realize you're an enemy, even when there is a personal vendetta involved.
That said, I think troop levels are probably why the AI rarely attacks. Right now, in my game, Queen Esmirelda has a 300-troop army with 10 20-man patrols (500 troops total). The problem is, almost all of those units are low level. She has about 30 crossbowmen (including those from the patrols) and 30 same-level infantry, and then all the other units are fresh recruits or peasants. With so few good units in her large army, she knows full well that a smaller, better-trained force would annihilate her. So she's never bothered anyone except those she can beat easily. I started a war with her and even when she started following me on the world map she refused to attack my settlement. She did well enough at killing my patrols, though. The AI is opportunistic: it fights the weak opponents first.
Edit: Also, it's good to keep in mind that the Custow faction is usually at war with the Peasants. So, they have nearly unlimited targets. If you are stronger than the Peasants, the Custow faction will usually choose to ignore you.