I had my own faction on day 75 with two castles, and for some reason those buggers targeted me specifically and took both castles ASAP. I was spread too thin to deal with 900 black knights (who wouldn't be?). Even the second siege, I thought I might be able to handle it at 300 of their guys, but even their crappy troops didn't take a single loss from the dozen or so arrows peppering them, nor the dozen knights at the top of the ramp. I didn't kill one guy.
So many many days later they've taken 4 or 5 more castles and I was working for the swadians. Left honorably and took a castle from the khan guys many leagues away from the black knights. They came immediately afterward to take it, this time with only 200 or so. I led them into the queen of swadia with her 400 troops and again went into battle, I've not got elite guys but a great deal of knights and warchiefs and barons, and the queen has 15-20 paladins leading her charge. We sent 400 men to their death and I think killed 2 of the crappier black troops. The paladins got wiped hard and fast. Then I autocalced and we wiped them, which makes me sad to have done. I wanted to go back and undo, but i'd like to try this whole having a castle thing for more than a week.
Do they typically single out the player so vigorously? Don't mean to whine, I expected difficulty, but not near-impossibility.
(EDIT)
Ah and grats on your victory, it sounds epic. I look forward to when I have more elite troops to have a good fight. Just caught unawares is all.