Taking a city is a matter of attacking over and over again. It's very difficult to take a city in one attempt.
I play on normal difficulty and it was very difficult to finally get a foothold and keep a castle. I must have attacked it 4 times before I actually took it. I lost it 3 times, but every time it was easier to take because when it gets re-stocked the troops are low level. The same goes with the faction armies. At one time I lost 100 (mine) vs. 250 theirs. After I had given them a couple beat-downs I was able to win 100 vs. 350. I was fighting a bunch of farmers when I fought them the 2nd time.
The castle defender numbers were:
first attack (it took 4 assaults before taking the castle): ~130 (difficult to take, lots of sharpshooters)
second attack: ~170 (easy to take, lots of low level guys)
third attack:~220 (easy to take, lots of low level guys)
Another thing to note is that some troops will die much more often than others. I had about 90 slaver chiefs when I was trying to take that castle the first time. They died like flies even with Surgery 12. Now I'm using Swadian knights with the same surgery skill and they rarely die, they get knocked unconscious instead.