I play with the difficulty cranked up to 177%, with 100% damage to me and the troops, and a decent Battlesizer setting. I've taken on the Almohad troops, and while I admit that my troops managed to wrest a castle from them during my brief stint as a mercenary for the HRE, I still lost many Teutonic/Imperial troops during the siege, and that was with a numerical advantage. Me and the King had 300 troops, while the enemy had 250. By the time the siege was over, 90 men from my side were dead, and with plenty more injured.
Even with 60% difficulty, I have difficulty (no pun intended) believing that 7 men could take down 60 troops, and much less that a few dozen troops could take a castle which had a garrison of 200. Unless, of course, you stationed all your archers in the right places and repeatedly snipe-and-run the enemy, which doesn't sound likely, since according to your post, you only brought a few Knights with you.