When you take over a castle or town, make sure you have a garrison ready to put in there. Lots of ranged and infantry units will make the siege much easier. I can usually win any siege defense with 150-200 garrison against anything on good/good difficulty, just stand near the ladder with a strong and long weapon and keep swinging.
Swadian Knights are probably the best cavalry, but they are also really expensive to maintain. You're going to definitely want a mix, especially for sieges and defenses, since the efficiency of mounted units during a castle or town battle is really low. I usually ride around with a group of Swadian Knights, but only use infantry and ranged for my castle garrisons or when I'm siegeing.
You can find Sea Raiders near Rivacheg. If you mean recruit them, you first have to take them prisoner, and then use the camp function and ask if any of them want to join your group.
There isn't really much you can do to help out cities, just try and kill any bandits nearby and make sure the villages surrounding it aren't getting looted.
If you leave your faction, you will lose all your stuff. If you "revolt" against it, by asking for a fief you recently captured and being denied, you will keep your stuff.