Keep faction power in mind as well. If your castle is deep into friendly territory just keep a small garrison there with just a few elite troops. That should stop individual lords from sieging. You are gonna ride to the aide of your castle from any marshalled force anyway so it really saves money to keep your heartland castles with a low garrison. Give the units to your Lords (If your a monarch) or your militia patrols (in 3.61, there is a bug in 3.6 that won't let you) in order to keep them fighting somewhere without costing you money.
For garrison try not to make them all high end troops, they will eat into your treasury like no one's business. If you have a 200 troop garrison a max of 60 should be top tier with the rest being just below or in the case of Empire ranged, almost completely low tier. Armored Crossbowmen are great in small amounts, Empire crossbowmen are awesome in siege work for a fraction of the cost. Think that way an you should be good.
Quantity over quality for garrisons. Keep a small amount of great and keep mostly good and decent. Avoid mercanaries and mounted units for garrison, way to expensive and most mounted units are crap in siege defense since they love to use lances and lances suck in ground combat.
Also keep in mind you should take enemy lords prisoner if you don't plan on eventually recruiting them. Always take Debauched lords prisoner (they are the ones that just get angry when you let them go). Invest in prisoner management, have a companion invest in looting and have another invest in trade. Sell your prisoners, ransom lords, then sell of all your battle loot. I found that Once I go independent I usually take a castle just to capture the besieging forces and sell them, letting them take the castle after the first failed assault