To put it another way, if you pack the castle with a large number of low-tier troops, numbers alone should deter most enemies from even trying to attack it, but if they do attack, it probably won't fare well. Using all top-tier defenders will cost you a fortune in wages (even at half price while in the garrison), and it'll still be costly in terms of losses if the castle is attacked, unless you're there personally to assist. The AI can easily replace its losses on fairly short notice (sometimes returning only a few days later with a brand-new high-level army bigger than the previous); your own army will take time and training to recover. A wide range of troops should allow enough numbers to deter most enemy lords from attacking, at an affordable price, while still having some teeth in case someone decides to attack in spite of the numbers. There's usually a big jump in wages from the second-highest tier to the top, so I usually include quite a few troops that still aren't at the highest rank, but are still reasonably effective in combat.
Some of the same factors apply to field armies, where running 30+ Swadian Knights can bankrupt a small country. I prefer to self-limit my use to a max of only 5 Swadian Knights in my field army, plus 5 more Men at Arms (3/4 of the effectiveness at 1/2 the wages), and then add 5-15 Khergit horse-archers of various types at a comparatively cheap wages. Those same Khergits are also pretty effective at defending a castle with arrow fire, as long as you have either a few ranked-up Companions with good gear, Swadian Knights, or high-level Nords holding the enemy back at the ladder head.
Given my aging reflexes, I'm a lot more effective at picking opponents off the ladder from a flanking tower than at hacking and slashing near the top of the ladder. That's generally fine until the opponent throws 900+ besiegers at you, in which case your 100-200 garrison troops and 30-40 man field army may just not be enough. There are times when you just have to collect as many high-tier troops into your field army as will fit, make a run for it, and figure on re-taking the castle after the enemy has gone elsewhere. (Example: Held off 500+ attackers, then destroyed two of the 5 retreating enemy field armies. Less than a week later, they returned with 700+ troops. I lured away one army and destroyed it, then another, leaving about 450 enemies, and then held off the second assault. Less than a week later, they came back with 900+ troops, and I didn't have enough men left to even try to hold the castle.) There is no such thing as "safely protected", as long as there are no long-term consequences for the AI losing army after army.