I have a fighting force of 477 and 5 unique characters. On average my troops are of medium quality, not Elites.
96 Cavalery
157 Artillery
216 Infantry
25 Horse archers.
My character can hold a maximum of 148 troops. The rest are garision for my town.
And I still don't appear in the Vote for the castle right next to mine, but the guy with 2 other castles and the guy with a faraway castle DO make it into the vote. And I did helpin the sieges.
That's not a fighting force of 477, that's a garrison of 477 minus 148 and a fighting force of 148!

I say this because your troop limit is still low; while my approximate unit size of 400 at the time of secession (note: this INCLUDES 2 policies which effectively grant +80 troop limit for freee, so my pre-monarch limit was around 320) and I can't be sure what the "suggested minimum" is (especially since it's only on PS4 where rendered troops per round is 350 rather than 1,000), I can say it helps to be a one-unit small army that outnumbers anything that isn't a formal enemy army.
But, yeah, your strength is still too small to do something as crazy as secede and hold your ground or otherwise take on all of Vlandia on your own, so biding your time is highly recommended. Leveling Steward and Leadership (Steward can be gotten through a companion quatermaster though) will each give around +80 troop limit by the time they're capped, and elevating your clan tier is about +15 each time unless I'm mistaken.
I don't recommend staffing your troops with too many high tier units though--not yet, at least, since the point of that is to have. a reserve army since they're very expensive--but I do recommend mass-recruiting Vlandian recruits and other tier 1s and tier 2s to fill in the capacity so that Sargot is practically un-attackable for the A.I. unless they have 2,000+ troops at the bare minimum.
The guy with two other castles is probably closer (and maybe poorer in fief prosperity) and helping doesn't contribute to your chances--being the allied commander is what designates you as the "conqueror." Even so, I'm tempted to recommend you prepare to defect so that you're #1 on the docket for pretty much any fief around you, especially Charas and Jaculan, since cities are usually where the real money's at and castles are mainly just useful as pit stops to store noble prisoners, maybe garrison reserve troops, and give away to promoted companions (an action that requires 50 Influence and 25,000 denars if I'm not mistaken, so you can't do it immediately after creating your own country). They do project your power though, and by that I mean it'd make you more likely to get Ortysia and Lageta, but...
Yeah, I think you should grind enemies and tournament prizes for money and defect to one of Vlandia's enemies since the real prizes are to your west, not your east, both in terms of effort to reward ratio and sustainability if you choose to go independent. Otherwise you're probably going to have to wait for all the little castles near Sargot to get captured before you can claim one for yourself, and it's really just a stepping stone to clam Ortysia, though once you have Ortysia it might not be a terrible idea to try to expand south into the Nahasa Desert... but it is a terrible idea to have a territory that's "too wide" since the time it takes to go from one end of it to the other will cost you dearly in multi-front wars.
mangalore:
Your army expenses sound pretty big for someone having no kingdom or multiple cities. I am late game and my main party of 380 elites is at a bit more than 5000 denars.
Only keep low tier units as garrison. Dismiss everyone expensive or take them into your party. The wages also go 2->3->5->8>12 so from tier 3 onward is alot more expensive than tier 1-2 when considering numbers. I usually keep garrison at tier 2 so I have some specialized archers and the infantry guys have shields. That already ups their efficiency for battle tremendously to add meat to your army. Your own party is what should have a hard core of elites. Still it may be worthwhile to limit yourself to 50 elites and not upgrade people just because you can. Leave them in the wings to replace losses and increase your elite core only with your income.
That said, shape your army around your budget and when in doubt your party beats garrison every time.
Although it's good to moderate garrison size normally, he's planning to use Sargot as a staging ground for a rebellion, which means it needs to be large enough that only a mega-huge army would dare attack it and be capable of reconstituting his own unit if he's ever defeated in battle (a very real probability with only 150 troops on his person) and high tier troops, especially if they aren't native to the region, are worth quite a bit in terms of time and opportunity costs since it's not a quick and easy manner to take low tiers and max them out (until you can command an absolute horde of them lol, but that's LATE late game), especially if you're in Vlandia and want elite horse archers from the Khuzaits or something like that.
I'd recommend going crazy with battles to have millions in reserve before adding on to these expenses, however, (except I'd recommend increasing expenses by ~500-1,000 denars per day by capping Sargot with bottom tier units for the purposes of padding it out), since that'll help in the short term. Then, before going independent, Boar Champion could start funding 4 parties (a crazy expensive thing to do, I'll point out to him/her now) so they can run around Vlandia recruiting troops before merging with his own unit to go attacking cities when he goes independent (or defects--which is practically the same thing in the short term, except aided by the nobles of your new country). Good opportunity to win over the Vlandian nobility with a capture-and-release policy as well lol.
mangalore:
Also if you want to stick with Sargot, get workshops there. 100-200 denars a day can pay a good chunk of your garrison.
I would also disagree on the caravans. If your problem is Vlandia being too peaceful they can actually be worthwhile.
Also the worthwhile thing about tournaments aren't the bets, but the victory prize. That can be 5000-10000 (particularly noble mounts). Sell them.
Overall however ponder what you want to do. You can still ditch Vlandia and seek a more action filled kingdom. You can also start your independence war (but your should be able to have a stable companion army of 500+ and money to pay yourself out of wars) or be more aggressive when a war happens. Wars usually pay for themselves so if you say your loot does not suffice, fight more equal battles alone. Battles should rake you in tens of thousands of denars with the ceiling being town merchant budgets, prisoners can regularly end up ín the thousands as ransom as well.
Also a smaller but faster force (more cavalry heavy) may be needed so you can hunt down enemy lords and caravans. Also makes a pretty penny and fighting outside a large army gives you more pieces of the cake.
Not sure if it's good to have all your workshops in one city, but it might hurt if they're different at least!
Caravans can cost upwards of 40,000 to form and can cost many thousands per day trying to reconstitute after a raid; I'm uncertain most peace times last long enough to recoup the losses so it's hard to recommend outside of special circumstances, like your only enemy is the Aserai and you're in the same region as them while everybody doesn't care for years on end.
And noble mounts/armors/weapons can sell from 20,000-50,000, so I doubly push your recommendation. You can tell if it's worth your time by checking it's tier; Tier 6 mounts/weapons/armors are usually in this range while Tier 5 tends to be in the thousands. I can't be bothered to grind anything less in value after the early game, so I won't recommend you do it either.
I don't think Prisoners are worth lugging around for chump change; instead I'd recommend keeping top tier unit prisoners since they can easily become "reserve troops" in effect after a while.
War can make bank, for sure, but I caution trying to go independent with merely 500+ troops, especially in a region like Vlandia where an army of 2,000 is the norm and a second one is likely on the way. Better go for ~1,000 on your own and try to resort to siege defenses for the huge enemy armies and using villages as traps to ambush enemy units who can outrun you, I think.