Well, with 50 men, you should be able to take a castle. Problem is, you also need to keep it, meaning placing enough troops in there to prevent some other lord from just waltzing in. Naturally, if 5-6 lords decide to attack, there's not much you can do...
40,000 denars is actually not a whole lot of money. You should stick to winning some tournaments (~40,000 denars each time), being a merchant, or completing quests while increasing your army size before trying to build your own kingdom. I'm on day 100, still a vassal of Rhodoks (!), with 100k denars and 72 troops. I have a castle (safe for now) and 2 villages. I don't expect to leave the Rodoks until at least day 200, I imagine.
Playing at normal difficulty., and took my first castle, as a Rhodok, around day 75.