Lords would in theory have businesses, and invested in trade routes I believe. So caravans and workshops make sense.
When I got my 2nd castle I just let it get defeated, used the troops that were donated to the garrison in the first place, to improve my own army and strengthen my first castles garrison.
Let castle defences play out by themselves, a garrison of 300 soldiers can actually hold off quite a sizeable army.
I'm using javelins to fund my fun and games though.
Finally, this isn't the final product, economy, paying for castles etc, isn't finished, and until it is, I don't see any issue with using javelins etc to fund your armies. If you don't want to be a smith, try and find a smithing companion? Charcoal alone can be very profitable.