I started out leaving them in my party too. I had no idea what caravans did, how they worked, and was worried about the inconsistent gold that I had read about. Now that I have my own kingdom, the early parts of fighting other kingdoms is very lopsided, so the money allows me to barter for peace. It might take me 40k to buy peace, but with my caravans, the "swimming in gold" part is very useful.