Yeah, leaving a town for a part of garrison could be risky. But I see it as tactical moment as well. You can also consider it just straight as RP behaviour assuming sometimes garrison reinforcements be doing big mistake leaving the town(castle) and sometimes doing that for a good reason.
- Lets assume a town can help only one village at the same time.
- The trigger for sending reinforcements could be frightened villages coming to town for help. It will get more time to start but anyway it has good chances to reduce the damage and looks more meaningful and systematic than just random friendly troop occasionally walking around. Also after frightened villages came to town, garrison gets info about raiders party size and decides to go or not.
- The main mecanics to save the moving garrison is its own fear in the way it is already realized in the game. If there are mighty troops close to town it will never leave the town, just like any lord does stay in town in the same situation. Also I suggest once moving garrison is frighted it goes straight back to town and doesn't make any other try for this particular raid.
Btw RP wise for garrison it is a matter of life and death too, as after village is get raided it's gonna be food shortage and dieing garrison troops.