The pistols and muskets have dual weapon modes, as ranged weapons or the "hit X and use as melee" mode, like thrown weapons in Warband. The bots are correctly changing to melee weapons when in close quarters, rather than continuing to try and fire (if they didn't that would be a bug), it's just that the weapons they are changing to are also their muskets...the butts of them to be precise. So, they are making the correct decisions (according to the Warband engine).
If you don't like this, you could always open up Morgh's Tools and edit all of the muskets/pistols (open up the item editor, sort for all items of type "crossbow"--yes pistols and muskets are using the Warband crossbow mechanics--and then un-tick the box "next item as melee" for each item). That way, when their AI says they are too close for ranged and should switch to a non-ranged weapon, they will be forced to use another weapon instead of the butt of their musket. Of course, this will remove your ability to use the butt of the same musket as melee, too...
