True, after raiding a lot of them and being disappointed every time I just stopped caring about hideouts, I just ignore them.
Some of them are becoming one of my favourite parts of the game! I like the Sea Raider layout in particular, with opportunities to creep up and snipe enemies near the start and let your infantry take care of the rest, then moving up the beach there are beached longships to use as cover (I had a very satisfying result yesterday when a Sea Raider walked around the end of one and started pulling out a javelin upon seeing me, and I took him out instantly with my crossbow). Then you have the cave system, with pockets of enemies to deal with. I like the flow and the lighting, the general ambience of the Sea Raider camps.
The Forest Bandit one, unless there are very few enemies inside, is still too difficult for my liking (how can I take on two dozen good bowmen in an outdoor environment with about one third as many men, no matter what weaponry my party has?!?!). I enjoy the Desert Bandit map, and to a lesser extent the Mountain Bandit one- the more open outdoor space there is, the less interesting (and the easier it is to lose) it is to me.
Creeping up on enemies and taking a number of them down as fast as you can, without alerting many other enemies, is satisfying for me (though the feasability of this is limited on some maps, such as F.Bandits). When there is some terrain or props to mask your approach (not sure if they actually
see you or not, but crouching and, I think, changing to walking speed lets you get closer) it is fun to get the jump on one or two guys. I've been using a seax recently and find it useful to get stabs in to help my allies if they get into a melee with several enemies, because there is often little time or space to use even a sword.
I don't know if a full siege type event would be more satisfying, but certainly some maps could benefit from more of the indoor cave environments, and more props. Making use of the ruins of the actual fort that is alluded to in the set-up to the quest would allow for a mini-siege, but I don't think it should be on the level of ballistas for either side- firstly because it doesn't make much sense for a bunch of bandits to have the skill or consider it worthwhile getting a ballista (they are not expecting to face massive armies coming, and if they saw them coming they would flee, this isn't a city they want to inhabit permanently), and the player getting large siege weapons up into the hills/into a forest doesn't sound right either. Secondly, in gameplay terms I can't imagine it feeling as anything but a shadow of a full sized siege.
As I've said, I enjoy some of these bandit maps and I like to vary my loadout to try new approaches- sometimes I take a shield but these are generally heavy, so I might dispense with it, so too my spear if I intend to get close to my enemies one-on-one and want as much speed as I can get. I sometimes ditch similar equipment from my companions' inventories too.
If we were to add something like a bandit-siege, I think it should be in addition to the current commando approach. Perhaps one could choose to take the full army up, in which case the bandits would see your approach from afar and, if you aren't so large that they simply decamp and resettle nearby, you get to do the siege, and they will retreat to the old fort ruins and hastily barricade them.
In contrast to the OP's suggestions of siege weapons being used, I would prefer a bandit-siege to be on a small scale; battering down small barricades that they have erected across the ruined sections of the old fort, and finding other ways to get into the top fort, maybe via a precipitous pathway up a cliffside to get over a forgotten gap in the walls. Or there could be cave routes up to the forts (in the case of Sea Raiders and Desert Bandits, these could of course be part of the existing cave system). These would be very narrow, rough-hewn versions of a tower staircase. You could draw the enemies attention with a main assault on the fort, which would leave only a couple of enemies defending these backroutes that I have described, breakthrough and kill the defenders of the front of the fort from their rear. Then the finale could be in a small ruined keep, fighting your way up to an exposed upper level of the old tower where you face the bandit boss.