I would say that one relatively simple way of spicing up settlement management would be to have culture specific buildings in towns, such as Empire towns having the possibility of building Roman (Calradian?) baths for something like garrison morale & prosperity, Battanians towns building Ceilidh Halls for loyalty, Khuzaits towns having the ability to build yurts which decreases housing costs & food consumption (which increases the prosperity softcap) etc.
Also, some actually simple things to add/change that would massively help distinguish castles from towns would be:
- Removing loyalty & security from castles, they're defensive fortifications & they already can't rebel, so why do they have it? Removing them would help castles with their building construction speeds massively
- Adding something like a Sherriff's office that lets castles create a patrol party from garrisoned troops, that patrols the villages surrounding the castle and the trade bound town of the castle villages. Each level of the Sherriff's office (or whatever its called) could increase the patrol size to 30/45/60 troops in a patrol party. This way, castles would actually complement towns instead of being inferior to them in every way (except having a meager reduction to their garrison wages).
I also had a post a while ago about adding artefacts, statues, paintings, unique settlement banners, etc. that can be pillaged from and placed in towns to grant additional effects, inspired by Starsector's colony system.