Villages from nearby castles also deliver their products to the nearest town of their own faction. If they belong to a different faction, food is missing. Delivering food yourself might help on short term, but in the long run it can reduce the price for food, hence merchants want to buy food from that town, instead of selling it. I played food delivery service for way too long, but it did not help at all. In the end it's best to move the borders of your faction away from that town by conquest and leave the town to its misery. My town still goes through food shortage from time to time but its a lot less severe now that the war has moved to a different region on the map. This also prevents armies from buying all the food there which happend a lot earlier. I hated this starvation thing at first, but its actualy kind of realistic.