Yeah once it gets too low it's kinda up in the air whether it recovers, BUT
Sell the town food, not too much as it will attracted caravans to buy it back up (STUPID BAD DESIGN town should not sell food if it will run out soon.)
Put in a governor that matches the fiefs culture and or has perks that boost security or loyalty.
Put in enough security to start getting +security (it will start to give you + loyalty when higher)
DO quests to remove -security and -loyalty from issues, DON'T DO gang leader quests as they actually give +security while un-solved.
Rant incoming after tranquillizers
For the garrison it's good to put in tier 3 or so troops to get more security per food, every 20 troops eat 1 food. Problem is, if you don't have enough food you
get the -5 loyalty starving malice so it can be a awkward timing to try to get enough food )+4 or 5)to support the 80 or so t3 troops to start security building up.
Also, turn building project off and turn on daily festivals so that as soon as possible the festival will run and help build loyalty.
Now, in current versions it has been reported that festivals won't run with under 25 loyalty, that stinks. I may be wrong but I thought in older version you could always use the festival even at abysmal loyalty. If this is a new change, BAD TW, BAD. I'm not going to load a old version to check though.
Also, if it's a lost cause and loyalty is just not getting better and you have high risk of rebellion and the militia is getting big and good trick is to just pull out all your troops from the garrison and items from your stash and let the rebels spawn, then attack them and re-take the town asap. After the rebels spawn the granary is filled and loyalty is boosted to 100, so if you take the town back quickly you have food and loyalty to have better chance of stabilizing the town.
Of course if you're a vassal it's not as guaranteed as if you do it yourself though.
Also, once you have some loyalty like 40+ build orchards 1 or granary 1, but turn the project off if it doesn't finish building by 25 loyalty, so you can use festival to bring it back up. Repeat this to at least get granary 1 and orchards 1, the high ones are good too, but will take longer.
Anyway, I've requested that we be able to give food directly to the town supply and also to be able to build farms and food supplies on our own for our towns, but no news if TW likes this idea at all. I think the simulation is a dumpster fire and I would much rather TW move away from it and just add more proactive things for the player to do and control. I only want to do things and make choices.
Towns need to store my troops and pay me money. Anything else is bad and annoying. I don't know what TW thinks the fiefs are good for. The only point of having them is the AI doesn't have them anymore.