They probably just escaped. You can try to maybe use tweabmb to tweak the percentage chance of prisoners escaping in a town with prison tower to zero.(don't know if it still works or not)
They do escape over time. I learned this the hard way on one playthrough when I thought I could save a bunch of Noldor prisoners as a rainy day fund, came back a couple months later and they were all gone. Moral of the story is sell them or free them right away
If the tweakmb option doesn't work, just tweak the amount of prisoners you can have per prisoner management, then drag them with you for eternity, until you decide to sell/free them.