There's a level limit for prisoners you can hire. Prisoners above a certain level cannot be hired to your party. Likewise, there are some enemies that cannot be knocked unconscious, mostly troops that are meant to be undead, but also make sure that the mace you're using actually does blunt damage. User error does not mean the system is bad, lol.