When the punishment of the crime shall be the confinement of the criminal in the penitentiary, the expenses shall be paid out of the State Treasury, on the certificate of the Governor and warrant of the Auditor of State; and in all other cases they shall be paid out of the county treasury in the county wherein the crime is alleged to have been committed. The expenses shall be the fees paid to the officers of the state on whose governor the requisition is made, and not exceeding cents a mile for all necessary travel in returning such prisoner.