A collector so removing, or the executor, administrator or guardian of a collector who neglects the duties required in section 4671 of this title, shall be liable for the whole amount of such tax bill to the town or other municipality, and shall not have authority to collect such unpaid taxes.