The road commissioners shall be subject to removal for cause by the judge of the circuit court or chancellor of the county, due notice having been given of the charges preferred.