The Department may remove from office any inspector or judge of election or clerk appointed by it for want of requisite qualifications or for cause. Such removal, unless made while such person is actually on duty on a day of election and for improper conduct as an inspector or judge of election or clerk, shall only be made after notice which shall set forth clearly and distinctly the reasons for the removal.