The commissioner first chosen shall hold office five years, the next four years, the next three years, the next two years, and the last one year. When the term of office of each commissioner expires, a successor shall be chosen for five years. Vacancies in the board may be filled by the remaining commissioners until the next annual meeting. However, a town so voting at an annual town meeting may limit the number of the board to three members and the term of office to three years.