Before 12:00 p.m. of the day following the service of such notice, a person notified as provided in section 3113 of this title shall commence to secure or remove such structure, and shall employ sufficient workers speedily to secure or remove it. If the public safety so requires, and if the mayor and aldermen, selectboard, or trustees, as the case may be, so order, the inspector shall immediately enter upon the premises with the necessary workers and cause such structure to be shored up, taken down, or otherwise secured without delay. Such inspector shall cause a proper fence or boarding put up for the protection of passers-by at the expense of the owner in the manner provided in section 3116 of this title, and such expense may be recovered in a civil action on this statute.