Whenever the Zoning Administrator determines that a nonconforming tree or structure has been abandoned or more than 50 percent torn down, physically deteriorated, or decayed, no building permit shall be granted that would allow such structure or tree to exceed the applicable height limit or otherwise deviate from the zoning regulations.