Campaigners say nesting swifts may have been affected after part of a building was demolished.