In a preprint study, researchers found that training a language model with human feedback teaches the model to generate incorrect responses that trick humans.