Pretraining Produces Misaligned Models

The initial training of LLMs on vast amounts of internet text results in models that absorb harmful content, biases, and can leak private information. Current methods for filtering this data before training are insufficient and can even worsen some biases.