PROPS: Progressively Private Self-alignment of Large Language Models