I tried to disconnect the cable (the signal from the PD) from the spectrum analyzer and the sr560.
The noise is still around 10'000ppm. So the noise doesn't come from some ground loops.

Then I changed the current of the laser diode and rotated the HWP to keep the DC at around 2V, but the noise didn't reduce a lot (still around 7000ppm)

Then I replaced the QWP with a PBS to clean the polarization and the noise reduced by a factor of 10

The alignment changed a bit after replacing the QWP. So the screenshot shows the new calibration scan. The AC signal went from 57mV to 45mV.

Now the noise is around 950ppm. See plot