After spending a lot of time generating polarization states and rotating the output polarizers I found out that this method is not too precise.
While the overall shape is similar to simulation, there are several points were the fitting is failing and gives polarization rotation 0 or 90 deg and retardation 45deg.
This is especially annoying as the measurement is far longer than expected (poor vi implementation limited measurement speed to 6Hz and need to rotate at least 36 times the ouput polarizer)...
In the end I removed the ouput polarizer and used the polarization camera readout.
THe measurement are agreeing now really well with simulations.
Note that in the simulation an arccos of the retardation limits the retardation to positive values only.
This result can be compared with simulation were I assumed for now same retardance vs voltage and extinction ratio equals to 1.
The rotation and retardation are shown in figures 1 and 2.
The retardation agrees really well with measurement if we wrap the measured retardation to positive values (fig 3).
Forthe rotation, it seems we have some kind of 90deg offset. I'm wondering if it could be because the first LC is rotated by -45deg instead of +45deg.