Hi Boldoo,
I simulated your project again with the fibers and amplifiers off and the constellation came out properly. Although I don’t think it is a traditional 16-PSK constellation it still looks good.
The fibers and amplifiers are the agents which cause the distortion here. Try a DCF to get a better result.
Regards