End-to-End Learning of Joint Geometric and Probabilistic Constellation Shaping

For the physical layer, deep learning has shown a great potential either to optimize individual digital signal processing (DSP) blocks or, more interestingly, to optimize the whole DSP blocks as a cascade of neural networks (NNs), referred to as an autoencoder. To mention examples of the former approach, NN-based digital pre-distortion has been proposed to pre-compensate the nonlinear impairments of optical coherent transmitters [4,5], outperforming other pre-distortion techniques such as Volterra-series based solutions. At the receiver side, different NN-based equalizations have been proposed to mitigate optical fiber and transceiver nonlinearities [6-8]. Another example is the use of deep neural networks for digital back-propagation with optimized computational complexity [9]. In the end-to-end (E2E) learning approach, the transmitter, channel, and receiver of a communication system are implemented as an autoencoder, jointly trained to best match the outputs to the inputs and thus, to better communicate [10]. In optical fiber communications, the E2E learning has been applied for both intensity modulation with direct detection (IM/DD) systems [11] and coherent systems [12-15]. For the latter, E2E learning is more involved due to the complex interplay of nonlinearity and chromatic dispersion in optical fibers. For coded-modulation systems, most of E2E learning studies focused so far on geometric constellation shaping, e.g.