1

Model-Based End-to-End Learning for WDMSystems With Transceiver Hardware ImpairmentsJinxiang Song, Student Member, IEEE, Christian Hager, Member, IEEE, Jochen Schroder, Member, IEEE,

Alexandre Graell i Amat, Senior Member, IEEE, and Henk Wymeersch, Senior Member, IEEE

(Invited Paper)

Abstract—We propose an autoencoder (AE)-based transceiverfor a wavelength division multiplexing (WDM) system impairedby hardware imperfections. We design our AE following thearchitecture of conventional communication systems. This enablesto initialize the AE-based transceiver to have similar performanceto its conventional counterpart prior to training and improvesthe training convergence rate. We first train the AE in a single-channel system, and show that it achieves performance improve-ments by putting energy outside the desired bandwidth, andtherefore cannot be used for a WDM system. We then train theAE in a WDM setup. Simulation results show that the proposedAE significantly outperforms the conventional approach. Morespecifically, it increases the spectral efficiency of the consideredsystem by reducing the guard band by 37% and 50% for a root-raised-cosine filter-based matched filter with 10% and 1% roll-off,respectively. An ablation study indicates that the performancegain can be ascribed to the optimization of the symbol mapper,the pulse-shaping filter, and the symbol demapper. Finally, weuse reinforcement learning to learn the pulse-shaping filter underthe assumption that the channel model is unknown. Simulationresults show that the reinforcement-learning-based algorithmachieves similar performance to the standard supervised end-to-end learning approach assuming perfect channel knowledge.

Index Terms—Autoencoders, deep learning, digital signal pro-cessing, end-to-end learning, reinforcement learning, wavelength-division multiplexing.

I. INTRODUCTION

The ever-growing demand for higher data rates drives therapid development of optical fiber communication systems.One of the most important developments is wavelength divi-sion multiplexing (WDM) transmission, where parallel datachannels are transmitted on different wavelengths simultane-ously. The throughput of modern WDM systems often exceedstens of Tb/s with more than 100 channels [2]. However, theoverall bandwidth of fiber systems is limited by the bandwidthof erbium-doped fiber amplifiers (EDFAs) that periodicallyamplify the signals along the fiber link [3]. Optimizing thespectral efficiency (SE), i.e., the number of bits that can be

Parts of this paper have been presented at the Optical Fiber CommunicationConference and Exhibition (OFC), San Diego, California, USA, 2021 [1].

This work was supported by the Knut and Alice Wallenberg Founda-tion, grant No. 2018.0090, and the Swedish Research Council under grantNo. 2018-0370. (Corresponding author: Jinxiang Song)

Jinxiang Song, Christian Hager, Alexandre Graell i Amat, and HenkWymeersch are with the Department of Electrical Engineering, ChalmersUniversity of Technology, 41296 Gothenburg, Sweden (emails: {jinxiang,christian.haeger, alexandre.graell, henkw}@chalmers.se).

Jochen Schroder is with the Department of Microtechnology andNanoscience, Chalmers University of Technology, 41296 Gothenburg, Sweden(email: jochen.schroeder@chalmers.se)

transmitted per unit time and frequency, is therefore crucial tofurther increase the throughput of fiber optical systems.

Over the last decade, most works have focused on in-creasing the per-channel SE via advanced modulation formatsusing coherent detection. The fiber nonlinearity and hardwareimpairments, such as the effective number of bits (ENOBs)of the digital-to-analog converter (DAC), however, severelylimit the per-channel SE. Furthermore, spectrum gaps betweenindividual channels, which are often referred to as guardbands, waste significant bandwidth and limit the overall systemthroughput. Hence, the guard bands between channels needto be minimized. The most promising solution has been theapplication of flexible grids, which allows for transmissionwith flexible channel bandwidths thus enabling simultaneoustransmission of mixed bit rates [4] and allowing to reduce SEloss from guard bands for optical filtering.

To minimize the guard bands between channels, it is com-mon to employ pulse shaping to create a near-rectangularspectrum in the frequency domain with a bandwidth close tothe symbol rate. However, in practice, generating a rectangularspectrum is difficult due to the finite pulse-shaping (PS) filterand transceiver hardware impairments, requiring computationexpensive digital signal processing to eliminate performancedegradation caused by inter-channel interference (ICI) [5].Guard bands therefore remain a major contributor to SE lossin WDM systems.

In recent years, the rapid improvement of machine learningtechniques has led to a resurgence of interest in applyingdeep learning techniques for communication systems [29],[30]. Most work has focused on supervised learning for aspecific functional block, e.g., modulation recognition [31],carrier recovery [32], and fiber nonlinearity mitigation [33],with the aim of finding better performing (or less complex) al-gorithms by replacing the conventional model-based methodswith neural networks (NNs). In contrast to focusing on specificfunctional blocks, end-to-end learning has been proposed todesign the transmitter and receiver jointly [6]. The key ideais to interpret the transceiver design as a reconstruction task,whereby the transmitter and the receiver can be implementedas an autoencoder (AE) and thus jointly optimized. Thismethod has led to several applications for both wireless [6],[9], [10] and optical communications [21], [22], [28]. A broad,but non-exhaustive overview of existing work is listed inTable I. We observe that (i) a majority of works relate towireless rather than optical communication; (ii) geometricconstellation shaping for different channels and applications

arX

iv:2

111.

1451

5v1

[ee

ss.S

P] 2

9 N

ov 2

021

2

TABLE I: Applications of end-to-end AE-learning in communication systems

Ref. year Application isolated ch. ICI ch. sim. exp. Description

Wireless

[6] 2017 geom. shaping X X const. mapper/demapper training over an AWGN channel[7] 2017 geom. shaping X X const. mapper/demapper training for PAPR reduction[8] 2017 geom. shaping & precoding X X MIMO precoding/decoding[9] 2017 geom. shaping X X X mapper/demapper training in sim., demapper tuning in exp.

[10] 2018 geom. shaping X X mapper/demapper training for OFDM system[11] 2018 geom. shaping X X mapper/demapper training over a learned channel via GAN[12] 2018 geom. shaping X X X mapper/demapper training without knowing the channel model[13] 2019 geom. shaping X X mapper/demapper training for PAPR reduction[14] 2019 joint channel/source coding X X Joint channel and source coding/decoding[15] 2019 geom. & prob. shaping X X Joint geom. and prob. shaping/demapping[16] 2020 geom. shaping & coding X X X mapper/demapper learning and error correction code design[17] 2020 geom. shaping X X mapper/demapper training for OFDM and multi-user system[18] 2021 geom. shaping X X mapper/demapper training for OFDM system[19] 2021 geom. shaping & waveform X X X Joint transceiver training

Fiber optic

[20] 2018 geom. shaping & waveform X X X Joint tranceiver learning for IM/DD system[21] 2018 geom. shaping X X mapper/demapper training for the nonlinear fiber channel[22] 2018 geom. shaping X X mapper/demapper training for the fiber channel[23] 2019 geom. shaping X X mapper/demapper training for optimizing GMI[24] 2019 geom. shaping & waveform X X X Joint transceiver learning for IM/DD system[25] 2020 geom. shaping X X mapper/demapper training for optimizing GMI[26] 2020 geom. shaping & waveform X X Joint transceiver learning for single channel transmission[27] 2020 Prob. shaping X X Prob. shaping[28] 2021 geom. shaping X X mapper/demapper training for varying SNR and laser linewidth[1] 2021 geom. shaping & waveform X X X Joint transceiver design for superchannel systems

This work 2021 geom. shaping & waveform X X X Joint transceiver design for densely-spaced WDM systems

isolated ch.: the channel does not suffer from ICI; ICI ch.: channel that suffers from ICI.

has been the main focus; (iii) there are few experimentalvalidations. Studies that also learn waveforms and equalizersare limited to [19], [20], [26]. In [20], the whole transceiver isimplemented as an AE, and transmission is demonstrated overa short-haul intensity modulation/direct detection (IM/DD)system. However, the NN in [20] is used as a “black-box”and it is difficult to interpret the learned solution. In [26], thetransmitter is implemented as a trainable constellation mappercombined with a trainable PS filter, and it is shown that the PSfilter can be learned to mitigate chromatic dispersion and Kerreffect. An explicit low-pass filter is used to reduce informationloss and thus avoid out-of-band (OOB) emissions. A relatedapproach has recently been applied in [19], where flexibleconstellations and waveforms for wireless dispersive channelsunder OOB power leakage constraints were learned.

In this paper, we apply end-to-end learning to an multi-channel WDM system. Similar to [19], [26] (for a single-channel system), we consider designing several transceiverblocks —constellation mapper, PS filter, digital pre-distortion(DPD), and demapper—jointly. Such an AE design incorpo-rates the expert domain knowledge of conventional commu-nication systems and therefore allows for (i) training speedimprovements via meaningful parameter initialization and (ii)performance gain explanation through an ablation study. Themain contributions of this paper are:

• We propose a novel end-to-end AE for WDM transceiverswith non-ideal DAC and in-phase and quadrature mod-ulator (IQM). We decompose the transmitter NN into aconcatenation of small (simple) NNs, each correspondingto a functional block of a conventional communicationsystem. Our approach differs from [19], [26] in termsof the considered hardware impairments and how OOBemissions are accounted for: instead of a low-pass fil-ter [26] or a constraint [19], we show that the AEautomatically learns to avoid/adapt OOB emissions to

minimize the end-to-end loss.• We highlight the potential pitfalls when using end-to-

end AE-learning for designing hardware-impaired com-munication systems. In particular, we show that whenthe AE is trained for a single-channel system, it achievesperformance improvements by putting energy outside thedesired signal bandwidth, which would cause large ICI inWDM systems when the channels are closely spaced. Wedemonstrate that if the AE is instead trained with threechannels, it learns to limit ICI while still outperformingthe considered baseline. However, care must be taken forthe sampling rates or bandwidths used during training tomatch experimental constraints to avoid unrealistic gains.

• We conduct a thorough ablation study and show thatthe performance improvement of the AE-based system isascribed to the optimization of the constellation mapper,the PS filter, and the demapper. Therefore, we show thatour proposed method increases the interpretability com-pared to conventional AE-based systems. Additionally,we provide reproducible open-source implementations ofour AEs and benchmark scheme.1

• We extend the model-free training algorithm proposedin [12], [34], so that the reinforcement learning (RL)based transmitter training algorithm can be applied totrain the PS filter, for which memory effects need to beconsidered. The resulting training algorithm is shown toachieve similar performance to the standard end-to-endlearning approach assuming a perfect channel model. Thisopens the door toward experimental implementation ofthe proposed AE.

The remainder of this paper is structured as follows. InSection II, we give a brief introduction to DL basics and theconcept of AE-based communications. Then, in Section III, we

1The complete source code to reproduce all results in this paper is availableat https://github.com/JSChalmers/AE-Based-WDM-Transceivers.

3

Transmitter Receiver

mk ∈ M map

“one-hot”

0

1

0...0

...

......

...

...

Nor

mal

izat

ionz1

···

z2N P(y|x)

x1

···

x2N

...

y1

···

y2N ......

...

0.01

0.89

0.02...

0.01

a

rgmax

mk

Fig. 1: Example of an AE-based communication system, where the transmitter and receiver are implemented by a pair of fully connected NNs.

introduce the generic setup of closely-spaced WDM systemsand the main hardware limitations. Section IV introduces theproposed AE-based WDM system and simulation results areprovided in Section V. Finally, the paper is concluded inSection VI.

Notation: Z, R, and C denote the sets of integers, realnumbers, and complex numbers, respectively. Column vectorswill be denoted with lower case letters in bold (e.g., x), withxn referring to the n-th entry in x, and x(L)

n denotes thecolumn vector consisting of (n−L)-th to (n+L)-th elementsof x; |·| returns the absolute value of a real number, and|={x}| and |={x}| return the absolute value of the real andimaginary part of each element in x, respectively; (·)> and(·)H denote transpose and conjugate transpose, respectively.Matrices will be denoted in bold capitals (e.g., X), and INdenotes identity matrix of size N ; [a, b]M is the M -foldCartesian product of the interval [a, b]. Lastly, E{·} denotesthe expectation operator.

II. DEEP LEARNING AND AUTOENCODER-BASEDCOMMUNICATION SYSTEMS

In this section, we start by reviewing the general theorybehind DL, followed by a brief introduction to the conceptof AE-based communication systems. Then, we introduce thetraining of AE-based communication systems under two as-sumptions: (i) the channel model is known and differentiable,and (ii) the channel model is unknown or not differentiable.

A. Neural Networks and Gradient-Based Learning

1) Feedforward NN: A feedforward NN with K layers isa parametric function f(r0;θ) : RN0 → RNK that maps aninput vector r0 ∈ RN0 to an output vector rK ∈ RNK throughK sequential processing steps according to

rk = fk(rk−1;θk), k = {1, . . . ,K} , (1)

where fk(rk−1;θk) : RNk−1 → RNk is the mapping carriedout by the k-th layer. Here, the mapping of the k–th layeris defined by the set of parameters θk, and the entire NNis defined by θ = {θ1, . . . ,θK}. A commonly used type offeedforward NN is the fully connected NN in which all layershave the form

fk(rk−1;θk) = σ(Wkrk−1 + bk) , (2)

where Wk ∈ RNk−1×Nk is a weight matrix, bk ∈ RNk isa bias vector, and σ(·) is a point-wise activation function.Hence, the set of trainable parameters of the k-th layer is θk ={Wk, bk}. An example of a fully connected NN is shown inthe transmitter and the receiver in Fig. 1.

2) Gradient-based learning: Training of the NN can beperformed in an iterative fashion with data-driven gradient-based optimization methods. Given a set of labeled trainingdata D ⊂ {X ×Y}, where X and Y are the input and outputalphabets, the training objective is to find the set of parametersθ such that the average loss

LD(θ) =1

|D|∑

(x,y)∈D`(f(x;θ), y) (3)

between the NN output y = f(x,θ) and the true label y ∈ Yis minimized. Here, |D| is the size of the training data set and`(f(x;θ), y) is the per-example loss function associated withreturning the output y = f(x;θ) when y is the true label.In practice, when the training data set D is large, computingthe gradients of the average loss over the whole training dataset is computationally expensive, and the parameter set θis commonly optimized by using stochastic gradient descent(SGD) or its variants as follows. For each training iteration t,a minibatch Bt is sampled from D. Then, the parameter set θis updated according to

θt+1 = θt − α∇θLBt(θt), (4)

where α > 0 is the learning rate. In practice, SGD some-times suffers from slow convergence rate due to problemslike small gradients at suboptimal values of θ. To improvethe convergence rate of SGD, many variants of SGD usingmomentum [35] or adaptive learning rate [36] have beenproposed.

B. End-to-End AE Learning-Based Communication Systems

1) AE-based Communication Systems: End-to-end learningof AE-based communication systems was originally proposedin [6], where the transceiver for a given channel with channellaw p(y|x) is implemented by a pair of NNs fτ : M →CN and fρ : CN → [0, 1]M . Here, M = {1, . . . ,M} is themessage set, N is the number of complex channel uses, and τand ρ are the sets of trainable NN parameters. Fig. 1 depictsthe general setup of an AE-based communication system.

4

Transmitter: Given a message mk ∈ M, it is first encodedas an M -dimensional “one-hot” vector, where the mk-thelement is 1 and all the others are 0.2 Then, the transmitter NNtakes this “one-hot” vector as input and generates a vector of2N outputs xk = fτ (mk), where the 2N outputs correspondto the real and imaginary part of the transmitted vectors. Theaverage transmit power constraint E{||xk||2} ≤ NPT , wherePT is the average transmit power per channel use, is enforcedby a normalization layer [6].

Receiver: The symbol xk is sent over the channel in Ncomplex channel uses, after which yk is observed at thereceiver. The receiver NN processes the received vector yk bygenerating an M -dimensional probability vector qk = fρ(yk),where the components of qk can be interpreted as the es-timated posterior probabilities of the messages. Finally, thetransmitter generates the estimate of the transmitted messageaccording to mk = argmaxm[qk]m, where [x]m returns them-th element of x.

2) End-to-End Training With a Known Channel Model:To optimize the transmitter and receiver parameters, it iscrucial to have a suitable optimization criterion. Due to thefact that the optimization relies on the empirical computationof gradients, a criterion like block error rate (BLER), i.e.,Pr{mk 6= mk}, cannot be used directly (as the BLER is notdifferentiable). Instead, a commonly used criterion is the cross-entropy loss [6], defined by

L(τ ,ρ) = −E{log[fρ(yk)]mk}, (5)

where the dependence of L(τ ,ρ) on τ is implicit through thedistribution of the channel output yk, which is a function ofthe channel input fτ (mk).

The transmitter and receiver parameters are optimized inan iterative fashion as follows. In each training iteration t,the transmitter maps a minibatch of |Bt| randomly chosenuniformly distributed training examples to symbols and thensends them over the channel. The receiver takes the chan-nel observations y1, · · · ,y|Bt| as input and generates |Bt|probability vectors fρ(y1), · · · , fρ(y|Bt|). Finally, the receivercomputes the empirical cross-entropy loss associated with the|Bt| training examples according to

LBt(τ ,ρ) = −1

|Bt|

|Bt|∑

k=1

log[fρ(yk)]mk , (6)

and the transmitter and receiver parameters are optimizedfollowing (4). This training process is repeated iteratively untila certain criterion is satisfied (e.g., a fixed number of trainingiterations, or a fixed number of iterations during which theloss has not significantly decreased).

3) Training Without a Channel Model: In case the channelis unknown or not differentiable, e.g., an experimental channel,the transmitter optimization becomes challenging due to thefact that the gradient of the instantaneous channel transfer

2The “one-hot” encoding is the standard way of representing categoricalvalues in most machine learning algorithms [37] and facilitates the minimiza-tion of the symbol error rate (SER). However, the dimension of the “one-hot”vector grows exponentially with the number of bits in each message andtherefore increases the NN size. Alternative embeddings [38] and multi-hotsparse categorical cross-entropy loss can be used to alleviate this problem.

function is unknown, thus hindering the numerical compu-tation of the transmitter gradients. One way to circumventthis limitation is to first learn a surrogate channel model,e.g., through supervised learning [39], [40] or an adversarialprocess [11], [41], and use the surrogate model to train thetransmitter. However, the performance of the resulting systemseverely degrades if the surrogate model deviates from the realchannel. A different approach based on a stochastic transmitterwas proposed in [12], [34]. For this approach, the transmitteris regarded as an RL agent, and the transmitter and receiverare optimized in an alternating fashion which we review next.

Receiver training: The receiver training is similar as be-fore. However, this time, the transmitter parameters τ areassumed to be fixed. At each training iteration, the transmit-ter maps a minibatch of |Bt| uniformly distributed trainingexamples to symbols and sends them over the channel. Thereceiver takes the channel observations y1, . . . ,y|Bt| as inputand generates |Bt| probability vectors fρ(y1), . . . , fρ(y|Bt|).Then, the receiver takes one optimization step according toρt+1 = ρt−α∇ρLBt(τt,ρt), where τt is fixed during receivertraining. This training process is repeated iteratively until acertain stop criterion is satisfied.

Transmitter training: For the transmitter optimization, thereceiver parameters are assumed to be fixed. At each trainingiteration, the transmitter performs the symbol mapping asbefore. In order to allow for the transmitter gradients com-putation, a small Gaussian perturbation is applied such thatx = x + w, w ∈ CN (0, σ2

pIN ), is sent over the channel.Therefore, the transmitter can be interpreted as stochastic andis described by

πτ (xk|mk) =1

(πσ2p)N

exp

(−||xk − fτ (mk)||22

σ2p

). (7)

Based on the received channel observations, the receivercomputes per-example losses `k = − log([fρ(yk)]mk), andsends them back to the transmitter. Finally, the transmitter pa-rameters τ are updated according to τt+1 = τt−α∇τLBt(τt),where ∇τLBt(τ ) is approximated by

∇τLBt(τ ) =1

NT

NT∑

k=1

`k∇τ log πτ (xk|mk), (8)

for which a theoretical justification can be found in [34].Similar to the receiver training, the transmitter learning processis repeated iteratively until a certain stopping criterion issatisfied. Then, the alternating optimization continues againwith the receiver learning.

III. WDM SYSTEM AND MAIN HARDWARE LIMITATIONS

A. System Model

Fig. 2 illustrates the considered WDM system. For eachchannel, a sequence of |Bt| messages m ∈ M|Bt|, whereM = {1, . . . ,M}, are mapped individually to constellationpoints according to a constellation C ∈ CM , to form thesequence of baseband symbols x ∈ C|Bt|. The basebandsymbols x are then upsampled to get u ∈ C|Bt|R, after whicha PS filter is applied to get the discrete-time baseband signals

5

Tx-N

· · ·Tx-2Tx-1

mapper ↑ PS DPDm x u s

DAC

DAC

linearPA

linearPA

y

ℜ{y}

ℑ{y}

ZI(t)

ZQ(t)π/2

MZM

E(t)

VI(t)

VQ(t)

Coupler

+

n(t)

Dec

oupl

er

Rx-N

· · ·Rx-2Rx-1

ADC MF ↓ demappery u x m

Fig. 2: Block diagram showing the conventional WDM system (↑: upsampling, ↓: downsampling). The mapper, DPD, and demapper operate on each individualentries of the input sequence, while the PS filter operates on a sequence of 2L1 +1 signals, where 2L1 +1 is the PS filter taps. Note, to allow close channelspacing close to the symbol rate, we assume that there are no optical filters or multiplexers.

s ∈ C|Bt|R, where R is the upsampling rate.3 To mitigate theperformance degradation caused by the hardware imperfec-tions (in this paper ENOB of the DAC and IQM nonlinearity),a DPD algorithm is applied. Then, the real and imaginarypart of the pre-distorted signals y ∈ C|Bt|R are separatelyfed to the DACs of the in-phase and quadrature branches.Finally, the DACs outputs ZI(t) and ZQ(t) are separatelyamplified to drive the IQM, where the driving voltages of thein-phase and quadrature branches are denoted by VI(t) andVQ(t), respectively. Similar to [42]–[44], the channel modelwe consider in this paper is restricted to a back-to-back setup,and only additive white Gaussian noise (AWGN) with constantpower is added to simulate the noise introduced by the boosteramplifier. At the receiver, the received signals are passedthrough an analog-to-digital converter (ADC), after which thedigitized channel observations y ∈ C|Bt|R are convolved witha matched filter (MF) and then down-converted with rate R.Finally, the downsampled signals x ∈ C|Bt| are individuallymapped to the estimates m ∈ M|Bt| of the transmittedmessages.Note that, as optical filters and multiplexers wouldprevent close channel spacing due to their finite response, weassume that channels are combined using broadband passivecouplers. Thus, there are no optical filters in our system; sucha system is often referred to as superchannel system.

IQM Model: The coherent optical transmitter used for high-order modulation schemes such as M–QAM, M–PAM is oftenbased on a dual parallel Mach-Zehnder modulator (MZM).For an ideal dual parallel MZM biased at the null point, it hasbeen shown that its transfer function becomes [45]

E(t) = E0

[sin

(πVI(t)

2Vπ

)+ j sin

(πVQ(t)

2Vπ

)], (9)

where E0 is the amplitude of the magnitude of the electricfield, Vπ is the required voltage difference to switch ON/OFFthe modulator, and VI(t) and VQ(t) are the driving voltageof the in-phase and quadrature branches, respectively. Theintrinsic sinusoidal form of the MZM leads to strong signaldistortions when driving with a high peak voltage Vp, whichmust be compensated, e.g., by pre-distortion with an arcsinfunction. Alternatively, one can use a low-driving voltage tooperate in the near-linear regime of the modulator. However,this significantly increases the modulator loss, which results

3An upsampler with N× upsampling rate increases the sample rate byinserting N − 1 zeros between samples. Moreover, the transients from theconvolution operation, eg., PS filtering and matched filtering, are assumed tobe removed.

in a degraded optical signal-to-noise ratio (SNR) after addingthe booster amplifier noise.

PA Model: The power amplifier (PA) used for amplifyingthe DAC outputs behaves as a nonlinear memory system, i.e.,the PA output at any time instant depends on the currentinstantaneous input as well as the inputs at previous timeinstances. Denoting the memory depth by L, the PA denotedby fPA : RL+1 → R, can be defined by

V (t) = fPA(Z(t), . . . , Z(t− L)), (10)

where fPA is a nonlinear function and Z(t) is the DAC outputof the real/imaginary branch. For an ideal PA without memoryeffect, its transfer function becomes V (t) = GZ(t), where Gis the PA gain.

DAC Model: DACs used for high-bandwidth optical com-munications typically have low resolution. Currently, deviceson the market provide 8 nominal bits. However, due to thesampling and jitter effects, the noise introduced by quantiza-tion is usually enhanced. One parameter to assess the amountof noise introduced by the DAC is the ENOB, which is definedas [46]

ENOB =SNDR(dB)− 1.76

6.02, (11)

where the signal-to-noise-plus-distortion ratio (SNDR) is ameasurable quantity, and is typically around 35 dB. Typi-cally, high-speed DACs with 8-bit nominal resolution can betranslated into ENOB ≤ 6 for operation within the devicebandwidth. However, it should be noted that ENOB is avarying quantity and it changes over frequency. In this paper,for the sake of simplicity, the ENOB is assumed to be constantover the considered bandwidth and is set to 6.4 We model theENOB noise introduced by the DAC as AWGN with variancedetermined by the ENOB of the device [47]

σ2q =

1

12

(Epeak

2ENOB−1 − 1

)2

, (12)

where Epeak = max(max(|<{y}|),max(|={y}|)) is the peakamplitude of the input signals. Note that the finite bit-resolution of the DAC limits the strength of the arcsin-basedpre-distortion that can be applied, because it increases thepeak amplitude, thereby resulting in higher noise. Therefore,there exists an optimum DAC driving voltage which balancesSNR degradation from MZM losses when driving in the linear

4This is a reasonable assumption for current generation transceivers.

6

Tx-N

· · ·Tx-2Tx-1

NN1 ↑ NN2 π(·|s) NN3m x u s s

DAC

DAC

linearPA

linearPA

y

ℜ{y}

ℑ{y}

ZI(t)

ZI(t)π/2

MZM

E(t)

VI(t)

VQ(t)

Coupler

+

n(t)

Dec

oupl

er

Rx-N

· · ·Rx-2Rx-1

ADC MF ↓ NN4

arg

maxy u x q m

Fig. 3: Block diagram showing the end-to-end AE-learning based WDM system (↑: upsampling, ↓: downsampling). The trainable components are highlightedin yellow. NN1, NN3, and NN4 operate separately on each entry of the input sequence, while NN2 takes a vector of length 2L1 + 1 signals as input, where2L1 + 1 is the PS filter length.

regime of the modulator and SNR degradation from limitedcompensation of MZM nonlinearity when driving at highvoltages.

IV. PROPOSED END-TO-END WDM SYSTEM

In this section, we start by introducing the proposed AEimplementation for the WDM system. The symbol rate andmodulation formats are assumed to be the same for allchannels, and we consider using the same AE configurationsfor all channels.

A. Autoencoder Design

In principle, the entire transmitter and receiver can beimplemented as an AE and trained by end-to-end learning asproposed in [6]. However, this leads to:

(a) Difficulty in interpretation: In contrast to conventionalcommunication systems, where the performance of eachtransmitter/receiver blocks can be measured separately,the AE implementation is a “black-box”, and it is hard tointerpret the learned solution and to quantify the originof the performance improvement.

(b) High training complexity: The transmitter needs to per-form several tasks, such as symbol mapping, PS, andpre-distortion jointly, and learning the transmitted wave-form involves sequential input data, which significantlyincreases the NN size with the “one-hot” encoding beingapplied, therefore increasing the training complexity.

(c) Parameter initialization: It is difficult to know whichparameter choice leads to good performance prior totraining, and random parameters initialization can slowdown or even completely stall the convergence pro-cess [48].

To address these issues, we design our AE following thearchitecture of conventional communication systems as shownin Fig. 3. The policy π(·|s) can be ignored for now. Thetransmitter NN is decomposed into a concatenation of threesimpler (small) NNs, each corresponding to one functionalblock of a conventional communication system. By doingthis, the parameters of these NNs can be initialized such thatthey initially perform close to their conventional counterparts.Moreover, the “block-wise” transceiver NN design allows foran ablation study and therefore makes it possible to partiallyexplain the learned solution. As a result, the proposed scheme

has decreased training complexity and increased interpretabil-ity as compared to a conventional AE.5

1) Transmitter: At the transmitter, the symbol mapper, thePS filter, and the DPD of the conventional communicationsystem are replaced by three NNs. We denote these three NNsby fθ1(·), fθ2(·), and fθ3(·), where θ1,θ2, and θ3 are thesets of trainable parameters. We define these three NNs in thefollowing:

(i) NN1 fθ1 : M → C maps each message mk ∈ m to aconstellation point according to xk = fθ1(mk), wherean average power constrain E{|xk|2} = 1 is enforced.

(ii) NN2 fθ2 : C2L1+1 → C generates each of the pulse-shaped baseband signals according to sk = fθ2(u

(L1)k ),

where u(L1)k = [uk−L1 , . . . , uk+L1 ]

>. Here, NN2 onlyhas a single layer applying a linear activation functionand can be interpreted as a standard finite impulseresponse (FIR) filter. Therefore, the generation of thepulse-shaped signal can be described by sk = θ>2 u

(L1)k .

(iii) NN3 fθ3 : C → C generates each of the pre-distorted signals according to yk = fθ3(s

′k), where

fθ3(·) operates separately on the in-phase and quadra-ture branches, and −1 ≤ <{s′k},={s′k} ≤ 1is obtained by normalizing sk according to s′k =sk/max{max{|<{s}|},max|={s}|}}, where s is thepulse-shaped signal sequence.

2) Receiver: At the receiver, only the symbol demapper isreplaced by an NN, denoted by NN4 fθ4 : C → M, whichmaps each of the downsampled signal yk to the estimate ofthe transmitted message as described in Section II-B1. We notethat, in principle, the MF can also be implemented by an NN.In a real system, however, the MF is usually implemented aspart of the adaptive equalizer, and we therefore have left it outof this discussion.

B. Learning With a Channel Model

Assuming that all transfer functions of the components inthe considered system are known and differentiable, the systemcan be optimized via standard end-to-end AE-learning [6]

5We note that such an AE implementation can potentially lead to perfor-mance degradation compared to the conventional AE, which we do not studyin this paper.

7

TABLE II: NN parameters

NN1 fθ1 NN2 fθ2 NN3 fθ3 NN4 fθ4

layer input hidden output input hidden output input hidden output input hidden output

(i)# of layers - 3 - - 0 - - 3 - - 2 -

# of neurons M 50 2 201 - 1 - 50 - 2 20 Mact. function - ReLU Linear - - Linear - ReLu - - ReLu Softmax

Algorithm 1 Optimization of the pulse shaping filter1: repeat2: . Transmitter3: Symbol mapping and upsampling: m→ x→ u4: Pulse shaping: u→ s5: Apply Gaussian: s→ s6: Apply DPD: s→ y7: Send y8: . Receiver9: Receive: y

10: Matched filtering and downsampling: y → u→ x11: Compute per example loss: `k12: Send `k13: . Transmitter14: Receive `k15: Update NN2 parameters according to (16)16: until Stop criterion is satisfied

by minimizing the Monte-Carlo approximation of the cross-entropy loss, defined by

LBt(θ1,θ2,θ3,θ4) =1

|Bt|

|Bt|∑

k=1

log[fθ4(yk)]mk . (13)

Similar to (5), the dependence of LBt(θ1,θ2,θ3,θ4) onθ1,θ2,θ3 is implicit through the distribution of the downsam-pled signal yk, which is a function of the channel input g(sk),where g(·) denotes the joint transfer function of the DAC, PA,and IQM, and sk is dependent on NNs 1–3 as can be seenin Fig. 3. For the optimization, in order to have a faster andmore stable convergence, all NNs are first initialized to mimictheir model-based counterparts via pre-training. Then, the setsof parameters θ1,θ2,θ3,θ4 are jointly optimized using theAdam optimizer [49].

C. Learning Without A Channel Model

In practice, training of the proposed AE in an experiment ischallenging due to the fact that the instantaneous gradients ofthe physical channel are unknown. To solve this problem, wefollow the alternative optimization approach that we reviewedin Section II-B3. The training of the demapper does not requiredifferentiation of the channel can therefore be performedvia supervised learning. For the transmitter training, sincethe transmitter consists of three NNs, one can perform thetransmitter training by alternating between the optimizationof the symbol mapper, the PS filter, and the DPD. In thispaper however, we only focus on training of the PS filter, forwhich memory effects need to be taken into account. For theoptimization of the mapper (i.e., NN1) or the DPD (i.e., NN3),

we refer the reader to [12], [34] and our recent paper [50].To that end, the parameters of the mapper, the DPD, and thedemapper (i.e., NN4) are assumed to be pretrained and fixedduring the PS filter training.

For the PS filter optimization, the training algorithm de-scribed in Section II-B3 cannot be used directly due to thememory introduced by the matched filtering. Therefore, weextend the training approach as follows. In each trainingiteration t, the transmitter generates a batch of |Bt| randomuniformly distributed messages within one message vectorm ∈ M|Bt| and maps them individually to the basebandsymbols after which R-time upsampling is applied. Then, thebaseband transmitted signals are generated by convolving theupsampled signals with a real-valued trainable filter accordingto s = u> ∗θ>2 , where ∗ denotes the convolution operator. Toallow for the gradient computation of the trainable PS filter, weconsider a Gaussian policy. To that end, a small perturbationwk ∈ CN (0, σ2) is applied to each of the pulse-shaped signalsbefore applying the DPD. Therefore, the DPD input s = s+wis stochastic and can be described by the PDF

πθ2(sk|u(L2)k ) =

1√2πσ2

e−|sk−θ

>2 u

(L1)k

|2

2σ2 . (14)

At the receiver, the channel observations y = [y1, . . . , y|Bt|R]are filtered by a MF and then downsampled with rate R. Then,the resulting signals x = [x1, . . . , x|Bt|] are used to computethe per-example loss defined by

`k = log[fθ4(xk)]mk , k = 1, . . . , |Bt|, (15)

where xk = ukR. The per-example losses are sent back to thetransmitter to perform the PS filter training. Due to memoryeffects introduced by the convolution operation in the MFand PS filter, `k is related to a subset of the entire sequencex and m. We denote the total number of samples relatedto `k by 2G + 1. The training objective is to optimize θ2such that the expected cross-entropy loss L(θ2) = E{`k} isminimized. Following [12], [34], we compute ∇θ2L(θ2) usingthe following proposition.

Proposition 1: The gradient of LBt(θ2) can be approxi-mated by

∇θ2LBt(θ2) (16)

≈ 1

σ2

G∑

g=−G

|Bt|∑

k=1

1

|Bt|`k(y,mk)(skR+g − [fθ2(fθ1(m))]kR+g)

×∇θ2 [fθ2(fθ1(m))]kR+g,

where we wrote fθi to highlight that the relation is applied tothe entire sequence in order to generate the entire correspond-ing output.

Proof: See Appendix A.

8

V. NUMERICAL RESULTS

In this section, we provide extensive numerical results toverify and illustrate the effectiveness of the proposed AE-basedWDM system. The system performance is measured in termsof SER, and for all the results presented below, the MF usedis the root-raised-cosine (RRC) filter.

A. Setup and Parameters

1) Simulation setup: We set M = 64, and consider a singlechannel system as well as a WDM system with 3 channels.For the 3-channel setup, the guard band between the adjacentchannels is ηfb (i.e., the channel spacing between neighboringchannels is (1+η)fb), where η ≥ 0 and fb is the symbol rate.The oversampling rate is set to R = 2 except for part ofSection V-B2, where we study the impact of the oversamplingrate on the performance. Both the PS filter and the MF have201 taps. The hardware impairments considered in this paperare restricted to the IQM nonlinearity and the limited ENOBof the DAC, while the PA is assumed to be linear, as the PAnonlinearity is negligible when compared to that of the MZM.However, it should be noted that the proposed approach canbe readily applied to a more general setup where the othertransmitter components are not idealized (e.g., nonlinear PAand bandwidth-limited DAC).

2) Transmitter and Receiver Networks: Following previouswork, all NNs are implemented as multi-layer fully-connectedNNs, where the ReLU function is chosen as the activationfunction for the hidden layers. The NN parameters used inthis paper are summarized in Table II.

3) Training: NN initialization: All AEs are trained by min-imizing the end-to-end cross-entropy loss, with the learningrate and batch size set to 0.0002 and 16000, respectively. Forthe 3-channel setup in particular, we consider using the sameAE configuration for all 3 channels, and we therefore onlyminimize the cross-entropy loss of the center channel and thenuse the parameters of the center channel AE for the AEs of theside channels. All AEs are trained for 10000 training iterations.In each training iteration, uniformly distributed training dataare randomly generated, and a total number of 1.6 × 108

data samples are used for each AE optimization. For theperformance evaluation, in order to avoid leakage of trainingdata into the testing set, independent uniformly distributed dataare randomly generated for testing.

4) Baseline: For the baseline, we use a geometricallyshaped constellation that is obtained via training a standardAE [6] over an AWGN channel at SNR = 18dB.6 The PSfilter is chosen as the RRC filter with roll-off factor β, whichis the same as the MF at the receiver. The DPD, which operatesseparately on the in-phase and quadrature components, isbased on the arcsin operation combined with clipping thatcan be described as [51]

6We find that training the standard AE at SNR = 18dB leads to aconstellation that is more performant than the standard square 64-QAM fora range of SNRs from 0 dB to 26 dB over the AWGN channel. The SNR ofthe considered WDM system falls into this SNR regime.

0.4 0.5 0.6 0.7 0.8 0.9 110−7

10−6

10−5

10−4

10−3

10−2

10−1

Normalized Vp

SER

W/o compensation

Baseline

Baseline, w/o ENOB

AE

Fig. 4: (a): SER performance versus Vp for the single channel scenario withβ = 10%, the blue curve corresponds to the baseline setup but withoutapplying the arcsin and clipping based DPD.

−2 −1 0 1 2

0

0.5

1

1.5

Normalized frequency (f/fb)

Frequen

cyresp

onse

|H(f

)|

RRC filter

Learned filter, single ch.

Fig. 5: Frequency response of the filter learned in the single-channel setup,showing OOB. The modulator driving swing is Vp = 1 and the receiver MFroll-off factor is set to β = 10%. The frequency response of the RRC filterwith β = 10% roll-off is also shown as a reference.

sk =

{min(π2 , Vclip arcsin(sk)) sk ≥ 0

max(−π2 , Vclip arcsin(sk)) sk < 0,(17)

where the arcsin linearizes the IQM response, while theclipping factor Vclip needs to be optimized to reduce the peak-to-average power ratio.

B. Results and Discussion

1) Single-Channel System: We start by investigating asingle-channel scenario (e.g., there is no ICI in the system),and we evaluate the performance of the proposed methodwith respect to the peak voltage Vp of the driving signals.For notation convenience, the peak voltage of driving signalsis normalized and the full swing of the MZM is used ifVp = 1. Due to the dependence of the MZM nonlinearitylevel on the driving voltage swing, a separate AE is trainedfor each considered Vp. Fig. 4 visualizes the SER of theproposed system when the receiver MF roll-off factor is set

9

t

h(t)

η = 0.04

t

η = 0.07

t

η = 0.1

t

η = 0.2

f

H(f)

f f f

(a) (b) (c) d

(e) (f) (g) (h)

Fig. 6: The impulse (top) and frequency (bottom) response of the learned filter (red curve) versus the guard band bandwidth for R = 2, Vp = 1 and β = 10%.The impulse and frequency response of the RRC filter (blue) with β = 10% are also shown as references; The green solid and black dashed curve correspondto the RRC filter and the learned filter of the adjacent channels.

t

h(t)

η = 0.04

t

η = 0.07

t

η = 0.1

t

η = 0.2

f

H(f)

f f f

(a) (b) (c) d

(e) (f) (g) (h)

Fig. 7: The impulse (top) and frequency (bottom) response of the filters learned in a WDM system with 5 channels for R = 4, Vp = 1 and β = 10%. Theimpulse and frequency response of the RRC filter (blue) with β = 10% are also shown as references.

to β = 10%. For a range of considered Vp, the proposedapproach achieves significantly better performance than theconsidered baseline. However, by looking at the frequencyresponse of the learned PS filter, as shown with the blue dashedcurve in Fig. 5, we observe that compared to the RRC filterwith 10% roll-off, the learned filter has a significant amountof OOB energy, which will introduce severe ICI betweennarrowly-spaced neighboring channels and make it unsuitablefor high SE WDM systems. This result indicates that thesystem designed for the single-channel setup cannot alwaysbe directly applied to a multi-channel setup, and additionalcare should be taken when designing multi-channel systems.

2) WDM System With 3 channels: We now train the pro-posed AE in a 3-channel setup. Fig. 6 visualizes the filterslearned with different guard band bandwidth. We start bylooking at the impulse response of the learned filters, whichappears to be very similar to the RRC filter. However, from thefrequency responses we observe that the trainable filter learnsto adjust its bandwidth according to the guard band betweenthe neighboring channels. In particular, when the guard bandis small (e.g., η = 0.04, Fig. 6 (e)) the filter learns to restrictthe OOB energy and has a narrower frequency response thanthe RRC filter, indicating that the trainable filter learns to limitICI. As we increase the guard band bandwidth, the bandwidthof the trainable filter increases as well. Similar to the single-channel scenario, the filter learns to put a significant amountof energy in the unoccupied spectrum when the guard band islarge (see Fig. 6 (h) for η = 0.2).

To train the multi-channel system it is necessary to usehigh oversampling rates to allow for placing the neighboringchannels in the considered spectrum. We emphasize that, inthis scenario, it is important to ensure that the PS filter cannotgenerate unrealistically high frequency components. This isillustrated in Fig. 7, which depicts the learned filters when thefilter is trained with 5 channels and R = 4 times oversamplingrate. Similar as before, the trainable filter learns to adjust itsbandwidth according to the channel spacing. However, thefilter also learns to put energy at high frequencies at theedges between the next two channels. Despite this interestingbehavior, such a filter is not feasible in practice due to thefact that a practical system would not operate at such highsampling rate because of the hardware limitations as well aspower constraints. This result reminds us again the importanceof using realistic setups when applying DL techniques for de-signing communication systems. Instead of upsampling to thefinal oversampling rate before the PS, one should use R = 2times oversampling rate for the PS and another upsamplingstep after the PS, which is the approach we followed for theother multi-channel simulations. An additional benefit of thismethod is that the number of filter taps is reduced for the sameFIR filter length, which improves convergence.

We now evaluate the performance of the proposed systemversus different guard band bandwidth, and we consider settingR = 2 and the receiver MF roll-off factor to β = 10%and β = 1%. The achieved SER for the center channel is

10

4 5 6 7 8 9 10

·10−2

10−5

10−4

10−3

10−2

η

SER

Baseline

AE w/ fixed NN1–3

AE w/ fixed NN1–2

AE w/ fixed NN1

AE

single channel

Fig. 8: SER performance versus Vp for the 3-channel scenario with β = 10%,the dashed blue curve corresponds to the baseline scheme for the single-channel scenario.

shown in Fig. 8 for β = 10% and in Fig. 9 for β = 1%.7

As a reference, the SER performance of the baseline schemeapplying arcsin combined with clipping is also shown. Weremark that the clipping factor Vclip and Vp are optimized forthe baseline scheme, while Vp is set to 1 in the proposedscheme for simplicity. Potentially, the performance of theproposed scheme can be further improved by optimizingVp—the optimal performance for the single-channel case isachieved at Vp = 0.9 (see Fig. 4). For roll-off factors of 10%(Fig. 8) and 1% (Fig. 9), the proposed approach outperformsthe baseline scheme over all considered guard bands. Moreimportantly, compared to the baseline scheme, the guard bandfor the proposed scheme can be significantly reduced withlimited impact on the SER performance —for the target SERwhere the baseline performance starts to saturate, the guardband can be reduced by around 37% for 10% roll-off andaround 50% for 1% roll-off. Such results indicate that theproposed approach can improve the SE of WDM systems byallowing to put the channels at a very narrow channel spacing.However, it should be noted that the reduction in guard bandsdoes not translate directly into the same gain in terms of SE,as the explicit SE depends on the applied modulation formats,the channel spacing, and the resulting SER.

3) Learned Constellation: Fig. 10 visualizes the learnedconstellation when the AE is trained in the 3-channel setupwith β = 10% and η = 0.05. The constellation optimized overthe AWGN channel and used for the baseline is also shown asa reference. It is shown that the constellation optimized overthe WDM setup has lower peak amplitude than the baseline,indicating the constellation optimized for the AWGN channelis suboptimal for a system that is impaired by hardwareimperfections. One possible explanation for such observationis that the AE learns to limit the peak voltage Vp by restrictingthe maximum amplitude of the constellation, so as to limit thesignal distortion caused by the nonlinear MZM.

7We note that the side channels have better SER performance than thecenter channel as they suffer from less ICI.

4 5 6 7 8 9 10

·10−2

10−5

10−4

10−3

10−2

η

SER

Baseline

AE w/ fixed NN1–3

AE w/ fixed NN1–2

AE w/ fixed NN1

AE

single channel

Fig. 9: SER performance versus Vp for the 3-channel scenario with β = 1%,the dashed blue curve corresponds to the baseline scheme for the single-channel scenario.

−1.5 −1 −0.5 0 0.5 1 1.5−1.5

−1

−0.5

0

0.5

1

1.5

Real part

Imag

inarypart

Fig. 10: Constellation used for the baseline (black) and constellation learnedin the 3-channel setup with η = 0.05 (red).

4) Learned DPD: Fig. 11 visualizes the transfer functionof the DPD (i.e., ANN3) learned for the 3-channel systemwith β = 10% and η = 0.05. The transfer functions ofthe conventional DPD employing arcsin and different clippingVclip are also shown as references. It is shown that the baselineDPD with optimized clipping has a response similar to thelearned DPD, suggesting that the considered DPD applyingarcsin combined with optimized clipping is near optimal forthe considered scenario.

5) Ablation study: In order to quantify the origin of the per-formance gains, we carry out an ablation study by first freezingall the pre-trained NNs and then individually unfreezing themin the order of NN4, NN3, NN2, and NN1. We start byunfreezing NN4. The resulting SER performance for β = 10%and β = 1% is shown in Fig. 8 and Fig. 9, respectively.Compared to the baseline scheme, it can be seen that theproposed approach achieves slightly better performance. Suchresult is what one would have expected, as the demamppertrained over the AWGN channel is likely to be suboptimal fora channel impaired by hardware imperfections. We then further

11

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

DPD Input

Out

put(

norm

aliz

edby

π 2)

arcsin w/ optimized Vclip

arcsin w/ Vclip = 1

arcsin w/ Vclip = 1.8

Learned DPD

−1 −0.5 0 0.5 1

−1

−0.5

0

0.5

1

DPD Input

Out

put(

norm

aliz

edby

π 2)

arcsin w/ optimized Vclip

arcsin w/ Vclip = 1

arcsin w/ Vclip = 1.8

Learned DPD

Fig. 11: Transfer function of NN3 and the conventional DPD using arcsincombined with optimized clipping for β = 10% and η = 0.05. The transferfunctions of the conventional DPD using arcsin and sub-optimal clippings arealso shown as references.

unfreeze the parameters of NN3 (i.e., NN1–2 are frozen).The resulting performance is very similar (slightly better) tothe case where NN1–3 are frozen. This result is consistentwith what is shown in Fig. 11. Finally, the parameters ofNN2 are also made trainable (i.e., only NN1 is fixed). In thiscase, the SER of the proposed approach improves significantly.Particularly, the largest gain achieved for β = 10% is η = 0.04while is η = 0.01 for β = 1%, indicating that the guardband can be optimized to improve the system performance.Finally, when all NNs are made trainable, the performanceof the proposed method further improves, which is consistentwith what is shown in Section. V-B3.

C. Model-Free Training of the Pulse-Shaping Filter

In this section, we extend our results to the case wherea differentiable channel model is unknown. Here, we onlyconsider training of the PS filter with the generalized trainingalgorithm discussed in Section IV-C. The reason for onlylearning the PS filter is that PS filter training contributes tomost of the performance gain as it is shown in the ablationstudy. RL-based training of the mapper and the DPD can befound in [12], [34], and [50], respectively.

Fig. 12 shows the achieved SER of the different schemesover a 3-channel WDM system. It is observed that the learnedPS filter using the RL-based algorithm achieves very similarperformance to the one using standard end-to-end learningassuming perfect channel knowledge. However, it should benoted that the RL-based approach allows for training of NNsin an experimental channel, and it has the potential to exceedthe performance of the conventional end-to-end learning-basedapproach, as the performance of the latter is highly dependenton the accuracy of the model used for training.

VI. CONCLUSION AND FUTURE WORK

We proposed a novel end-to-end AE for WDM systemsthat are impaired with non-ideal hardware components. Incontrast to most of the conventional AEs, which are usually

4 5 6 7 8

·10−2

10−4

10−3

10−2

η

SER

Baseline

model-aware

model-free

Fig. 12: SER performance comparison for β = 10% when the proposed AEis trained with and without the perfect channel knowledge.

implemented as a pair of NNs, our AE design follows thearchitecture of conventional communication systems, and ourtransmitter is implemented by a concatenation of simple NNs.Simulation results show that the proposed AE-based systemachieves significantly better performance than the consideredbaseline, and allows to increase the SE of WDM systems byreducing the channel spacing without severe SER performancedegradation. By means of an ablation study, we quantifythe origin of the performance improvement. It is shownthat the performance gain can be ascribed to the optimizedconstellation mapper, PS filter, and demapper. In addition,in case the channel model is unknown, we have shown thatthe PS filter can be trained using RL, and our simulationresults indicate that the extended RL-based training approachcan achieve similar performance to the standard end-to-endlearning assuming perfect channel knowledge.

For future work, there are several important aspects con-cerning the use of AEs which deserve further study:• Channel models: We have considered an optical back-to-

back channel due to the fact that the hardware distortionsalone significantly degrade the system. However, practicalsystems further suffer from performance loss caused bythe nonlinear crosstalk between adjacent channels. TheAE-based method may help to reduce the impact of thecrosstalk and provide significant performance improve-ment.

• The current AE design assumes that the WDM channelsoperate at the same rate. Practical systems, however,allow for transmission at different rates. New AE designand training methods may be needed to allow for flexibletransmission rates.

APPENDIX

We work with complete sequences, so that the loss is givenby:

L(θ2) = Em,s,s,y,y,x{`k} (18)= Em,s|m,y|s{`k}

12

where in the second step we remove all the deterministicrelations. Hence

L(θ2) =∑

m

∫∫p(m)p(s|m)p(y|s)`k(y,mk)dsdy (19)

=∑

m

∫∫p(m)π(s|fθ2(fθ1(m)))p(y|s)`k(y,mk)dsdy,

where we wrote fθi to expressly denote that the relation isapplied to the entire sequence in order to generate entire thecorresponding output. Exploiting the policy gradient theorem[34] and using the fact that ∇x log(g(x)) = ∇xg(x)

g(x) , it thenfollows that

∇θ2L(θ2)

=∑

m

∫∫p(m)∇θ2π(s|fθ2(fθ1(m)))p(y|s)`k(y,mk)dsdy

= E{`k(y,mk)∇θ2 log π(s|fθ2(fθ1(m)))}

= E{`k(y,mk)

R|Bt|∑

i=1

∇θ2 log π(si|[fθ2(fθ1(m))]i)}

≈ E{`k(y,mk)

G∑

g=−G∇θ2 log π(skR+g|[fθ2(fθ1(m))]kR+g)}

=1

σ2

G∑

g=−GE{`k(y,mk)(skR+g − [fθ2(fθ1(m))]kR+g)

×∇θ2 [fθ2(fθ1(m))]kR+g}

≈ 1

σ2

G∑

g=−G

|Bt|∑

k=1

1

|Bt|`k(y,mk)(skR+g − [fθ2(fθ1(m))]kR+g)

×∇θ2 [fθ2(fθ1(m))]kR+g,

which leads us to (16). The first approximation considers that`k(y,mk) is only affected by 2G + 1 surrounding samples,while the second approximation is used to compute the ex-pectation by averaging over the batch. We ignored boundaryeffect at the start and end of the sequence.

REFERENCES

[1] J. Song et al., “End-to-end autoencoder for superchannel transceiverswith hardware impairment,” in Proc. Optical Fiber CommunicationsConference and Exhibition, 2021.

[2] P. J. Winzer et al., “Fiber-optic transmission and networking: theprevious 20 and the next 20 years,” Optics express, vol. 26, no. 18,pp. 24 190–24 239, 2018.

[3] M. Yamada et al., “Gain-flattened tellurite-based EDFA with a flatamplification bandwidth of 76 nm,” Photonics Technology Letters,vol. 10, no. 9, pp. 1244–1246, 1998.

[4] D. Rafique et al., “Flex-grid optical networks: spectrum allocation andnonlinear dynamics of super-channels,” Optics Express, vol. 21, no. 26,pp. 32 184–32 191, 2013.

[5] M. Mazur et al., “Joint superchannel digital signal processing forultimate bandwidth utilization,” arXiv preprint arXiv:1911.02326, 2019.

[6] T. O’shea et al., “An introduction to deep learning for the physical layer,”Transactions on Cognitive Communications and Networking, vol. 3,no. 4, pp. 563–575, 2017.

[7] M. Kim et al., “A novel PAPR reduction scheme for OFDM systembased on deep learning,” Communications Letters, vol. 22, no. 3, pp.510–513, 2017.

[8] T. J. O’Shea et al., “Physical layer deep learning of encodings forthe MIMO fading channel,” in Proc. Annual Allerton Conference onCommunication, Control, and Computing, 2017, pp. 76–80.

[9] S. Dorner et al., “Deep learning based communication over the air,”Journal of Selected Topics in Signal Processing, vol. 12, no. 1, pp.132–143, 2017.

[10] A. Felix et al., “OFDM-autoencoder for end-to-end learning of commu-nications systems,” in Proc. International Workshop on Signal Process-ing Advances in Wireless Communications, 2018.

[11] H. Ye et al., “Channel agnostic end-to-end learning based communica-tion systems with conditional GAN,” in Globecom Workshops, 2018.

[12] F. A. Aoudia et al., “End-to-end learning of communications systemswithout a channel model,” in Proc. Asilomar Conference on Signals,Systems, and Computers, 2018, pp. 298–303.

[13] M. Zhang et al., “Neural network assisted active constellation extensionfor papr reduction of OFDM system,” in Proc. International Conferenceon Wireless Communications and Signal Processing, 2019.

[14] K. Choi et al., “Neural joint source-channel coding,” in Proc. Interna-tional Conference on Machine Learning, 2019, pp. 1182–1192.

[15] M. Stark et al., “Joint learning of geometric and probabilistic constel-lation shaping,” in Globecom Workshops, 2019.

[16] S. Cammerer et al., “Trainable communication systems: Concepts andprototype,” Transactions on Communications, vol. 68, no. 9, pp. 5489–5503, 2020.

[17] T. Van Luong et al., “Deep learning-aided multicarrier systems,” Trans-actions on Wireless Communications, vol. 20, no. 3, pp. 2109–2119,2020.

[18] F. A. Aoudia et al., “End-to-end learning for OFDM: From neuralreceivers to pilotless communication,” Transactions on Wireless Com-munications, 2021.

[19] ——, “Waveform learning for next-generation wireless communicationsystems,” arXiv preprint arXiv:2109.00998, 2021.

[20] B. Karanov et al., “End-to-end deep learning of optical fiber communica-tions,” Journal of Lightwave Technology, vol. 36, no. 20, pp. 4843–4855,2018.

[21] S. Li et al., “Achievable information rates for nonlinear fiber com-munication via end-to-end autoencoder learning,” in Proc. EuropeanConference on Optical Communication, 2018.

[22] R. T. Jones et al., “Geometric constellation shaping for fiber op-tic communication systems via end-to-end learning,” arXiv preprintarXiv:1810.00774, 2018.

[23] ——, “End-to-end learning for GMI optimized geometric constellationshape,” in Proc. European Conference on Optical Communication, 2019.

[24] B. Karanov et al., “End-to-end optimized transmission over dispersiveintensity-modulated channels using bidirectional recurrent neural net-works,” Optics express, vol. 27, no. 14, pp. 19 650–19 663, 2019.

[25] K. Gumus et al., “End-to-end learning of geometrical shaping maximiz-ing generalized mutual information,” in Proc. Optical Fiber Communi-cations Conference and Exhibition, 2020.

[26] T. Uhlemann et al., “Deep-learning autoencoder for coherent and non-linear optical communication,” in Proc. ITG-Symposium on PhotonicNetworks, 2020.

[27] B. Karanov et al., “End-to-end learning in optical fiber communications:Experimental demonstration and future trends,” in Proc. EuropeanConference on Optical Communications, 2020.

[28] O. Jovanovic et al., “End-to-end learning of a constellation shaperobust to variations in SNR and laser linewidth,” arXiv preprintarXiv:2106.00431, 2021.

[29] C. Jiang et al., “Machine learning paradigms for next-generation wirelessnetworks,” Wireless Communications, vol. 24, no. 2, pp. 98–105, 2016.

[30] F. N. Khan et al., “Machine learning methods for optical communicationsystems,” in Proc. Signal Processing in Photonic Communications, 2017,pp. SpW2F–3.

[31] N. E. West et al., “Deep architectures for modulation recognition,” inProc. International Symposium on Dynamic Spectrum Access Networks,2017.

[32] D. Zibar et al., “Machine learning techniques in optical communication,”Journal of Lightwave Technology, vol. 34, no. 6, pp. 1442–1452, 2015.

[33] C. Hager et al., “Nonlinear interference mitigation via deep neuralnetworks,” in Proc. Optical Fiber Communications Conference andExposition, 2018.

[34] F. A. Aoudia et al., “Model-free training of end-to-end communicationsystems,” Journal on Selected Areas in Communications, vol. 37, no. 11,pp. 2503–2516, 2019.

[35] A. C. Wilson et al., “The marginal value of adaptive gradient methodsin machine learning,” arXiv preprint arXiv:1705.08292, 2017.

[36] M. D. Zeiler, “Adadelta: an adaptive learning rate method,” arXivpreprint arXiv:1212.5701, 2012.

[37] I. Goodfellow, Y. Bengio, and A. Courville, Deep learning. MIT press,2016.

13

[38] P. a. Rodrıguez, “Beyond one-hot encoding: Lower dimensional targetembedding,” Image and Vision Computing, vol. 75, pp. 21–31, 2018.

[39] M. Li et al., “End-to-end learning for optical fiber communication withdata-driven channel model,” in Proc. Opto-Electronics and Communi-cations Conference, 2020.

[40] D. Wang et al., “Data-driven optical fiber channel modeling: a deeplearning approach,” Journal of Lightwave Technology, vol. 38, no. 17,pp. 4730–4743, 2020.

[41] B. Karanov et al., “Concept and experimental demonstration of opticalim/dd end-to-end system optimization using a generative model,” inProc. Optical Fiber Communications Conference and Exhibition, 2020.

[42] V. Curri et al., “Optimization of DSP-based Nyquist-WDM PM-16QAMtransmitter,” in Proc. European Conference and Exhibition on OpticalCommunication. Optical Society of America, 2012.

[43] G. Khanna et al., “A robust adaptive pre-distortion method for opticalcommunication transmitters,” Photonics Technology Letters, vol. 28,no. 7, pp. 752–755, 2015.

[44] P. W. Berenguer et al., “Nonlinear digital pre-distortion of transmittercomponents,” Journal of lightwave technology, vol. 34, no. 8, pp. 1739–

1745, 2015.[45] A. Napoli et al., “Digital pre-compensation techniques enabling high-

capacity bandwidth variable transponders,” Optics Communications, vol.409, pp. 52–65, 2018.

[46] C. Laperle et al., “Advances in high-speed DACs, ADCs, and DSP foroptical coherent transceivers,” Journal of lightwave technology, vol. 32,no. 4, pp. 629–643, 2014.

[47] A. Napoli et al., “Digital compensation of bandwidth limitations forhigh-speed DACs and ADCs,” Journal of Lightwave Technology, vol. 34,no. 13, pp. 3053–3064, 2016.

[48] D. Mishkin et al., “All you need is a good init,” arXiv preprintarXiv:1511.06422, 2015.

[49] D. P. Kingma et al., “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014.

[50] J. Song et al., “Over-the-fiber digital predistortion using reinforcementlearning,” arXiv preprint arXiv:2106.04934, 2021.

[51] Y. Tang et al., “Coherent optical OFDM transmitter design employingpredistortion,” Photonics Technology Letter, vol. 20, no. 11, pp. 954–956, 2008.