Cyclical Annealing Schedule:A Simple Approach to Mitigating KL Vanishing

Hao Fu1†, Chunyuan Li2†∗, Xiaodong Liu2, Jianfeng Gao2, Asli Celikyilmaz2, Lawrence Carin1

1Duke University 2Microsoft Research

Abstract

Variational autoencoders (VAEs) with an auto-regressive decoder have been applied for manynatural language processing (NLP) tasks. TheVAE objective consists of two terms, (i) re-construction and (ii) KL regularization, bal-anced by a weighting hyper-parameter β. Onenotorious training difficulty is that the KLterm tends to vanish. In this paper we studyscheduling schemes for β, and show that KLvanishing is caused by the lack of good latentcodes in training the decoder at the beginningof optimization. To remedy this, we propose acyclical annealing schedule, which repeats theprocess of increasing β multiple times. Thisnew procedure allows the progressive learn-ing of more meaningful latent codes, by lever-aging the results of previous learning cyclesas warm re-starts. The effectiveness of cycli-cal annealing is validated on a broad range ofNLP tasks, including language modeling, di-alog response generation and semi-supervisedtext classification.

1 Introduction

Variational autoencoders (VAEs) (Kingma andWelling, 2013; Rezende et al., 2014) have been ap-plied in many NLP tasks, including language mod-eling (Bowman et al., 2015; Miao et al., 2016),dialog response generation (Zhao et al., 2017;Wen et al., 2017), semi-supervised text classifi-cation (Xu et al., 2017), controllable text genera-tion (Hu et al., 2017), and text compression (Miaoand Blunsom, 2016). A prominent component of aVAE is the distribution-based latent representationfor text sequence observations. This flexible repre-sentation allows the VAE to explicitly model holis-tic properties of sentences, such as style, topic, andhigh-level linguistic and semantic features. Sam-ples from the prior latent distribution can produce

∗Corresponding author †Equal Contribution

diverse and well-formed sentences through simpledeterministic decoding (Bowman et al., 2015).

Due to the sequential nature of text, an auto-regressive decoder is typically employed in theVAE. This is often implemented with a recurrentneural network (RNN); the long short-term mem-ory (LSTM) (Hochreiter and Schmidhuber, 1997)RNN is used widely. This introduces one notori-ous issue when a VAE is trained using traditionalmethods: the decoder ignores the latent variable,yielding what is termed the KL vanishing problem.

Several attempts have been made to amelioratethis issue (Yang et al., 2017; Dieng et al., 2018;Zhao et al., 2017; Kim et al., 2018). Among them,perhaps the simplest solution is monotonic KL an-nealing, where the weight of the KL penalty termis scheduled to gradually increase during train-ing (Bowman et al., 2015). While these techniquescan effectively alleviate the KL-vanishing issue,a proper unified theoretical interpretation is stilllacking, even for the simple annealing scheme.

In this paper, we analyze the variable depen-dency in a VAE, and point out that the auto-regressive decoder has two paths (formally definedin Section 3.1) that work together to generate textsequences. One path is conditioned on the latentcodes, and the other path is conditioned on previ-ously generated words. KL vanishing happens be-cause (i) the first path can easily get blocked, dueto the lack of good latent codes at the beginning ofdecoder training; (ii) the easiest solution that anexpressive decoder can learn is to ignore the latentcode, and relies on the other path only for decod-ing. To remedy this issue, a promising approach isto remove the blockage in the first path, and feedmeaningful latent codes in training the decoder, sothat the decoder can easily adopt them to generatecontrollable observations (Bowman et al., 2015).

This paper makes the following contributions:(i) We provide a novel explanation for the KL-

vanishing issue, and develop an understanding ofthe strengths and weaknesses of existing schedul-ing methods (e.g., constant or monotonic anneal-ing schedules). (ii) Based on our explanation,we propose a cyclical annealing schedule. It re-peats the annealing process multiple times, andcan be considered as an inexpensive approach toleveraging good latent codes learned in the pre-vious cycle, as a warm restart, to train the de-coder in the next cycle. (iii) We demonstrate thatthe proposed cyclical annealing schedule for VAEtraining improves performance on a large range oftasks (with negligible extra computational cost),including text modeling, dialog response genera-tion, and semi-supervised text classification.

2 Preliminaries

2.1 The VAE model

To generate a text sequence of length T , x =[x1, · · · , xT ], neural language models (Mikolovet al., 2010) generate every token xt conditionedon the previously generated tokens:

p(x) =T∏t=1

p(xt|x<t),

where x<t indicates all tokens before t.The VAE model for text consists of two parts,

generation and inference (Kingma and Welling,2013; Rezende et al., 2014; Bowman et al., 2015).The generative model (decoder) draws a continu-ous latent vector z from prior p(z), and generatesthe text sequence x from a conditional distributionpθ(x|z); p(z) is typically assumed a multivariateGaussian, and θ represents the neural network pa-rameters. The following auto-regressive decodingprocess is usually used:

pθ(x|z) =T∏t=1

pθ(xt|x<t, z). (1)

Parameters θ are typically learned by maximiz-ing the marginal log likelihood log pθ(x) =log∫p(z)pθ(x|z)dz. However, this marginal

term is intractable to compute for many decoderchoices. Thus, variational inference is considered,and the true posterior pθ(z|x) ∝ pθ(x|z)p(z)is approximated via the variational distributionqφ(z|x) is (often known as the inference modelor encoder), implemented via a φ-parameterizedneural network. It yields the evidence lower bound

(ELBO) as an objective:

log pθ(x) ≥ LELBO = (2)

Eqφ(z|x)[log pθ(x|z)

]− KL(qφ(z|x)||p(z))

Typically, qφ(z|x) is modeled as a Gaussian dis-tribution, and the re-parametrization trick is usedfor efficient learning (Kingma and Welling, 2013).

2.2 Training Schedules and KL VanishingThere is an alternative interpretation of the ELBO:the VAE objective can be viewed as a regular-ized version of the autoencoder (AE) (Goodfel-low et al., 2016). It is thus natural to extend thenegative of LELBO in (2) by introducing a hyper-parameter β to control the strength of regulariza-tion:

Lβ = LE + βLR, with (3)

LE = −Eqφ(z|x)[log pθ(x|z)

](4)

LR = KL(qφ(z|x)||p(z)) (5)

where LE is the reconstruction error (or negativelog-likelihood (NLL)), andLR is a KL regularizer.

The cost function Lβ provides a unified per-spective for understanding various autoencodervariants and training methods. When β = 1,we recover the VAE in (2). When β = 0, andqφ(z|x) is a delta distribution, we recover the AE.In other words, the AE does not regularize thevariational distribution toward a prior distribution,and there is only a point-estimate to represent thetext sequence’s latent feature. In practice, it hasbeen found that learning with an AE is prone tooverfitting (Bowman et al., 2015), or generatingplain dialog responses (Zhao et al., 2017). Hence,it is desirable to retain meaningful posteriors inreal applications. Two different schedules for βhave been commonly used for a text VAE.

Constant Schedule The standard approach is tokeep β = 1 fixed during the entire training proce-dure, as it corresponds to optimizing the true VAEobjective. Unfortunately, instability on text anal-ysis has been witnessed, in that the KL term LRbecomes vanishingly small during training (Bow-man et al., 2015). This issue causes two undesir-able outcomes: (i) an encoder that produces poste-riors almost identical to the Gaussian prior, for allobservations (rather than a more interesting pos-terior); and (ii) a decoder that completely ignoresthe latent variable z, and a learned model that re-duces to a simpler language model. This is knownas the KL vanishing issue in text VAEs.

xzx ✓�

(a) Traditional VAE

z

x<t

xtx

t=1,· · ·,T

Path A

Path B

✓�

✓

(b) VAE with an auto-regressive decoder

Figure 1: Illustration of learning parameters {φ,θ} inthe two different paradigms. Starting from the obser-vation x in blue circle, a VAE infers its latent code zin the green circle, and further generates its reconstruc-tion in the red circle. (a) Standard VAE learning, withonly one path via {φ,θ} from x to its reconstruction;(b) VAE learning with an auto-regressive decoder. Twopaths are considered from x to its reconstruction: PathA via {φ,θ} and Path B via θ.

Monotonic Annealing Schedule. A simpleremedy has been proposed in (Bowman et al.,2015) to alleviate KL collapse. It sets β = 0 atthe beginning of training, and gradually increasesβ until β = 1 is reached. In this setting, we donot optimize the proper lower bound in (2) dur-ing the early stages of training, but nonethelessimprovements on the value of that bound are ob-served at convergence in previous work (Bowmanet al., 2015; Zhao et al., 2017).

The monotonic annealing schedule has becomethe de facto standard in training text VAEs, and hasbeen widely adopted in many NLP tasks. Thoughsimple and often effective, this heuristic still lacksa proper justification. Further, how to best sched-ule β is largely unexplored.

3 Cyclical Annealing Schedule3.1 Identifying Sources of KL Vanishing

In the traditional VAE (Kingma and Welling,2013), z generates x directly, and the reconstruc-tion depends only on one path of {φ,θ} passingthrough z, as shown in Figure 1(a). Hence, zcan largely determine the reconstructed x. In con-trast, when an auto-regressive decoder is used ina text VAE (Bowman et al., 2015), there are twopaths from x to its reconstruction, as shown inFigure 1(b). Path A is the same as that in thestandard VAE, where z is the global representa-tion that controls the generation of x; Path B leaks

the partial ground-truth information of x at everytime step of the sequential decoding. It generatesxt conditioned on x<t. Therefore, Path B can po-tentially bypass Path A to generate x, leading toKL vanishing.

From this perspective, we hypothesize that themodel-collapse problem is related to the low qual-ity of z at the beginning phase of decoder training.A lower quality z introduces more difficulties inreconstructing x via Path A. As a result, the modelis forced to learn an easier solution to decoding:generating x via Path B only.

We argue that this phenomenon can be easilyobserved due to the powerful representation capa-bility of the auto-regressive decoder. It has beenshown empirically that auto-regressive decodersare able to capture highly-complex distributions,such as natural language sentences (Mikolov et al.,2010). This means that Path B alone has enoughcapacity to model x, even though the decodertakes {x<t, z} as input to produce xt. Zhanget al. (2017) has shown that flexible deep neuralnetworks can easily fit randomly labeled trainingdata, and here the decoder can learn to rely solelyon x<t for generation, when z is of low quality.

We use our hypothesis to explain the learningbehavior of different scheduling schemes for β asfollows.

Constant Schedule The two loss terms in (2)are weighted equally in the constant schedule. Atthe early stage of optimization, {φ,θ} are ran-domly initialized and the latent codes z are of lowquality. The KL term LR pushes qφ(z|x) closeto an uninformative prior p(z): the posterior be-comes more like an isotropic Gaussian noise, andless representative of their corresponding observa-tions. In other words, LR blocks Path A, and thusz remains uninformative during the entire trainingprocess: it starts with random initialization andthen is regularized towards a random noise. Al-though the reconstruction term LE can be satisfiedvia two paths, since z is noisy, the decoder learnsto discard Path A (i.e., ignores z), and choosesPath B to generate the sentence word-by-word.

Monotonic Annealing Schedule The mono-tonic schedule sets β close to 0 in the early stage oftraining, which effectively removes the blockageLR on Path A, and the model reduces to a denois-ing autoencoder 1. LE becomes the only objective,

1The Gaussian sampling remains for qφ(z|x)

which can be reached by both paths. Though ran-domly initialized, z is learned to capture useful in-formation for reconstruction of x during training.At the time when the full VAE objective is con-sidered (β = 1), z learned earlier can be viewedas the VAE initialization; such latent variables aremuch more informative than random, and thus areready for the decoder to use.

To mitigate the KL-vanishing issue, it is key tohave meaningful latent codes z at the beginningof training the decoder, so that z can be utilized.The monotonic schedule under-weights the priorregularization, and the learned qφ(z|x) tends tocollapse into a point estimate (i.e., the VAE re-duces to an AE). This underestimate can result insub-optimal decoder learning. A natural questionconcerns how one can get a better distribution es-timate for z as initialization, while retaining lowcomputational cost.3.2 Cyclical Annealing ScheduleOur proposal is to use z ∼ qφ(z|x), which hasbeen trained under the full VAE objective, as ini-tialization. To learn to progressively improve la-tent representation z, we propose a cyclic anneal-ing schedule. We start with β = 0, increase β ata fast pace, and then stay at β = 1 for subsequentlearning iterations. This encourages the model toconverge towards the VAE objective, and infers itsfirst raw full latent distribution.

Unfortunately, Path A is blocked at β = 1. Theoptimization is then continued at β = 0 again,which perturbs the VAE objective, dislodges itfrom the convergence, and reopens Path A. Impor-tantly, the decoder is now trained with the latentcode from a full distribution z ∼ qφ(z|x), andboth paths are considered. We repeat this processseveral times to achieve better convergences.

Formally, β has the form:

βt =

{f(τ), τ ≤ R1, τ > R

with (6)

τ =mod(t− 1, dT/Me)

T/M, (7)

where t is the iteration number, T is the total num-ber of training iterations, f is a monotonicallyincreasing function, and we introduce two newhyper-parameters associated with the cyclical an-nealing schedule:

• M : number of cycles (default M = 4);• R: proportion used to increase β within a cy-

cle (default R = 0.5).

0

0.5

1

β

Monotonic

0 5K 10K 15K 20K 25K 30K 35K 40K

Iteration

0

0.5

1

β

Cyclical

Figure 2: Comparison between (a) traditional mono-tonic and (b) proposed cyclical annealing schedules.M = 4 cycles are illustrated, R = 0.5 is used for in-creasing within each cycle.

In other words, we split the training process intoM cycles, each starting with β = 0 and endingwith β = 1. We provide an example of a cyclicalschedule in Figure 2(b), compared with the mono-tonic schedule in Figure 2(a). Within one cycle,there are two consecutive stages (divided by R):

• Annealing. β is annealed from 0 to 1 in thefirst R dT/Me training steps over the courseof a cycle. For example, the steps [1, 5K]in the Figure 2(b). β = f(0) = 0 forcesthe model to learn representative z to recon-struct x. As depicted in Figure 1(b), there isno interruption from the prior on Path A, z isforced to learn the global representation of x.By gradually increasing β towards f(R) = 1,q(z|x) is regularized to transit from a pointestimate to a distribution estimate, spreadingout to match the prior.• Fixing. As our ultimate goal is to learn a

VAE model, we fix β = 1 for the rest oftraining steps within one cycle, e.g., the steps[5K, 10K] in Figure 2(b). This drives themodel to optimize the full VAE objective un-til convergence.

A Practical Recipe The existing schedules canbe viewed as special cases of the proposed cycli-cal schedule. The cyclical schedule reduces to theconstant schedule when R = 0, and it reduces toan monotonic schedule when M = 1 and R isrelatively small 2. In theory, any monotonicallyincreasing function f can be adopted for the cycli-cal schedule, as long as f(0) = 0 and f(R) = 1.In practice, we suggest to build the cyclical sched-ule upon the success of monotonic schedules: we

2In practice, the monotonic schedule usually anneals ina very fast pace, thus R is small compared with the entiretraining procedure.

adopt the same f , and modify it by setting M andR (as default). Three widely used increasing func-tions for f are linear (Fraccaro et al., 2016; Goyalet al., 2017), Sigmoid (Bowman et al., 2015) andConsine (Lai et al., 2018). We present the compar-ative results using the linear function f(τ) = τ/Rin Figure 2, and show the complete comparison forother functions in Figure 6 of the SupplementaryMaterial (SM).

3.3 On the impact of βThis section derives a bound for the trainingobjective to rigorously study the impact of β;the proof details are included in SM. For nota-tional convenience, we identify each data sam-ple with a unique integer index n ∼ q(n), drawnfrom a uniform random variable on {1, 2, · · · , N}.Further we define q(z|n) = qφ(z|xn) andq(z, n) = q(z|n)q(n) = q(z|n) 1

N . Follow-ing (Makhzani et al., 2016), we refer to q(z) =∑N

n=1 q(z|n)q(n) as the aggregated posterior.This marginal distribution captures the aggregatedz over the entire dataset. The KL term in (5) canbe decomposed into two refined terms (Chen et al.,2018; Hoffman and Johnson, 2016):FR = Eq(n)[KL(q(z|n)||p(z))]

= Iq(z, n)︸ ︷︷ ︸F1: Mutual Info.

+KL(q(z)||p(z))︸ ︷︷ ︸F2: Marginal KL

(8)

where F1 is the mutual information (MI) mea-sured by q. Higher MI can lead to a higher cor-relation between the latent variable and data vari-able, and encourages a reduction in the degree ofKL vanishing. The marginal KL is represented byF2, and it measures the fitness of the aggregatedposterior to the prior distribution.

The reconstruction term in (5) provides a lowerbound for MI measured by q, based on Corollary3 in (Li et al., 2017):

FE = Eq(n),z∼q(z|n)(log p(n|z))] +Hq(n)

≤ Iq(z, n) (9)

where H(n) is a constant.

Analysis of β When scheduled with β, the train-ing objective over the dataset can be written as:

F = −FE + βFR (10)

≥ (β − 1)Iq(z, n) + βKL(q(z)||p(z)) (11)

To reduce KL vanishing, we desire an increasein the MI term I(z, n), which appears in both FE

and FR, modulated by β. It shows that reduc-ing KL vanishing is inversely proportional with β.When β = 0, the model fully focuses on maximiz-ing the MI. As β increases, the model graduallytransits towards fitting the aggregated latent codesto the given prior. When β = 1, the implemen-tation of MI becomes implicit in KL(q(z)||p(z)).It is determined by the amortized inference reg-ularization (implied by the encoder’s expressiv-ity) (Shu et al., 2018), which further affects theperformance of the generative density estimator.

4 Visualization of Latent SpaceWe compare different schedule methods by visual-izing the learning processes on an illustrative toyproblem. Consider a dataset consisting of 10 se-quences, each of which is a 10-dimensional one-hot vector with the value 1 appearing in differ-ent positions. A 2-dimensional latent space isused for the convenience of visualization. Boththe encoder and decoder are implemented usinga 2-layer LSTM with 64 hidden units each. Weuse T = 40K total iterations, and the schedulingschemes in Figure 2.

The learning curves for the ELBO, reconstruc-tion error, and KL term are shown in Figure 3. Thethree schedules share very similar values. How-ever, the cyclical schedule provides substantiallylower reconstruction error and higher KL diver-gence. Interestingly, the cyclical schedule im-proves the performance progressively: it becomesbetter than the previous cycle, and there are clearperiodic patterns across different cycles. This sug-gests that the cyclical schedule allows the modelto use the previously learned results as a warm-restart to achieve further improvement.

We visualize the resulting division of the la-tent space for different training steps in Figure 4,where each color corresponds to z ∼ q(z|n), forn = 1, · · · , 10. We observe that the constantschedule produces heavily mixed latent codes zfor different sequences throughout the entire train-ing process. The monotonic schedule starts with amixed z, but soon divides the space into a mixtureof 10 cluttered Gaussians in the annealing process(the division remains cluttered in the rest of train-ing). The cyclical schedule behaves similarly tothe monotonic schedule in the first 10K steps (thefirst cycle). But, starting from the 2nd cycle, muchmore divided clusters are shown when learning ontop of the 1st cycle results. However, β < 1 leadsto some holes between different clusters, making

0 5K 10K 15K 20K 25K 30K 35K 40K# Iteration

0

2.5

5E

LBO

ConstantIncreasingCyclical

0 5K 10K 15K 20K 25K 30K 35K 40K# Iteration

0

2.5

5

Rec

. Err

or

0 5K 10K 15K 20K 25K 30K 35K 40K# Iteration

0

2.5

5

KL

(a) ELBO (b) Reconstruction Error (c) KL termFigure 3: Comparison of the learning curves for the three schedules on a toy problem.

Figure 4: Visualization of the latent space along the learning dynamics on a toy problem.

q(z) violate the constraint of p(z). This is allevi-ated at the end of the 2nd cycle, as the model istrained with β = 1. As the process repeats, wesee clearer patterns in the 4th cycle than the 2ndcycle for both β < 0 and β = 1. It shows thatmore structured information is captured in z usingthe cyclical schedule, which is beneficial in down-stream applications as shown in the experiments.

5 Related WorkSolutions to KL vanishing Several techniqueshave been proposed to mitigate the KL vanish-ing issue. The proposed method is most closelyrelated to the monotonic KL annealing techniquein (Bowman et al., 2015). In addition to intro-ducing a specific algorithm, we have comprehen-sively studied the impact of β and its schedulingschemes. Our explanations can be used to inter-pret other techniques, which can be broadly cate-gorized into two classes.

The first category attempts to weaken Path B,and force the decoder to use Path A. Word dropdecoding (Bowman et al., 2015) sets a certain per-centage of the target words to zero. It has shownthat it may hurt the performance when the droprate is too high. The dilated CNN was consideredin (Yang et al., 2017) as a new type of decoderto replace the LSTM. By changing the decoder’sdilation architecture, one can control Path B: the

effective context from x<t.The second category of techniques improves the

dependency in Path A, so that the decoder uses la-tent codes more easily. Skip connections were de-veloped in (Dieng et al., 2018) to shorten the pathsfrom z to x in the decoder. Zhao et al. (2017)introduced an auxiliary loss that requires the de-coder to predict the bag-of-words in the dialog re-sponse (Zhao et al., 2017). The decoder is thusforced to capture global information about the tar-get response. Semi-amortized training (Kim et al.,2018) was proposed to perform stochastic varia-tional inference (SVI) (Hoffman et al., 2013) ontop of the amortized inference in VAE. It sharesa similar motivation with the proposed approach,in that better latent codes can reduce KL vanish-ing. However, the computational cost to run SVIis high, while our monotonic schedule does not re-quire any additional compute overhead. The KLscheduling methods are complementary to thesetechniques. As shown in experiments, the pro-posed cyclical schedule can further improve them.

β-VAE The VAE has been extended to β-regularized versions in a growing body ofwork (Higgins et al., 2017; Alemi et al., 2018).Perhaps the seminal work is β-VAE (Higginset al., 2017), which was extended in (Kim andMnih, 2018; Chen et al., 2018) to consider β on

the refined terms in the KL decomposition. Theirprimary goal is to learn disentangled latent rep-resentations to explain the data, by setting β >1. From an information-theoretic point of view,(Alemi et al., 2018) suggests a simple method toset β < 1 to ensure that latent-variable modelswith powerful stochastic decoders do not ignoretheir latent code. However, β 6= 1 results in animproper statistical model. Further, β is static intheir work; we consider dynamically scheduled βand find it more effective.

Cyclical schedules Warm-restart techniques arecommon in optimization to deal with multimodalfunctions. The cyclical schedule has been used totrain deep neural networks (Smith, 2017), warmrestart stochastic gradient descent (Loshchilov andHutter, 2017), improve convergence rates (Smithand Topin, 2017), and obtain model ensem-bles (Huang et al., 2017). All these works appliedcyclical schedules to the learning rate. In contrast,this paper represents the first to consider the cycli-cal schedule for β in VAE. Though the techniquesseem simple and similar, our motivation is differ-ent: we use the cyclical schedule to re-open Path Ain Figure 1(b) and provide the opportunity to trainthe decoder with high-quality z.

6 ExperimentsThe source code to reproduce the experimental re-sults will be made publicly available on GitHub3.For a fair comparison, we follow the practicalrecipe described in Section 3.2, where the mono-tonic schedule is treated as a special case of cycli-cal schedule (while keeping all other settings thesame). The default hyper-parameters of the cycli-cal schedule are used in all cases unless stated oth-erwise. We study the impact of hyper-parametersin the SM, and show that larger M can providehigher performance for various R. We show themajor results in this section, and put more detailsin the SM. The monotonic and cyclical schedulesare denoted as M and C, respectively.

6.1 Language Modeling

We first consider language modeling on the PennTree Bank (PTB) dataset (Marcus et al., 1993).Language modeling with VAEs has been a chal-lenging problem, and few approaches have beenshown to produce rich generative models that donot collapse to standard language models. Ideally

3https://github.com/

Schedule Rec KL ELBO PPL

VAEM 101.73 0.907 -102.63 108.09C 100.51 1.955 -102.46 107.25

SA-VAEM∗ 100.75 1.796 -102.54 107.64M 101.83 1.053 -102.89 109.33C 100.50 2.261 -102.76 108.71

βm=0.5M 97.36 9.605 -106.96 132.57C 95.22 9.484 -104.70 118.78

Table 1: Comparison of language modeling on PennTree Bank (PTB). The last iteration results are reported.Since SA-VAE tends to overfit, we report its best re-sults in row M∗.

a deep generative model trained with variationalinference would pursue higher ELBO, making useof the latent space (i.e., maintain a nonzero KLterm) while accurately modeling the underlyingdistribution (i.e., lower reconstruction errors).

The latent variable is 32-dimensional, and 40epochs are used. We compare the proposed cycli-cal annealing schedule with the monotonic sched-ule baseline that, following (Bowman et al., 2015),anneals linearly from 0 to 1.0 over 10 epochs.We also compare with semi-amortized (SA) train-ing (Kim et al., 2018), which is considered as thestate-of-the-art technique in preventing KL van-ishing. We set SVI steps to 10.

Results are shown in Table 1. The perplexityis reported in column PPL. The cyclical scheduleoutperforms the monotonic schedule for both stan-dard VAE and SA-VAE training. SA-VAE trainingcan effectively reduce KL vanishing, it takes 472sper epoch. However, this is significantly moreexpensive than the standard VAE training whichtakes 30s per epoch. The proposed cyclical sched-ule adds almost zero cost. The full learning curvesare shown in Figure 7 in the SM.

Finally, we further investigate whether our im-provements are from simply having a lower β,rather than from the cyclical schedule re-openingPath A for better learning. To test this, we use amonotonic schedule with maximum β = 0.5. Weobserve that the reconstruction and KL terms per-form better individually, but the ELBO is substan-tially worse than β = 1, because β = 0.5 yieldsan improper model. Even so, the cyclical scheduleimproves its performance.

6.2 Conditional VAE for Dialog

We use a cyclical schedule to improve the la-tent codes in (Zhao et al., 2017), which are keyto diverse dialog-response generation. Follow-ing (Zhao et al., 2017), Switchboard (SW) Cor-

Context Alice: yeah you know its interesting especially when my experience has always been at a public university.Topic: Choose a College Target Bob (statement): yeah that’s right

C

1. yes

M

1. i’m not sure2. oh really 2. and i’m not sure3. and there’s a lot of <unk> there’s a lot of people 3. and i’m not sure4. yeah 4. i’m not sure5. and i think that’s probably the biggest problem i’ve ever seen in the past 5. i’m not sure

Table 2: Generated dialog responses from the cyclical and monotonic schedules.

Model CVAE CVAE+BoWSchedule M C M CRec-P ↓ 36.16 29.77 18.44 16.74KL Loss ↑ 0.265 4.104 14.06 15.55B4 prec 0.185 0.234 0.211 0.219B4 recall 0.122 0.220 0.210 0.219A-bow prec 0.957 0.961 0.958 0.961A-bow recall 0.911 0.941 0.938 0.940E-bow prec 0.867 0.833 0.830 0.828E-bow recall 0.784 0.808 0.808 0.805

Table 3: Comparison on dialog response genera-tion. Reconstruction perplexity (Rec-P) and BLEU (B)scores are used for evaluation.

pus (Godfrey and Holliman, 1997) is used, whichhas 2400 two-sided telephone conversations.

Two latent variable models are considered. Thefirst one is the Conditional VAE (CVAE), whichhas been shown better than the encoder-decoderneural dialog (Serban et al., 2016). The second isto augment VAE with a bag-of-word (BoW) lossto tackle the KL vanishing problem, as proposedin (Zhao et al., 2017).

Table 2 shows the sample outputs generatedfrom the two schedules using CVAE. Caller Alicebegins with an open-ended statement on choos-ing a college, and the model learns to generate re-sponses from Caller Bob. The cyclical schedulegenerated highly diverse answers that cover multi-ple plausible dialog acts. On the contrary, the re-sponses from the monotonic schedule are limitedto repeat plain responses, i.e., “i’m not sure”.

Quantitative results are shown in Table 3, usingthe evaluation metrics from (Zhao et al., 2017).Higher values indicate more plausible responses.The full results are in the SM. The BoW indeedreduces the KL vanishing issue, as indicated bythe increased KL and decreased reconstructionperplexity. When applying the proposed cyclicalschedule to CVAE, we also see a reduced KL van-ishing issue. Interestingly, it also yields the high-est BLEU scores. This suggests that the cyclicalschedule can generate dialog responses of higher

fidelity with lower cost, as the auxiliary BoWloss is not necessary. Further, BoW can be im-proved when integrated with the cyclical schedule,as shown in the last column of Table 3.

6.3 Semi-Supervised Text Classification

We consider the Yelp dataset, as pre-processedin (Shen et al., 2017) for semi-supervised learn-ing (SSL). Text features are extracted as the latentcodes z of VAE models, pre-trained with mono-tonic and cyclical schedules. The AE is used asthe baseline. A good VAE can learn to cluster datainto meaningful groups (Kingma and Welling,2013), indicating that well-structured z are highlyinformative features, which usually leads to higherclassification performance. To clearly compare thequality of z, we build a simple one-layer classifieron z, and fine-tune the model on different propor-tions of labelled data.

0.1 1.0 10.0 100.0Proportion (%) of labelled data

70

75

80

85

Acc

urac

y (%

)

CyclicalMonotonicAE

Figure 5: Accuracy of SSL onthe Yelp dataset.

The results areshown in Figure 5.The cyclical sched-ule consistentlyyields the highestaccuracy relativeto other methods.We visualize thetSNE embed-dings (Maaten and Hinton, 2008) of z in Figure 9of the SM, and observe that the cyclical scheduleexhibits clearer clustered patterns.

7 ConclusionsWe provide a novel two-path interpretation to ex-plain the KL vanishing issue, and identify itssource as a lack of good latent codes at the be-ginning of decoder training. This provides an un-derstanding of various β scheduling schemes, andmotivates the proposed cyclical schedule. By re-opening the path at β = 0, the cyclical sched-ule can progressively improve the performance, byleveraging good latent codes learned in the previ-ous cycles as warm re-starts. We demonstrate theeffectiveness of the proposed approach on three

NLP tasks, and show that it is superior to or com-plementary to other techniques.

ReferencesAlexander Alemi, Ben Poole, Ian Fischer, Joshua Dil-

lon, Rif A Saurous, and Kevin Murphy. 2018. Fix-ing a broken elbo. In ICML.

Samuel R Bowman, Luke Vilnis, Oriol Vinyals, An-drew M Dai, Rafal Jozefowicz, and Samy Ben-gio. 2015. Generating sentences from a continuousspace. arXiv preprint arXiv:1511.06349.

Boxing Chen and Colin Cherry. 2014. A systematiccomparison of smoothing techniques for sentence-level bleu. In Proceedings of the Ninth Workshop onStatistical Machine Translation, pages 362–367.

Ricky TQ Chen, Xuechen Li, Roger Grosse, and DavidDuvenaud. 2018. Isolating sources of disentangle-ment in vaes. NIPS.

Adji B Dieng, Yoon Kim, Alexander M Rush, andDavid M Blei. 2018. Avoiding latent variable col-lapse with generative skip models. arXiv preprintarXiv:1807.04863.

Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet,and Ole Winther. 2016. Sequential neural modelswith stochastic layers. In NIPS.

J Godfrey and E Holliman. 1997. Switchboard-1 re-lease 2: Linguistic data consortium. SWITCH-BOARD: A User’s Manual.

Ian Goodfellow, Yoshua Bengio, and Aaron Courville.2016. Deep learning, volume 1. MIT press Cam-bridge.

Anirudh Goyal Alias Parth Goyal, Alessandro Sor-doni, Marc-Alexandre Cote, Nan Rosemary Ke, andYoshua Bengio. 2017. Z-forcing: Training stochas-tic recurrent networks. In NIPS.

Irina Higgins, Loic Matthey, Arka Pal, ChristopherBurgess, Xavier Glorot, Matthew Botvinick, ShakirMohamed, and Alexander Lerchner. 2017. beta-vae:Learning basic visual concepts with a constrainedvariational framework. ICLR.

Sepp Hochreiter and Jurgen Schmidhuber. 1997. Longshort-term memory. Neural computation.

Matthew D Hoffman, David M Blei, Chong Wang, andJohn Paisley. 2013. Stochastic variational inference.The Journal of Machine Learning Research.

Matthew D Hoffman and Matthew J Johnson. 2016.Elbo surgery: yet another way to carve up the vari-ational evidence lower bound. In Workshop in Ad-vances in Approximate Bayesian Inference, NIPS.

Zhiting Hu, Zichao Yang, Xiaodan Liang, RuslanSalakhutdinov, and Eric P Xing. 2017. Toward con-trolled generation of text. ICML.

Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu,John E Hopcroft, and Kilian Q Weinberger. 2017.Snapshot ensembles: Train 1, get m for free. ICLR.

Hyunjik Kim and Andriy Mnih. 2018. Disentanglingby factorising. ICML.

Yoon Kim, Sam Wiseman, Andrew C Miller, DavidSontag, and Alexander M Rush. 2018. Semi-amortized variational autoencoders. ICML.

Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. ICLR.

Guokun Lai, Bohan Li, Guoqing Zheng, and YimingYang. 2018. Stochastic wavenet: A generative latentvariable model for sequential data. ICML workshop.

Chunyuan Li, Hao Liu, Changyou Chen, Yuchen Pu,Liqun Chen, Ricardo Henao, and Lawrence Carin.2017. Alice: Towards understanding adversariallearning for joint distribution matching. In NIPS.

Chia-Wei Liu, Ryan Lowe, Iulian V Serban, MichaelNoseworthy, Laurent Charlin, and Joelle Pineau.2016. How not to evaluate your dialogue system:An empirical study of unsupervised evaluation met-rics for dialogue response generation. arXiv preprintarXiv:1603.08023.

Ilya Loshchilov and Frank Hutter. 2017. Sgdr:Stochastic gradient descent with warm restarts.ICLR.

Laurens van der Maaten and Geoffrey Hinton. 2008.Visualizing data using t-sne. Journal of machinelearning research, 9(Nov):2579–2605.

Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly,Ian Goodfellow, and Brendan Frey. 2016. Adver-sarial autoencoders. ICLR workshop.

Mitchell P Marcus, Mary Ann Marcinkiewicz, andBeatrice Santorini. 1993. Building a large annotatedcorpus of english: The penn treebank. Computa-tional linguistics.

Yishu Miao and Phil Blunsom. 2016. Language as alatent variable: Discrete generative models for sen-tence compression. EMNLP.

Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neuralvariational inference for text processing. In ICML.

Tomas Mikolov, Martin Karafiat, Lukas Burget, JanCernocky, and Sanjeev Khudanpur. 2010. Recur-rent neural network based language model. InEleventh Annual Conference of the InternationalSpeech Communication Association.

Danilo Jimenez Rezende, Shakir Mohamed, and DaanWierstra. 2014. Stochastic backpropagation andapproximate inference in deep generative models.ICML.

Iulian Vlad Serban, Alessandro Sordoni, Yoshua Ben-gio, Aaron C Courville, and Joelle Pineau. 2016.Building end-to-end dialogue systems using gener-ative hierarchical neural network models. In AAAI.

Tianxiao Shen, Tao Lei, Regina Barzilay, and TommiJaakkola. 2017. Style transfer from non-parallel textby cross-alignment. In NIPS.

Rui Shu, Hung H Bui, Shengjia Zhao, Mykel J Kochen-derfer, and Stefano Ermon. 2018. Amortized infer-ence regularization. NIPS.

Leslie N Smith. 2017. Cyclical learning rates for train-ing neural networks. In WACV. IEEE.

Leslie N Smith and Nicholay Topin. 2017. Super-convergence: Very fast training of residual net-works using large learning rates. arXiv preprintarXiv:1708.07120.

Tsung-Hsien Wen, Yishu Miao, Phil Blunsom, andSteve Young. 2017. Latent intention dialogue mod-els. ICML.

Weidi Xu, Haoze Sun, Chao Deng, and Ying Tan.2017. Variational autoencoder for semi-supervisedtext classification. In AAAI.

Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, andTaylor Berg-Kirkpatrick. 2017. Improved varia-tional autoencoders for text modeling using dilatedconvolutions. ICML.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Ben-jamin Recht, and Oriol Vinyals. 2017. Understand-ing deep learning requires rethinking generalization.ICLR.

Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi.2017. Learning discourse-level diversity for neuraldialog models using conditional variational autoen-coders. ACL.

A Comparison of different schedules

We compare the two different scheduling schemesin Figure 6. The three widely used monotonicschedules are shown in the top row, including lin-ear, sigmoid and cosine. We can easily turn theminto their corresponding cyclical versions, shownin the bottom row.

B Proofs on the β and MI

When scheduled with β, the training objectiveover the dataset can be written as:

F = −FE + βFR (12)

We proceed the proof by re-writing each term sep-arately.

B.1 Bound on FEFollowing (Li et al., 2017), on the support of(x, z), we denote q as the encoder probabilitymeasure, and p as the decoder probability mea-sure. Note that the reconstruction loss for z canbe writen as its negative log likelihood form as:

FE = −Ex∼q(x),z∼q(z|x)[log p(x|z)]. (13)

Lemma 1 For random variables x and z withtwo different probability measures, p(x, z) andq(x, z), we have

Hp(z|x)= −Ez∼p(z),x∼p(x|z)[log p(z|x)]= −Ez∼p(z),x∼p(x|z)[log q(z|x)]− Ez∼p(z),x∼p(x|z)

[log p(z|x)− log q(z|x)

]= −Ez∼p(z),x∼p(x|z)[log q(z|x)]− Ep(x)(KL(p(z|x)‖q(z|x)))≤ −Ez∼p(z),x∼p(x|z)[log q(z|x)] (14)

where Hp(z|x) is the conditional entropy. Simi-larly, we can prove that

Hq(x|z) ≤ −Ex∼q(x),z∼q(z|x)[log p(x|z)] (15)

From lemma 1, we have

Corollary 1 For random variables x and z withprobability measure p(x, z), the mutual informa-tion between x and z can be written as

Iq(x, z) = Hq(x)−Hq(x|z) ≥ Hq(x)

+ Ex∼q(x),z∼q(z|x)[log p(x|z)]= Hq(x) + FE (16)

B.2 Decomposition of FRThe KL term in (5) can be decomposed into tworefined terms (Hoffman and Johnson, 2016):FR= Eq(x)[KL(q(z|x)||p(z))]= Eq(z,x)[(log q(z|x)− log p(z))]

= Eq(z,x)[(log q(z|x)− log q(z)]

+ Eq(z,x)[log q(z)− log p(z))]

= Eq(z,x)[(log q(z,x)− log q(x)− log q(z)]

+ Eq(z,x)[log q(z)− log p(z))]

= Iq(z,x)︸ ︷︷ ︸F1: Mutual Info.

+KL(q(z)||p(z))︸ ︷︷ ︸F2: Marginal KL

(17)

C Model Description

C.1 Conditional VAE for dialogEach conversation can be represented via threerandom variables: the dialog context c composedof the dialog history, the response utterance x, anda latent variable z, which is used to capture the la-tent distribution over the valid responses(β = 1)(Zhao et al., 2017). The ELBO can be written as:

log pθ(x|c) ≥ LELBO (18)

= Eqφ(z|x,c)[log pθ(x|z, c)

]− βKL(qφ(z|x, c)||p(z|c))

C.2 Semi-supervised learning with VAEWe use a simple factorization to derive the ELBOfor semi-supervised learning. α is introduced toregularize the strength of classification loss.

log pθ(y,x) ≥ LELBO (19)

= Eqφ(z|x)[log pθ(x|z) + α log pψ(y|z)

]− βKL(qφ(z|x)||p(z)) (20)

where ψ is the parameters for the classifier.Good latent codes z are crucial for the the

classification performance, especially when sim-ple classifiers are employed, or less labelled datais available.

D More Experimental Results

D.1 Language Modeling on PTBCode & Evaluation We implemented differentschedules based on the code4 published by Kimet al. (2018). Note that ELBO is closely related to

4https://github.com/harvardnlp/sa-vae

0

0.5

1

β

Monotonic

0 5K 10K 15K 20K 25K 30K 35K 40K

Iteration

0

0.5

1

β

Cyclical

0

0.5

1

β

Monotonic

0 5K 10K 15K 20K 25K 30K 35K 40K

Iteration

0

0.5

1

β

Cyclical

0

0.5

1

β

Monotonic

0 5K 10K 15K 20K 25K 30K 35K 40K

Iteration

0

0.5

1

β

Cyclical

(a) Linear (b) Sigmoid (c) Cosine

Figure 6: Comparison between traditional monotonic and proposed cyclical annealing schedules. The top rowshows the traditional monotonic schedules, and the bottom row shows their corresponding cyclical schedules.M = 4 cycles are illustrated, R = 0.5 is used for annealing within each cycle.

PPL. PPL = `/Ns and PPL = exp(`/Nw), where` is the total loss, Ns is the number of sentences,and Nw is the number of words.

Results We show the learning curves for VAEand SA-VAE in Figure 7. The cyclical sched-ule exhibits periodical learning behaviours. WhileELBO and PPL ar similar, the cyclical scheduleimproves the reconstruction ability and KL values.

D.2 CVAE for Dialog Response Generation

Code & Dataset We implemented differentschedules based on the code5 published by Zhaoet al. (2017). In the SW dataset, there are 70available topics. We randomly split the data into2316/60/62 dialogs for train/validate/test.

Evaluation metrics (1) Smoothed Sentence-level BLEU (Chen and Cherry, 2014): BLEUis a popular metric that measures the geometricmean of modified n-gram precision with a lengthpenalty. We use BLEU-1 to 4 as our lexical sim-ilarity metric and normalize the score to 0 to 1scale. (2) Cosine Distance of Bag-of-word Em-bedding (Liu et al., 2016): a simple method toobtain sentence embeddings is to take the averageor extreme of all the word embeddings in the sen-tences. We used Glove embedding and denote theaverage method as A−bow and extreme method asE−bow. The score is normalized to [0, 1].

Results The results on full BLEU scores areshown in Table 4. The cyclical schedule outper-forms the monotonic schedule in both settings.The learning curves are shown in Figure 8. Undersimilar ELBO results, the cyclical schedule pro-vide lower reconstruction errors, higher KL val-ues, and higher BLEU values than the monotonic

5https://github.com/snakeztc/NeuralDialog-CVAE

schedule. Interestingly, the monotonic scheduletends to overfit, while the cyclical schedule doesnot, particularly on reconstruction errors. It meansthe monotonic schedule can learn better latentcodes for VAEs, thus preventing overfitting.

D.3 Semi-supervised Text ClassificationDataset Yelp restaurant reviews dataset utilizesuser ratings associated with each review. Reviewswith rating above three are considered positive,and those below three are considered negative.Hence, this is a binary classification problem. Thepre-processing in (Shen et al., 2017) allows senti-ment analysis on sentence level. It further filtersthe sentences by eliminating those that exceed 15words. The resulting dataset has 250K negativesentences, and 350K positive ones. The vocabu-lary size is 10K after replacing words occurringless than 5 times with the “<unk>” token.

Results The tSNE embeddings are visualized inFigure 9. We see that cyclical β provides muchmore separated latent structures than the other twomethods.

D.4 Hyper-parameter tuningThe cyclical schedule has two hyper-parametersM and R. We provide the full results on M and Rin Figure 10 and Figure 11, respectively. A largernumber of cycles M can provide higher perfor-mance for various proportion value R.

D.5 Implementation DetailsTo enhance the performance, we propose to applythe cyclical schedule to the learning rate η on realtasks. It ensures that the optimizer has the samelength of optimization trajectory for each β cycle(so that each cycle can fully converge). To inves-tigate the impact of cyclical on η, we perform twomore ablation experiments: (i) We make only β

Model CVAE CVAE+BoWSchedule M C M CB1 prec 0.326 0.423 0.384 0.397B1 recall 0.214 0.391 0.376 0.387B2 prec 0.278 0.354 0.320 0.331B2 recall 0.180 0.327 0.312 0.323B3 prec 0.237 0.299 0.269 0.279B3 recall 0.153 0.278 0.265 0.275B4 prec 0.185 0.234 0.211 0.219B4 recall 0.122 0.220 0.210 0.219

Table 4: Comparison on dialog response generation.BLEU (B) scores 1-4 are used for evaluation. Mono-tonic (M) and Cyclical (C) schedules are tested on twomodels.

Schedule Rec KL ELBOCyc β + Const η 101.30 1.457 -102.76Mon β + Const η 101.93 0.858 -102.78Cyc β + Cyc η 100.61 1.897 -102.51Mon β + Cyc η 101.74 0.748 -102.49

Table 5: Comparison of cyclical schedules on β and η,tested with language modeling on PTB.

cyclical, keep η constant. (ii) We make only ηcyclical, keep β monotonic. The last epoch num-bers are shown in 5. The learning curves on shownin Figure 12. Compared with the baseline, we seethat it is the cyclical β rather than cyclical η thatcontributes to the improved performance.

5 10 15 20 25 30 35 40# Epoch

102

103

104

105

106

107

108

ELB

O

Cyc VAEMon VAECyc SA-VAEMon SA-VAE

5 10 15 20 25 30 35 40# Epoch

106

108

110

112

114

116

118

120

PP

L

(a) ELBO (b) PPL

5 10 15 20 25 30 35 40# Epoch

98

99

100

101

102

103

104

105

106

Rec

onst

ruct

ion

erro

r

5 10 15 20 25 30 35 40# Epoch

1

2

3

4

5

6

7

KL

(c) Reconstruction error (d) KL

Figure 7: Full results of VAE and SA-VAE on PTB. Under similar ELBO and PPL results, the cyclical scheduleprovide lower reconstruction errors and higher KL values than the monotonic schedule.

0 5 10 15 20 25 30 35# Epoch

50

55

60

65

70

75

ELB

O

0 5 10 15 20 25 30 35# Epoch

0.14

0.16

0.18

0.20

0.22

0.24

BLE

U4

(F1)

(a) ELBO (b) BLEU-4 (F1)

0 5 10 15 20 25 30 35# Epoch

10

15

20

25

30

35

40

45

50

Rec

ontru

ctio

n pe

plex

ity

0 5 10 15 20 25 30 35# Epoch

0

5

10

15

20

25

30

35

40

KL

(c) Reconstruction error (d) KL

Figure 8: Full results of CVAE and BoW+CVAE on SW dataset. Under similar ELBO results, the cyclical scheduleprovide lower reconstruction errors, higher KL values, and higher BLEU values than the monotonic schedule.Interestingly, the monotonic schedule tends to overfit, while the cyclical schedule does not.

(a) Cyclical β VAE (b) Monotonic β VAE (c) AE

Figure 9: Comparison of tSNE embeddings for three methods on Yelp dataset. This can be considered as theunsupervised feature learning results in semi-supervised learning. More structured latent patterns usually lead tobetter classification performance.

1 2 3 4 5 6 7 8 9Number of cycles M

0

1

2

R = 0.1

1 2 3 4 5 6 7 8 9Number of cycles M

0

1

2

R = 0.2

1 2 3 4 5 6 7 8 9Number of cycles M

0

1

2

R = 0.3

1 2 3 4 5 6 7 8 9Number of cycles M

0

1

2

R = 0.4

1 2 3 4 5 6 7 8 9Number of cycles M

0

1

2

R = 0.5

1 2 3 4 5 6 7 8 9Number of cycles M

0

1

2

R = 0.6

1 2 3 4 5 6 7 8 9Number of cycles M

0

1

2

R = 0.7

1 2 3 4 5 6 7 8 9Number of cycles M

0

1

2

R = 0.8

1 2 3 4 5 6 7 8 9Number of cycles M

0

1

2

R = 0.9

Figure 10: The impact of hyper-parameter M : number of cycles. A larger number of cycles lead to better perfor-mance. The improvement is more significant when R is small. The improvement is small when R is large.

.1 .2 .3 .4 .5 .6 .7 .8 .9 1.0Ratio R

0

1

2

M = 1

.1 .2 .3 .4 .5 .6 .7 .8 .9 1.0Ratio R

0

1

2

M = 2

.1 .2 .3 .4 .5 .6 .7 .8 .9 1.0Ratio R

0

1

2

M = 3

.1 .2 .3 .4 .5 .6 .7 .8 .9 1.0Ratio R

0

1

2

M = 4

.1 .2 .3 .4 .5 .6 .7 .8 .9 1.0Ratio R

0

1

2

M = 5

.1 .2 .3 .4 .5 .6 .7 .8 .9 1.0Ratio R

0

1

2

M = 6

.1 .2 .3 .4 .5 .6 .7 .8 .9 1.0Ratio R

0

1

2

M = 7

.1 .2 .3 .4 .5 .6 .7 .8 .9 1.0Ratio R

0

1

2

M = 8

.1 .2 .3 .4 .5 .6 .7 .8 .9 1.0Ratio R

0

1

2

M = 9

Figure 11: The impact of hyper-parameter R: proportion for the annealing stage. A larger R leads to betterperformance for various M . Small R performs worse, because the schedule becomes more similar with constantschedule. M = 1 recovers the monotonic schedule. Contrary to the convention that typically adopts small R, ourresults suggests that larger R should be considered.

5 10 15 20 25 30 35 40# Epoch

102

103

104

105

106

107

108

ELB

O

Cyc Cyc Mon Cyc Cyc Con Mon Con

5 10 15 20 25 30 35 40# Epoch

107.5

110.0

112.5

115.0

117.5

120.0

122.5

125.0

PP

L

(a) ELBO (b) PPL

5 10 15 20 25 30 35 40# Epoch

98

100

102

104

106

108

Rec

onst

ruct

ion

erro

r

5 10 15 20 25 30 35 40# Epoch

1

2

3

4

5

6

KL

(c) Reconstruction error (d) KL

Figure 12: Ablation study on cyclical schedules on β and η.