Learning in the Machine:Random Backpropagation and the Deep Learning Channel

(Extended Abstract)∗

Pierre Baldi1† , Peter Sadowski1 and Zhiqin Lu2

1Department of Computer Science, University of California, Irvine2Department of Mathematics, University of California, Irvinepfbaldi@ics.uci.edu, peter.sadowski@hawaii.edu, zlu@uci.edu

AbstractRandom backpropagation (RBP) is a variant of thebackpropagation algorithm for training neural net-works, where the transpose of the forward matricesare replaced by fixed random matrices in the calcu-lation of the weight updates. It is remarkable bothbecause of its effectiveness, in spite of using ran-dom matrices to communicate error information,and because it completely removes the requirementof maintaining symmetric weights in a physicalneural system. To better understand RBP, we com-pare different algorithms in terms of the informa-tion available locally to each neuron. In the pro-cess, we derive several alternatives to RBP, includ-ing skipped RBP (SRBP), adaptive RBP (ARBP),sparse RBP, and study their behavior through simu-lations. These simulations show that many variantsare also robust deep learning algorithms, but thatthe derivative of the transfer function is importantin the learning rule. Finally, we prove several math-ematical results including the convergence to fixedpoints of linear chains of arbitrary length, the con-vergence to fixed points of linear autoencoders withdecorrelated data, the long-term existence of solu-tions for linear systems with a single hidden layerand convergence in special cases, and the conver-gence to fixed points of non-linear chains, when thederivative of the activation functions is included.

1 IntroductionModern artificial neural networks are optimized usinggradient-based algorithms. Gradients can be computed rel-atively efficiently via the backpropagation algorithm, but thegradient at each weight generally depends on both the dataand all other weights in the network. This high degree of in-terdependence costs energy, both in biological neural systemsand in the artificial neural networks simulated using digitalcomputers. Furthermore, the calculation of the gradients inthe backpropagation algorithm includes the forward weight∗This paper is an extended abstract of the article Baldi, et al.,

Artificial intelligence,260:1–35, 2018.†Contact Author

matrices, a requirement known as the weight symmetry prob-lem that has long been an objection to the hypothesis thatbiological neurons learn via gradient descent (e.g. [Crick,1989]). New learning algorithms that do not require full gra-dient calculations could lead to more efficient neuromorphichardware and could help explain learning in the brain.

Are gradients really needed for learning in deep neural net-works (NNs)? Recent work suggests they are not (e.g. [Jader-berg et al., 2017]). In the random backpropagation algorithm(RBP) [Lillicrap et al., 2016], deep layers of a NN learnuseful representations even when the forward weight matri-ces are replaced with fixed, random matrices in the back-propagation equations. This algorithm differs from greedyunsupervised layer-wise approaches [Hinton et al., 2006;Bengio et al., 2007] because the deep weights depend on in-formation about the targets, and it differs from greedy super-vised layer-wise approaches [Gilmer et al., 2017; Mostafa etal., 2017] because the deep weights depend on the NN outputlayer, and hence all the other weights.

In this work we connect the RBP algorithm to the notion ofthe deep learning channel that communicates error informa-tion from the output layer to the deep hidden layers [Baldi andSadowski, 2016]. This channel is necessary to converge tocritical points of the objective, and can be studied using toolsfrom information and complexity theory. We classify learn-ing algorithms by the information that is transmitted alongthis channel, and our analysis leads to several new learningalgorithms, which we analyze through experiments on theMNIST [LeCun et al., 1998], CIFAR-10 [Krizhevsky andHinton, 2009], and HIGGS [Baldi et al., 2014] benchmarkdata sets. Furthermore, we prove that these algorithms con-verge to a global optimum of the objective function for im-portant special cases.

2 Random Backpropagation AlgorithmsIn this work we consider layered, feed-forward, neural net-works in which neurons in layer h are fully-connected to theneurons in the previous layer h − 1. Layer activity Oh iscomputed as a function of the preceding layer as

Oh , fh(Sh)

Sh ,WhOh−1 for 1 < h ≤ L (1)

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

6348

where O0 = I is the input data and fh is a non-linear acti-vation function. We focus here on supervised learning withtypical output activation functions and loss functions, includ-ing linear, sigmoid, and softmax output layers, for which thederivative of the loss E with respect to SL for a single input-target pair (I, T ) is given by

∂E∂SL

= OL − T. (2)

The backpropagation algorithm works by first computinggradients at each neuron recursively, then computing the gra-dients at the weights. These gradients are then used to updatethe weights:

Bh , − ∂E∂Sh

=

{(T −OL), for h = L

(fh)′ � (Wh+1)TBh+1, for h < L

∆Wh = −η ∂E∂Wh

= ηOh−1Bh (BP)

where the derivative (fh)′ is evaluated at Oh, and η is thelearning rate. In random backpropagation, the gradients Bhare replaced with a randomized error signalRh defined recur-sively

Rh ,

{(T −OL), for h = L

(fh)′ �ChRh+1, for h < L

∆Wh = ηOh−1Rh (RBP)

with constant matrices {Ch}1≤h≤L replacing the transposeof the weight matrices in each layer. In skip random back-propagation [Baldi et al., 2018], the randomized error signalsare sent directly to the deep layers rather than propagatingthrough each intermediate layer.

Rh ,

{(T −OL), for h = L

(fh)′ �ChRL, for h < L

∆Wh = ηOh−1Rh (SRBP)

where random matrix Ch now connects layer h to the outputlayer.

BP

Input

Output

RBP SRBP

Figure 1: The path of the error signal (red) from an output neuron toa deep, hidden neuron in backpropagation (BP), random backpropa-gation (RBP), and skip random backpropagation (SRBP).

These learning rules can be compared in terms of the in-formation required to update each weight. RBP solves theweight-symmetry problem by removing the dependency ofthe update on the forward weights in the backpropagationstep; the updates still depend on every other weight in thenetwork, but all that information is subsumed by the errorsignal at the output, T − OL. In SRBP, we also remove thedependency of ∆Wh on the derivative of the transfer func-tion in the downstream layers, (f l)′, l > h. Despite thesedifferences, we show that the these learning algorithms (aswell as adaptive variants) still converge to critical points ofthe objective function in network architectures conducive tomathematical analysis, unlike other alternative deep learningalgorithms such as greedy, layer-wise “pre-training.”

In addition, we introduce the idea of adaptive randombackpropagation (ARBP), where the backpropagation matri-ces in the learning channel are initialized randomly, then pro-gressively adapted during learning using the product of thecorresponding forward and backward signals, so that

∆Ch = ηOhRh+1.

In this case, the forward channel becomes the learning chan-nel for the backward weights. This adaptive behavior can alsobe used with the skip version (ASRBP).

In all these algorithms, the weight updates inthe last layer are equivalent to those of BP, soBP=RBP=SRBP=ARBP=ASRBP in the top layer. They onlydiffer in the way they train the hidden layers. In experiments,we also compare to the case where only the top layer istrained, and the hidden layers remain fixed after a randominitialization.

3 Results3.1 Mathematical ResultsThrough mathematical analysis, we prove that RBP andSRBP converge to a fixed point corresponding to the globaloptimum of the training set loss for the following neural net-works architectures, starting from almost any set of initialweights (except for a set of measure 0). Proofs for the case ofARBP are provided in [Baldi et al., 2017].

• A chain of single linear neurons of arbitrary length([1, . . . , 1]).

• An expansive architecture of linear neurons [1, N, 1].

• A compressive architecture of linear neurons [N, 1, N ].

• A simple [1, 1, 1] architecture, with a power functionnon-linearity in the hidden neuron of the form f(x) =xµ. Setting µ = 1/3 for instance gives an S-shaped acti-vation. Furthermore, we show that this system generallydoes not converge for µ 6= 1 when the derivative of thetransfer function is omitted from the learning rule.

For the linear architectures, under a set of standard assump-tions, we can derive a set of polynomial, autonomous, ordi-nary differential equations (ODEs) for the average time evo-lution of the weights under the different learning algorithms.As soon as there is more than one variable and the system is

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

6349

non-linear, there is no general theory to understand the cor-responding behavior. In fact, even in two dimensions, theproblem of understanding the upper bound on the number andrelative position of the limit cycles of a system of the formdx/dt = P (x, y) and dy/dt = Q(x, y), where P and Q arepolynomials of degree n is open–in fact this is Hilbert’s 16-th problem in the field of dynamical systems [Smale, 1998;Ilyashenko, 2002]. Thus, for the general N0, . . . , NL net-work architectures used in practiced, we turn to experiments.

−3 −2 −1 0 1 2 3a1

−3

−2

−1

0

1

2

3

a 2 dP dt

Figure 2: Vector field of RBP dynamics in the [1, 1, 1] linear casewith two network weights, a1 and a2, plotted on the x and y axis,and a fixed random backprop weight c1 = 1. The critical points ofthe learning equations correspond to the two hyperbolas, and all crit-ical points are fixed points and global minima of the error functions.Arrows are colored according to the value of dP/dt = d(a1a2)/dt,showing how the critical points inside the parabola a2 = −a2

1/c1are unstable. All other critical points are attractors. Reversing thesign of c1 leads to a reflection across both the a1 and a2 axes.

3.2 Empirical ResultsWe performed extensive experiments on variants of ran-dom backpropagation algorithms. We tested the followingvariations through experiments on MNIST, CIFAR-10, andHIGGS benchmark classification data sets, using architec-tures with 5-10 layers of tanh or relu neurons. The overallconclusion is that RBP and its variants are surprisingly robustto these variations.

• Activations: Random backpropagation algorithms workwith different transfer functions including linear, logis-tic, tanh, and relu units.

• Derivatives: The derivative of the activation function,f ′ appears to be an important factor in the learning rule.Our mathematical analysis found a case where RBP onlyconverges if this term is included, and in experimentswith real data we observe similar results (Figure 3).

• Adaptation: ARBP and ASRBP, the adaptive versions,are both able to learn. along with a variant reminiscentof Spike-Time Dependent Plasticity (STDP).

0 50 100

Epoch

0.7

0.8

0.9

1.0

Test

Accura

cy

BP

SBP

RBP

SRBP

Top only

Figure 3: MNIST test accuracy for backpropagation (BP), randombackpropagation (RBP), skip random backpropagation (SRBP), anda skipped variant of backpropagation (SBP) in a 5-layer architecture.BP, RBP, and SRBP converge to high accuracy while training the toplayer only (Top only) does not. Variants without transfer derivativesin the learning rule (dashed lines) also fail.

• Convolution: RBP and SRBP both can be used to trainconvolutional neural network architectures on CIFAR-10. (Confirmed also by [Nøkland, 2016].)

• Precision: We show that reducing the precision of theerror signals {Rh}1≤h≤L leads to gracious degradation,as opposed to catastrophic failure, as the error signalsare quantized down to a single bit (Figure 4). Similarresults are observed when we limit the precision of theweight updates ∆Wh rather than that of Rh.

• Sparsity: RBP and SRBP work when the random learn-ing channel matrices {Ch}1≤h≤L are sparse, even ifeach element is a single bit (Figure 5).

• Rank: The performance of both RBP and SRBP de-grades in as the rank of the matrices decrease to zero(Figure 6).

• Dropout: The dropout algorithm can be used in thelearning channel in both RBP and SRBP. This does notappear to have a large impact on learning, and it is un-clear whether it has the same regularizing effect.

• Stochasticity: If the matrices of the learning channelare randomly sampled from a Normal or Uniform dis-tribution with mean zero at each stochastic mini-batchupdate, then performance is poor and similar to trainingonly the top layer.

• Sign-Concordance: If each element of Ch is con-strained to have the same sign as the correspondingforward weight in RBP, then the system learns. Thisis the sign-concordance algorithm explored by Liao, etal. [Liao et al., 2016].

• Initialization: If the elements of the matrices of thelearning channel in RBP or SRBP are sampled from auniform or normal distribution with non-zero mean, per-formance is unchanged. This is also consistent with the

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

6350

sparsity experiments above, where the means of the sam-pling distributions are not zero.

• Sign: If we remove the sign information from the weightupdates, keeping only the absolute value, the systemdoes not learn.

• Weight Cooperation: If a different random backwardweight is used to send an error signal to each individ-ual weight, rather than to a hidden neuron which thenupdates all it’s incoming weights using the same signal,the system does not learn.

• Fixing Layers: If we fix the top layer(s) of a network(those closest to the output) and only train the lower lay-ers, the network does not learn. This is different fromgradient descent.

0 50 100

Epoch

0.85

0.90

0.95

1.00

Test

Accura

cy

BP

BP 5-bits

BP 3-bits

BP 1-bit

SRBP

SRBP 5-bits

SRBP 3-bits

SRBP 1-bit

RBP

RBP 5-bits

RBP 3-bits

RBP 1-bit

Figure 4: MNIST test accuracy with low-precision error signals(gradients in the case of BP). The error signals Bh and Rh are quan-tized on a log scale symmetric around 0. Performance is close tofull precision at 7-bit quantization, and progressively degrades asthe number of bits is decreased.

4 ConclusionHere we have derived several variants of RBP and studiedthem through simulations and mathematical analyses. Theemerging picture is that many of the variants lead to ro-bust deep learning even without the computation of gradi-ents. These algorithms often lead to slower learning whencompared to backpropagation on networks of a fixed size, butthey should be useful in the future both to better understandbiological neural systems, and to implement new neural phys-ical systems in silicon or other substrates.

Additional variants are studied in two followup papers.[Baldi et al., 2017] considers symmetry issues such as havinga learning channel with an architecture that is not a symmet-ric version of the forward architecture, or having non-linearunits in the learning channel that are similar to the non-linearunits of the forward architecture. [Baldi and Sadowski, 2018]connects RBP to the recirculation learning algorithm for au-toencoder networks [Hinton and McClelland, 1988], show-ing that they use the same learning principle. Recirculationis noteworthy because it removes a second major objectionto the biological-plausibility of backpropagation: the use offundamentally different types of signals in the forward andbackward propagation steps.

0 20 40 60 80 100

n

0.90

0.92

0.94

0.96

0.98

1.00

Fin

al accura

cy

Train

Test

Figure 5: MNIST accuracy after training with sparse SRBP, plot-ted against the sparsity in the random weight matrices {Ch}1≤h≤5.The random backpropagation matrix connecting any two layers iscreated by sampling each entry using a (0,1) Bernoulli distribution,where each element is 1 with probability p = n/(fan− in) and 0otherwise. For extreme values of n, sparse SRBP fails: for n = 0,all the backward weights are set to zero and no error signals are sent;for n = 100 all the backward weights are set to 1, and all the neu-rons in a given layer receive the same error signal. The performanceof the algorithm is surprisingly robust in between these extremes.

0 2 4 6 8 10Rank r of random matrices

0.7

0.8

0.9

1.0Fi

nal A

ccur

acy

TrainTest

Figure 6: MNIST accuracy after training with SRBP, plotted againstthe rank of the random weight matrices in network with seven hid-den layers of 100 tanh units each. Random weight matrices wereinitialized using a Glorot-scaled uniform distribution, then a rank-r approximation to the matrix was computed using the truncatedsingular value decomposition. Both SRBP and RBP (not shown)perform best with full-rank matrices in all layers.

The robustness and other properties of these algorithms cryfor explanations and more general principles. We have pro-vided both intuitive and formal explanations for several ofthese properties. On the mathematical side, polynomial learn-ing rules in linear networks lead to systems of polynomial dif-ferential equations. We have shown in several cases that thecorresponding ODEs converge to an optimal solution. How-ever these polynomial systems of ODEs rapidly become com-plex and, while the results provided are useful, they are notyet complete, thus providing directions for future research.

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

6351

References[Baldi and Sadowski, 2016] Pierre Baldi and Peter Sad-

owski. A theory of local learning, the learning channel,and the optimality of backpropagation. Neural Networks,83:51–74, 2016.

[Baldi and Sadowski, 2018] Pierre Baldi and Peter Sad-owski. Learning in the machine: Recirculation is randombackpropagation. Neural Networks, 108:479–494, 2018.

[Baldi et al., 2014] Pierre Baldi, Peter Sadowski, and DanielWhiteson. Searching for exotic particles in high-energyphysics with deep learning. Nature Communications, 5,2014.

[Baldi et al., 2017] Pierre Baldi, Peter Sadowski, and ZhiqinLu. Learning in the machine: The symmetries of the deeplearning channel. Neural Networks, 95:110–133, 2017.

[Baldi et al., 2018] Pierre Baldi, Peter Sadowski, and ZhiqinLu. Learning in the machine: Random backpropaga-tion and the deep learning channel. Artificial intelligence,260:1–35, 2018.

[Bengio et al., 2007] Yoshua Bengio, Pascal Lamblin, DanPopovici, Hugo Larochelle, Universite De Montreal, andMontreal Quebec. Greedy layer-wise training of deep net-works. In Advances in Neural Information Processing Sys-tems 19. MIT Press, 2007.

[Crick, 1989] Francis Crick. The recent excitement aboutneural networks. Nature, 337(6203):129–132, 1989.

[Gilmer et al., 2017] Justin Gilmer, Colin Raffel, Samuel S.Schoenholz, Maithra Raghu, and Jascha Sohl-Dickstein.Explaining the learning dynamics of direct feedback align-ment. 2017.

[Hinton and McClelland, 1988] Geoffrey E Hinton andJames L McClelland. Learning representations by recir-culation. In Neural information processing systems, pages358–366. New York: American Institute of Physics, 1988.

[Hinton et al., 2006] Geoffrey E. Hinton, Simon Osindero,and Yee-Whye Teh. A Fast Learning Algorithm for DeepBelief Nets. Neural Computation, 18(7):1527–1554, 2006.

[Ilyashenko, 2002] Yu Ilyashenko. Centennial history ofhilbert’s 16th problem. Bulletin of the American Mathe-matical Society, 39(3):301–354, 2002.

[Jaderberg et al., 2017] Max Jaderberg, Wojciech MarianCzarnecki, Simon Osindero, Oriol Vinyals, Alex Graves,David Silver, and Koray Kavukcuoglu. Decoupled neu-ral interfaces using synthetic gradients. In Proceedings ofthe 34th International Conference on Machine Learning-Volume 70, pages 1627–1635. JMLR. org, 2017.

[Krizhevsky and Hinton, 2009] Alex Krizhevsky and Geof-frey Hinton. Learning multiple layers of features from tinyimages. 2009.

[LeCun et al., 1998] Y. LeCun, L. Bottou, Y. Bengio, andP. Haffner. Gradient-based learning applied to documentrecognition. Proceedings of the IEEE, 86(11):2278–2324,1998.

[Liao et al., 2016] Qianli Liao, Joel Leibo, and Tomaso Pog-gio. How important is weight symmetry in backpropaga-tion? In Proceedings of the Thirtieth AAAI Conference onArtificial Intelligence, pages 1837–1844, 2016.

[Lillicrap et al., 2016] Timothy P. Lillicrap, Daniel Cown-den, Douglas B. Tweed, and Colin J. Akerman. Randomsynaptic feedback weights support error backpropagationfor deep learning. Nature Communications, 7:13276,November 2016.

[Mostafa et al., 2017] Hesham Mostafa, Vishwajith Ramesh,and Gert Cauwenberghs. Deep supervised learning usinglocal errors. arXiv preprint arXiv:1711.06756, 2017.

[Nøkland, 2016] Arild Nøkland. Direct feedback alignmentprovides learning in deep neural networks. In Advancesin Neural Information Processing Systems, pages 1037–1045, 2016.

[Smale, 1998] Steve Smale. Mathematical problems for thenext century. The Mathematical Intelligencer, 20(2):7–15,1998.

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

6352