Phase transitions and optimal algorithms for semi-supervised classifications ongraphs: from belief propagation to graph convolution network

Pengfei ZhouCAS Key Laboratory for Theoretical Physics, Institute of Theoretical Physics,

Chinese Academy of Sciences, Beijing 100190, China andSchool of Physical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China

Tianyi LiSystem Dynamics Group, Sloan School of Management,

Massachusetts Institute of Technology, Cambridge, MA 02139, USA

Pan Zhang∗

CAS Key Laboratory for Theoretical Physics, Institute of Theoretical Physics,Chinese Academy of Sciences, Beijing 100190, China

(Dated: December 11, 2019)

We perform theoretical and algorithmic studies for the problem of clustering and semi-supervisedclassification on graphs with both pairwise relational information and single-point feature informa-tion, upon a joint stochastic block model for generating synthetic graphs with both edges and nodefeatures. Asymptotically exact analysis based on the Bayesian inference of the underlying model areconducted, using the cavity method in statistical physics. Theoretically, we identify a phase transi-tion of the generative model, which puts fundamental limits on the ability of all possible algorithmsin the clustering task of the underlying model. Algorithmically, we propose a belief propagationalgorithm that is asymptotically optimal on the generative model, and can be further extended to abelief propagation graph convolution neural network (BPGCN) for semi-supervised classification ongraphs.

For the first time, well-controlled benchmark datasets with asymptotially exact properties andoptimal solutions could be produced for the evaluation of graph convolution neural networks, andfor the theoretical understanding of their strengths and weaknesses. In particular, on these syn-thetic benchmark networks we observe that existing graph convolution neural networks are subjectto an sparsity issue and an overfitting issue in practice, both of which are successfully overcome byour BPGCN. Moreover, when combined with classic neural network methods, BPGCN yields ex-traordinary classification performances on some real-world datasets that have never been achievedbefore.

I. INTRODUCTION

Learning on graphs is an important task in machinelearning and data sciences with a lot of applications invarious fields, including social sciences (e.g., social net-work analysis), biology (e.g., protein structure predictionand molecular finger prints learning), and computer sci-ence (e.g., knowledge graph analysis). The key differ-ence between learning with graph data and conventionalmachine learning with images and natural languages isthat, in addition to content features on each item, thereare also relational features between items that are en-coded by edges in the graph, which adds an extra layerof complexity to the analysis.

One classical problem of learning on graphs is the clas-sification of nodes into groups. Consider such a problemin the setting of a citation network, where each articleis represented by a graph node, and the groups of nodesare scientific research fields. In addition to the edges be-tween nodes, which represent the citations between arti-cles, each node is also associated with some features (i.e.,

∗ panzhang@itp.ac.cn

key words), which encode typical information of researchfields. If the group information of a small subset of nodesare known, then practically, these nodes could serves asa training set, and the learning task is to determine thegroup membership of the rest of the nodes by exploringthe direct group information via features, as well as indi-rect information via relationships (represented by edgesof the graph) with the training nodes. Essentially, thislearning task is semi-supervised classification on graphs, aproblem that recently has drawn much attention in bothnetworks sciences and machine learning communities; onthis problem, we witnessed the burst of Graph Convolu-tion Neural Networks (GCN), which is a powerful neuralnetwork architecture that yields ground-breaking perfor-mances [1].

Deep convolution neural networks have made greatsuccess in machine learning and artificial intelligence [2].Since there are many situations in which data are repre-sented as graphs, rather than voices and images that onone or two-dimensional grids, a lot of efforts have beenmade to extend convolution networks from applying ongrid data to applying on graph data, with a heavy fo-cus on constructing linear convolution kernels to extractlocal features in graphs and on learning effective rep-

arX

iv:1

911.

0019

7v2

[co

nd-m

at.s

tat-

mec

h] 1

0 D

ec 2

019

2

resentations of graph objects. During the past severalyears, many GCNs have been proposed, using differenttypes of convolution kernels and different network archi-tectures [1, 3–5]. Recent studies showed that GCNs havequickly dominated among neural network techniques inthe performance on various learning tasks, including textor graph-object classification, link prediction, forecast-ing, importance sampling, and are also believed to havea big potential on relational reasoning [6]. Nevertheless,despite that GCNs have achieved the state-of-the-art per-formance on semi-supervised classification, so far thereis little theoretical understanding on the mathematicalprinciples behind graph convolutions, and on the extentthat they may probably work in a particular problemsetting. The main difficulty is that, in previous studies,GCNs are often only tested on real-world datasets whichdo not have clear theoretical structures, thus the successor failure of GCNs is hard to be pinned down in theoret-ical analysis. A set of network datasets with establishedmathematical properties, and the analysis of GCNs onsuch datasets are missing and greatly welcomed.

In this study, we propose to study semi-supervised clas-sification on graphs, based on the celebrated stochasticblock model (SBM) [7]. The joint model consists of twograph components, characterizing both the relational in-formation and the feature information of items, capturedrespectively by a standard SBM and a bi-partite SBM[8, 9]. We call the model Joint Stochastic Block Model(JSBM). This model was originally proposed in [8] forthe problem of link and node predictions through theMarkov chain Monte Carlo method. In this study we ana-lyze theoretical properties of the JSBM, and design tech-niques for semi-supervised learning on graphs based onits properties. Filling the gap discussed above, the JSBMproduces well-controlled benchmark graphs with contin-uously tunable parameters for the evaluation of GCN’sclassification performance, and for the theoretical under-standing of their strengths and weaknesses under certainconditions.

On graphs generated by the JSBM, the clustering andclassification problems can be translated to a Bayesian in-ference problem, which could be solved theoretically withthe statistical physics approach in an asymptotically ex-act manner. This approach leads to a message-passingalgorithm, known in computer science as the belief prop-agation (BP) algorithm [10], which we claim to be asymp-totically exact on large random graphs generated by theJSBM. Through analyzing the stability of fixed points ofthe constructed BP equations on the JSBM, a phase tran-sition — the detectability transition is identified; beyondthe phase transition point, no algorithm is able to con-duct successful clustering on JSBM in an unsupervisedmanner. This is an extension of the detectability phasetransition [11] in the standard SBM [7], and puts funda-mental limits on the ability of algorithms in the clusteringtasks on graphs that could be modeled by JSBM.

In the semi-supervised classification setting, where asmall fraction of nodes have ground-true group labels and

could be used as the training data, the BP algorithm forJSBM could be embedded into a graph convolution net-work architecture. The unknown generative parametersof the JSBM graph could be learned in a standard classifi-cation approach, through forward-passing of (truncated)BP equations together with backward-passing of the gra-dients of the loss function. This novel GCN algorithm,which we term as BPGCN, guarantees to yield Bayesoptimal classification results [12] on synthetic graphsgenerated by the JSBM, and performs comparably withstate-or-the-art GCNs on real-world networks.

The paper is organized as follows. In Sec. II we in-troduce the joint stochastic block model. In Sec. III weformulate the Bayesian inference problem for clusteringand classification on the joint stochastic block model, andderive the belief propagation equations for the JSBM. InSec. IV we study the detectability phase transition ofJSBM using stability analysis of the BP algorithm. InSec. V we convert the BP equations on JSBM to a graphconvolution neural network and propose a new GCN algo-rithm, BPGCN. In Sec. VI the performance of BPGCN isevaluated and compared with the performance of severalstate-of-the-art GCNs, on both synthetic and real-worldnetworks. Sec. VII concludes the study.

II. JOINT STOCHASTIC BLOCK MODEL

The idea of the JSBM, literally the joint of two stochas-tic block models, is to simultaneously model item nodesand feature nodes in the network setting, by represent-ing both a connectivity graph over item nodes and anattribute graph between item nodes and feature nodes.The connectivity graph corresponds to the relation net-work in the traditional sense, and the attribute graphis a bipartite graph established on top of the relationnetwork; both graphs associate each node in the graphwith a unique group membership. These two graphs, con-structed from two SBM processes, constitute the JSBMgraph G; a similar framework was proposed by [8] tostudy link predictions.

Assume n item nodes in the (undirected) connectiv-ity graph, belonging to κ groups. Each node i has anunknown label t∗i denoting its independent group mem-bership (i.e. t∗i ∈ 1, 2, ...κ); each group label is chosenat random by nodes, according to a κ-dimensional prob-ability vector α, whose entries sum to 1. Edge connec-tivities are exclusively determined by their group mem-berships: for each pair of nodes i and j, there exists anedge (i, j) ∈ E with probability pt∗i t∗j , where E is the en-

tire edge set of the graph. This stochastic generativeprocess could be understood as a measuring process forthe ground truth t∗i , whose information is encoded im-plicitly in the edge connectivities as measuring results.The n × n adjacency matrix of this connectivity graphA ∈ 0, 1n×n follows the generative process and is there-fore stochastic, controlled by the deterministic κ×κ gen-eration probability matrix P = pt∗i t∗j . If the diagonal

3

elements in P are larger than the off-diagonal elements, itcorresponds to the situation (known as assortative SBM)where there are more edges within node groups than be-tween groups, and vice versa (known as disassortativeSBM). An intuitive example for an assortative SBM isa citation network with nodes denoting research articles(items), which belong to a certain research area (group)and are linked through citations (edges). Apparently, ar-ticles from the same research area are more likely to citeeach other.

Besides having membership in a certain research area,moreover, each research article may also be associatedwith some keywords which further denote its categoricalinformation; expectedly, the research area that an ar-ticle belongs to could be inferred from these keywords.This notion underlines the idea of making inference onthe joint SBM, i.e. inferring hidden group membershipof an item node from its relationships to known featurenodes in the attribute graph. Assume such a graph withm features over n item nodes. Same as item nodes, eachfeature node µ is embedded with an independent labelt∗µ ∈ 1, 2, ...κ, chosen randomly from the κ groups ac-cording to probabilities in a κ-dimensional vector β. Re-lationship between an item node i and a feature nodeµ exists with probability qt∗i t∗µ , which corresponds to an

edge (i, µ) ∈ F in the attribute graph whose edge setis F . This generative process yields a bipartite graphbetween item nodes and feature nodes, analogous to thebipartite SBM [9]; the resulting n×m adjacency matrixof the attribute graph F ∈ 0, 1n×m is stochastic and isgoverned by a κ × κ matrix Q = qt∗i t∗µ, similar to thecase of A and P. A specific JSBM is thus represented bythe two adjacent matrices A and F for the connectivitygraph and attribute graph, respectively, and is controlledby the generation parameters θ = P,Q, α, β (Fig. 1).It consists of a uni-partite graph and a bipartite graph,similar to the structure in semi-restricted Boltzmann ma-chines [13]. In this setting, a feature µ plays the role ofa hidden variable, or a functional node collecting mul-tiple interactions with the item nodes connected to it;hence edges on the attribute graph, i.e. relationships be-tween item nodes and feature nodes, are referred to ashyper edges. It is also worth noting that in the currentmodel we make no assumption on the relationships be-tween different feature nodes (i.e. the adjacent matrixon the attribute graph); such connectivities may provideinformation on the group membership of feature nodes,but not directly on the membership of item nodes, whichis the ground truth under concern.

III. BAYESIAN INFERENCE ON THE JSBM

On the graph G generated by the JSBM, the prob-lem of clustering is defined as recovering ground-true la-bels t∗i exclusively using edge information in the con-nectivity graph and the attribute graph (i.e., adjacentmatrics A and F); the problem of semi-supervised clas-

FIG. 1. Illustration of the joint stochastic block model(JSBM). Circles denote item nodes, whose inter-connectionsconstitute the connectivity graph, with adjacency matrix Adefined over edges; boxes denote feature nodes. The attributegraph is the bi-partite graph between items and features, withadjacency matrix F defined over hyper edges. The JSBMgraph is generated by parameters θ = P,Q, α, β, with κgroups of nodes built in for both items and features.

sification, in a slightly different manner, asks to conductthe same recovery but utilizes as extra information asmall number of training labels t∗

i where i belongs to

the training set Ω. If the parameters θ in generating theJBSM are known, this classification task can essentiallybe translated into an inference problem on the group la-bels t∗i , which could be viewed as hidden parametersin the model, provided with measurements on a specifictype of outcomes (edges in the two graphs) of these pa-rameters t∗i . Bayesian inference, which amounts tocomputing the posterior distribution, is typically used forsuch an inference task. For our problem, under Bayesianrules, the posterior is written as

P (ti, tµ|G, θ) =P (G|ti, tµ, θ)P0(ti, tµ)∑ti P (G|ti, tµ, θ)P0(ti, tµ)

.

(1)where P0(ti, tµ) represents prior information on thelabels (e.g., information regarding generative parametersα and β), and P (G|ti, tµ, θ) is the likelihood of ob-serving graph G given labels ti, tµ and parametersθ, which is the product-form probability of generatingexistent edges and hyper edges in G:

P (G|ti, tµ, θ) =∏i

αti∏

(ij)∈E

pti,tj∏

(ij) 6∈E

(1− pti,tj )

·∏µ

βtµ∏

(iµ)∈F

qti,tµ∏

(iµ)6∈F

(1− qti,tµ).

(2)

The clustering problem corresponds to adopting a flatprior, i.e., P0(ti, tµ) = constant, and semi-supervisedclassification corresponds to adopting strong prior onitem nodes that belong to the training set, such that theprobability marginals of (both ground-true and unclassi-fied) item nodes are pinned in the direction of traininglabels [12].

It is well known that computing the normalization ofthe posterior distribution (i.e., the denominator of (1))

4

is a #P problem, thus efficient and accurate approxi-mations are needed for the inference. In the languageof statistical physics, the representation of Bayesianinference on the posterior distribution Eq. (1) corre-sponds to a Boltzmann distribution at unit tempera-ture: the negative log-likelihood − logP (G|ti, tµ, θ)represents the energy; the normalization constant∑ti P (G|ti, tµ, θ)P0(ti, tµ) for the posterior is

the partition function; the prior information P0 plays therole of external fields acting on item nodes. For the clus-tering problem with a flat prior, the external field is zerofor all nodes; for semi-supervised classification, the ex-ternal field is infinity for nodes in the training set andzero for unlabelled nodes [12].

For random sparse graphs, the inference could be stud-ied at the thermodynamic limit using the cavity methodfrom statistical physics [10, 14]. If the parameters usedin generating the SBM is known a priori, the system is onthe Nishimori line [15, 16] and no spin glass phase couldappear. Moreover, the replica symmetry cavity methodnaturally translates into the belief propagation (BP) algo-rithm, a well-known algorithm for computation on SBM.In BP, cavity messages are passed along directed edges ofthe factor graph; when the propagation converges, theyare used to compute the posterior marginals. Inheritingthe message-passing idea, for JSBM, where the factorgraph G is two-fold (i.e., each edge acts as a two-bodyfactor; each hyper edge acts as a multi-body factor), weformulate the iterative equations of BP for three types ofmessages (see Appendix A for derivations):

ψi→jti = αtie−hti

Zi→j

∏k∈∂i\j

∑tk

ptitkψk→itk

∏µ∈∂i

∑tµ

qtitµψµ→itµ

ψi→µti = αtie−hti

Zi→µ

∏k∈∂i

∑tk

ptitkψk→itk

∏ν∈∂i\µ

∑tν

qtitνψν→itν

ψµ→itµ = βtµe−htµ

Zµ→i

∏j∈∂µ\i

∑tj

qtµtjψj→µtj . (3)

Here ψi→jti are the cavity marginals (messages) passingthrough item node i to item node j, representing theprobability of item node i taking label ti when the itemnode j is removed from the graph. Similarly, ψi→µti repre-sents the probability of item node i taking label ti whenfeature node µ is removed from the graph, and ψµ→itµrepresents the probability of feature node µ taking la-bel tµ when item node i is removed. Zi→µ, Zµ→i, andZi→j are normalizing factors; ∂i denotes the set of neigh-bors of item node i in the graph. In the three equations,variables hti and htµ are adaptive fields contributed bynon-existent edges of the graph, which are formulated as

(see Appendix A):

hti =∑k

∑tk

ptitkψktk

+∑µ

∑tµ

qtitµψµtµ

htµ =∑j

∑tj

qtµtjψjtj (4)

Once the above iterative equations converge (i.e., mes-sages do not change significantly), using the determinedcavity messages, the posterior marginals on the twographs (connectivity, attribute) could be calculated by:

ψiti = αtie−hti

Zi

∏k∈∂i

∑tk

ptitkψk→itk

∏µ∈∂i

∑tµ

qtitµψµ→itµ ,

ψµtµ = βtµe−htµ

Zµ

∏j∈∂µ

∑tj

qtµtjψj→µtj . (5)

In the end, based on these computed marginals, one isable to estimate the label of each item node, which is thespecific label that maximizes the item node’s marginal:

ti = argmaxt∈1,2,...,κ

ψiti . (6)

In Bayesian inference, ti is the maximum posterior esti-mate, representing the optimal result with the minimummean square error (MMSE) [16]. In terms of algorithmdesign, BP equations essentially adopt the Bethe approx-imation [10, 17], which is a variational distribution:

P (ti, tµ|G, θ) =

∏ij

∏iµ Φi,µti,tµΦi,jti,tj∏

i(ψiti)|∂i|−1

∏µ(ψµtµ)|∂µ|−1

, (7)

where i, µ represent item nodes and feature nodes. Therelationships between variational parameters, the two-point marginals Φi,µti,tµ , Φi,jti,tj (not used in BP equations)

and the single-point marginals ψiti (or ψµtµ), are adjustedduring the message-passing process in minimizing theBethe free energy. The core assumption is the conditionalindependence assumption, which is exact on trees. Thusthe Bethe approximation (7) is always correct for a treegraph in describing joint probabilities, in which case BPalgorithms yield exact posterior marginals. Empirically,BP results are shown to be good approximations to trueposterior marginals, if the graph is sparse and of locallytree-like structures; hence the BP algorithm is widely ap-plied to inference problems in sparse systems [14].

IV. DETECTABILITY TRANSITIONS OF THEJSBM

For the graph generated by the JSBM with param-eters θ, the cavity method provides asymptotically ex-act analysis, and the belief propagation algorithm (al-most) always converges, due to the Nishimori line prop-erty [15, 16]. Thus, asymptotically exact properties of the

5

JSBM, such as the phase diagram, can be studied directlyat the thermodynamic limit by analysing the messages ofthe belief propagation.

From (4) and (6), observe that there is a trivial fixedpoint of BP equations (3)

ψi→jti = ψi→µti = ψiti = αti ,

ψµ→itµ = ψµtµ = βtµ . (8)

This fixed point corresponds to the situation where ev-ery node in the graph has equal probability of belong-ing to every group; therefore it is known as the para-magnetic fixed point or liquid fixed point. In this case,the marginals do not provide any information about theground true group labels, whereas only reflect the permu-tational symmetry of the system. When this paramag-netic fixed point is stable, the system is in the paramag-netic state, where it is believed that no algorithm can dobetter than a random guess in revealing planted grouplabels. This scenario is known as the non-detectablephase for SBM, whose existence has been mathematicallyproved in [18]; in this study we extend the analysis tothe JSBM with two SBM components. Conceptually, thenon-detectable phase in the JSBM is analogous to the fer-romagnetic Ising model in paramagnetic phase where theunderlying ground-true labels correspond to the all-oneconfiguration, as well as the Hopfield model where un-derlying ground-true labels refer to the stored patterns.From the viewpoint of statistical inference, edges (ij)and hyper edges (iµ) are observations of the signal (i.e.the ground-true labels), so the paramagnetic phase de-notes the situation where the number of observations istoo few to reveal any valid information of the signal, suchthat the system evolves to the paramagnetic fixed pointwhere the label assignment is of equal probability for anynode.

When the number of observations increases, the para-magnetic fixed point will eventually become unstable,leading to a non-trivial fixed point of BP (3) whose valuesare correlated with the ground true group labels. Wherethe paramagnetic fixed point of BP becomes unstable in-dicates the position of the detectability transition for theJSBM, which posits fundamental limits on the ability ofalgorithms in revealing information of the ground truth,independent of the specific algorithm being used.

This phase transition point can be determined by thestability analysis of the paramagnetic fixed point of theBP (8). Assume that the JSBM graph has n→∞ itemnodes and m→∞ feature nodes. Each item node is con-nected to on average c1 item nodes and c2 feature nodes,and each feature node is connected to on average c3 itemnodes; when node’s degree distribution is Poisson, as inJSBM, c1 and c2 also equal to the average excess degree.Consider putting random noises with zero mean and unitvariance on every node (both items and features) of thegraph. Assuming a local-tree topology of the graph, af-ter one iteration of the BP equations, the noises will bepropagated to on average c1 item nodes through edges,

FIG. 2. Illustration of noise propagation on JSBM. Noisesare transmitted from leaves to the root node i, with eachtransmission weighted by eigenvalues of the Jacobian matri-ces T, determined by BP message passing equations (10).Red circles denote item nodes, green boxes represent edgesconnecting item nodes, and blue boxes denote feature nodes.

and c2c3 item nodes through hyper edges (Fig. 2). If ev-ery leaf node of the tree is associated with random noiseof zero mean and unit variance, after l → ∞ iterationsof BP equations (3), the aggregated variance of noises onthe root node i can be computed as (see Appendix B fordetails of derivations)

V = liml→∞

(c1λ2A + c2c3λ

4F )l. (9)

λA and λF are the largest eigenvalues of the Jacobian ma-trices Ti→j and Tµ→i, related to the connectivity graphand the attribute graph (i.e. the adjacency matrices Aand F), respectively, which are evaluated at the param-agnetic fixed point (together with a third matrix Ti→µ):

T i→jtitj =∂ψj→xtj

∂ψi→jti

∣∣αti

= αti(nptitjc1

− 1),

Tµ→ititµ =∂ψi→xti

∂ψµ→itµ

∣∣αti ,βtµ

= αti(mqtitµc2

− 1),

T i→µtµti =∂ψµ→jtµ

∂ψi→µti

∣∣αti ,βtµ

= βtµ(mqtµtic2

− 1). (10)

Since l → ∞, the paramagnetic fixed point is unsta-ble under random perturbations whenever V > 1. As aresult, the detectability phase transition locates at

c1λ2A + c2c3λ

4F = 1. (11)

This kind of stability conditions are known in the spinglass literature as the Almeida-Thouless local stabilitycondition [19]), and in computer sciences as the Kesten-Stigum bound on reconstruction on trees [20, 21], andthe robust reconstruction threshold [22]).

6

We demonstrate the phase transition Eq. (11) througha simple case of JSBM. Consider a P matrix with pin be-ing the diagonal and pout being the off-diagonal elements,and a similar Q matrix with qin on the diagonal and qout

on the off-diagonal. Notice that pin and pout, as well asqin and qout, are related by:

pin + (κ− 1)pout =κc1n,

qin + (κ− 1)qout =κc2m, (12)

hence there is only one free parameter for each matrix.We introduce ε1 = pin

poutand ε2 = qin

qout. For this simple

JSBM, the first eigenvalues of the two Jacobian matricesare expressed as:

λA =1− ε1

1 + (κ− 1)ε1λF =

1− ε21 + (κ− 1)ε2

. (13)

Numerical experiments are conducted to verify our the-oretical results. The performance of BP is evaluated us-ing the overlap O between the BP results ti obtainedfrom Eq. (6) and the ground truth t∗i :

O(ti, t∗i ) = maxπ

1

n

n∑i=1

δπ(ti),t∗i, (14)

which is maximizing over all permutations of groups π(i.e., |π| = κ!). In Eq. (14), if the inferred labels tiare chosen randomly which has nothing to do with theground truth t∗i , the overlap O = 1/κ (lower bound);if there is an exact match between ti and t∗i , O = 1(upper bound). For a small number of groups, overlapis a commonly used metric for estimating the similaritybetween two group assignments. When the number ofgroups is large, or with different group sizes, more ad-vanced measures are expected, such as the normalizedmutual information (NMI) and its variances [23, 24].

Results are shown in Fig. 3. In Figure 3(a), ε1 is fixedat 0.3 and ε2 varies; in Figure 3(b) it is the other wayaround. In both Figure 3(a) and 3(b), the overlap be-tween BP results and the (synthetic) ground truth areplotted against the varied ε (ε1 or ε2), which are optimalin the thermodynamic limit among using all possible al-gorithms. The overlap is close to 1.0 for small ε, indi-cating near-perfect reconstructions of the ground truth;with increased values of ε, the accuracy of BP inferencedecreases, which eventually downgrades to 0.5. This isconsistent with the theoretical limit of the detectabil-ity phase transition Eq. (11) (indicated by dashed lines).For a large ε, the system goes beyond the phase tran-sition point and lies in the paramagnetic phase, wherethe overlap is always 0.5, i.e., results of the BP detectionare indistinguishable from that of a 50-50 random guess.In Figure 3(c), the accuracy of BP for JSBM is shownon the ε1-ε2 plane, where the overlap decays from largevalues (yellow) to the non-informative 0.5 (blue). Thedashed line represents the theoretical phase transitionpoint, consistent with the numerical results.

V. FROM BELIEF PROPAGATION TO GRAPHCONVOLUTION NETWORK

When applying BP algorithms (Eq.(4)) to real-worldgraphs or synthetic graphs without a priori knowledge ofthe generative process, a critical problem is to determinethe parameters θ of the underlying JSBM. For a pureclustering problem, the classical approach for this taskis using the Expectation Maximization (EM) algorithm[25], in which parameters are updated through maximiz-ing the total log-likelihood of data (i.e., minimizing thetotal free energy). In practice, however, the EM approachis prone to overfitting [11] and may often be trapped inlocal minima. A different approach could be taken if asmall number of ground truth labels are available, as isthe setting of the current study, since now the parameterscould be determined in a semi-supervised fashion. Sucha semi-supervised update of parameters could possiblybe achieved through back-propagation on a neural net-work structure, which also facilitates the message passingof BP since it needs to be done in an iterative manner.With this idea in mind, we propose a solution frameworkfor the JSBM which contains two steps in each epoch:in the forward step, BP equations propagate to finitetime steps (layers) and conduct messages passing, andin the backward step, the parameters θ of the underly-ing JSBM are determined in a supervised approach, andback-propagate to BP layers.

Essentially, the graph convolution network (GCN) [1]structure is adopted in the above framework, which is anoutstanding neural network model that triggers a largenumber of variants (see below). On a GCN, values ongraph nodes X propagate forward among layers; in ahigh-level description, the propagation takes the follow-ing form:

Xl+1 = σ(UXlW), (15)

where σ(·) represents an activation function, Xl+1 ∈Rn×κl is the neural network state at layer l (κl denotingthe dimension of state at layer l, which is not necessarilyequal to κ in the GCN). The matrix W ∈ Rκl+1,κl is thetrainable weight matrix at the l-th layer, and the prop-agator U is responsible for convolving the neighborhoodof nodes, a kernel shared over the entire graph.

Based on the GCN structure, we propose the BeliefPropagation Graph Convolution Network (BPGCN), asour solution framework of the JSBM. Marginals and cav-ity messages ψ are network states; kernels take the jobof the products and summations in Eq. (3) and (5); andparameters of the JSBM P and Q become the weight ma-trices of the neural network, which are to be graduallylearnt through back-propagation. Under these conven-tions, the propagation from the l-th layer to the l + 1-th

7

0 0.25 0.5 0.75 1

2

0.5

0.75

1

Ove

rla

p 1=0.3

(a)

0 0.25 0.5 0.75 1

1

0.5

0.75

1

Ove

rla

p 2=0.3

(b)

Detectable phase

Paramagnetic phase

0 0.25 0.5 0.75 1

1

0

0.25

0.5

0.75

1

2

0.5

0.6

0.7

0.8

0.9

(c)

FIG. 3. Detectability phase transition of JSBM. Synthetic JSBM graphs are generated with n = 2 ∗ 105,m = 2 ∗ 105, groupnumbers κ = 2, and average degree c1 = c2 = 3. (a) & (b) Overlap as a function of ε1 or ε2, with the other parameter fixed to0.3 ((a) ε1 fixed; (b) ε2 fixed). The numerical detectability phase transition point agrees with the theoretical calculation (11),except for small fluctuations due to the finite size effect. (c) Overlap on the ε1 – ε2 plane. Dashed line indicates the theoreticaldetectability phase transition point, separating the detectable phase from the paramagnetic phase. Numerical and analyticalresults are consistent.

layer on a BPGCN is formulated as:

Ψµl+1 = Softmax

[UI log(Ψi→µ

l P)]

Ψil+1 = Softmax

[UII log(Ψµ→i

l Q) + UIII log(Ψi→jl P)

]Ψi→jl+1 = Softmax

[BI log(Ψµ→i

l Q) + BII log(Ψi→jl P)

]Ψi→µl+1 = Softmax

[BIII log(Ψi→j

l P) + BIV log(Ψµ→il Q)

]Ψµ→il+1 = Softmax

[BV log(Ψi→µQ)

](16)

where Softmax(zt) = ezt∑κs=1 e

zs is used as the activa-

tion function, inherited naturally from BP which asksto normalize marginal probabilities with κ components.If κ = 2, the Softmax activation function reduces to theSigmoid function. Also note the logarithm on ψ insidethe Softmax, which might be considered as a part of theactivation function in a strict sense. Kernel matrices UI

to UIII, BI to BV are non-backtracking matrices [26]that encode adjacency information of cavity messagesand marginals (see Appendix D for details). In prac-tice, random values are used in the input layer of thenetwork, for marginals Ψi

0 ∈ Rn×κ and Ψµ0 ∈ Rm×κ, as

well as messages Ψi→j0 ∈ R2MA×κ, Ψi→µ

0 ∈ R2MF×κ and

Ψµ→i0 ∈ R2MF×κ, where MA and MF are the number of

edges (in the connectivity graph) and hyper-edges (in theattribute graph), respectively.

The marginals in the last layer of BPGCN Ψ =Ψi

L ∈ [0, 1]n×κ are the output of a L−layer BPGCN.A loss function on the fraction of marginals belong totraining labels is adopted for the supervised learning. Acommon choice of the loss function for classification isthe cross entropy L; in our model, it is defined as:

L = −∑i∈Ω

κ∑s=1

(yi)s ln(ΨiL)s (17)

where Ω denotes the training set of item nodes (a fractionof ground-true nodes), yi = 0, 0, .., 1position(t∗i ), ...0 is aone-hot vector denoting the ground-truth label of node iin the training set.

Training the BPGCN is the same as training otherGCNs, as mentioned earlier: in each epoch, we first do aforward pass to obtain the computed marginals in the lastlayer, based on which we calculate the loss function on thetraining set; after that, we use the Back Propagation [2]algorithm to compute the gradients of the loss functionwith respect to elements in P and Q, and then apply(stochastic) gradient descent or its variants (e.g. ADAM[27]) to update the parameters. The iteration stops whenthe results converge, or after a finite number of epochs.In the end, the performance of the BPGCN algorithm isevaluated by the accuracy (i.e., overlap, Eq.(14)) of labelassignments on the ground-true item nodes in the testset.

A critical difference between BPGCN and traditionalBP (including the semi-supervised version [12]) is that,BP minimizes the (Bethe) free energy, whereas BPGCNminimizes the loss function evaluated on the trainingdata. On JSBM synthetic graphs with matched param-eters, the free energy is theoretically the best loss func-tion to minimize; however, on graphs not generated byJSBM, minimizing the (free) energy is prone to overfit-ting [11, 28]. More importantly, minimizing the loss func-tion is more flexible since the function could be formu-lated in different ways, for example, new (informative,or regularization) terms could be added. Notably, oneimplicit feature of inference on JSBM is that, the classi-fication on feature nodes (i.e., Ψµ

L) is left unconstrained,in both BP (equation (6)) and BPGCN (equation (17)),since the ground truth of feature labels are often notavailable. Nevertheless, in specific situations, such con-straints could be readily incorporated in the loss func-tion. For example, sometimes it is reasonable to believethat, the distribution of classified feature labels should

8

be close to a uniform distribution (or Gaussian in othercases); thus in these cases, we are able to append a termin the loss function to constrain the classification perfor-mance on features. In particular, using a one-hot vectorwµ = 1, 1, .., 1 to characterize the uniform distribution,we could formulate a new loss function as:

L′ = −∑i∈Ω

κ∑s=1

(yi)s ln(ΨiL)s − η

∑µ∈Ωm

κ∑s=1

(wµ)s ln(ΨµL)s

(18)where η is a damping factor, and Ωm denotes the set of allfeature nodes successfully classified. This new loss func-tion may probably help enhance the classification per-formance of BPGCN on real-world networks, althoughit is certainly not optimal for synthetic JSBM graphswith nonzero η. It is also worth noting that the adaptivefields (Eq. (4)), which in traditional BP are contributedby non-edges and are indispensable, are not necessary inBPGCN, because the traning labels automatically bal-ance the group sizes.

Comparing BPGCN with canonical GCNs, two ma-jor differences emerge. First, the activation function inBPGCN (Softmax) is not chosen arbitrarily, but ratherdetermined by BP message passing equations; this is incontrast with common GCNs where the activation func-tion may take various forms, such as ReLU, PReLU orTanh, but without sufficient physical or mathematicalwarrants. Second, there are only few parameters to betrained in BPGCN, which are the elements of P andQ, and they are shared across all layers of the neuralnetwork. Although this may narrow the overall repre-sentational power of BPGCN, the problem of overfittingcould nevertheless be obviated to a great extent. Indeed,in semi-supervised classification, the amount of trainingdata is often much less than that of (completely) super-vised classification, whereas the number of observations,in the case of networks, may be proportional to the num-ber of edges (and hyper edges in our model); therefore,the sharing of parameters across layers is believed to beextremely helpful in preventing overfit of the trainingdata.

VI. COMPARING BPGCN WITH OTHERGRAPH CONVOLUTION NETWORKS

In this section we compare the classification perfor-mance of BP and BPGCN, constructed on the jointstochastic block model, with several state-of-the-artgraph convolution networks, including the (standard)GCN, the Approximate Personalized Propagation of Neu-ral Predictions (APPNP), the Graph attention network(GAT), and the Simplified Graph Convolution Network(SGCN).1. (standard) Graph Convolution Network (GCN) [1]The standard GCN is probably the most famous graphconvolution network, which drastically outperformed allnon-neural-network algorithms when first proposed in

2017. The forward propagation rule of standard GCNis formulated as

H(l+1) = ReLU(AH(l)W(l)

), (19)

where H(l) and W(l) are states of hidden variables andweight matrices at the l-th layer. A is the graph convo-lution kernel, defined as

A = D−12 (A + I)D−

12 . (20)

D is a diagonal degree matrix with Dii = 1 +∑k Aik.

The propagation rule of the standard GCN was moti-vated by a first-order approximation of localized spec-tral filters (see Appendix C). Hence a two-layer standardGCN is written as:

Z = Softmax[ARelu(ARW0)W1], (21)

where R is the encoded information and Z ∈ Rn×κ is theclassification result.2. Approximate Personalized Propagation of Neural Pre-dictions (APPNP) [3]APPNP extracts feature information (encoded in R) tohidden neuron states H using a multi-layer perceptron(MLP):

H = Z(1) = MLP(R). (22)

Similar to the GCN, the hidden states H are propagatedvia the personalized PageRank scheme to produce pre-dictions of node labels:

Z(k+1) = (1− ω)AZ(k) + ωH

Z = Softmax[(1− ω)AZ(K−1) + ωH], (23)

where Z are explicit components of the hidden state Hof each layer, and ω ∈ [0, 1] is a weight factor. In therecent benchmarking study on the performance of vari-ous GCNs, APPNP produces the best results on multipledatasets [29].3. Graph attention network (GAT) [4]GAT adopts the attention mechanism [30] in which at-tention coefficients between pairs of connected nodes areregarded as weights. This scheme can be viewed as basedon the adjacency matrix of the graph, but with adjustableweights on edges.4. Simplified Graph Convolution Networks (SGCN) [31]SGCN tries to remove redundant and unnecessary com-putations from GCN by adopting a low-pass filter fol-lowed by a linear classifier. As a result, SGCN takes asimplified propagation rule using the k-th power of the

adjacency matrix A, as in GCN:

Z = Softmax(AkRW), (24)

where R, W, Z are feature information, weights andclassification results, respectively. Empirical results showthat the simplification process may yield positive impactson the accuracy of GCN, and could dramatically speedup the computation.

9

A. Results on synthetic networks

First we compare these GCN algorithms on syntheticnetworks, generated by the JSBM. Each JSBM graphconsists of n = 10, 000 item nodes, m = 10, 000 featuresnodes, and κ = 5 groups; each item node has on averagec1 neighbors in the connectivity graph and c2 neighborsin the attribute graph; each feature node is connected toon average c3 item nodes. In the semi-supervised experi-ments, 5% randomly chosen item nodes with ground-truelabels are used as the training set, and another 5% nodesare used as the validation set. The rest of nodes belongto the test set, on which the accuracy of the classifica-tion results is evaluated, by the measure of overlap (Eq.(14)). For BP and BPGCN, diagonal matrices are usedas the initial values for P and Q, with pin (and qin) onthe diagonal and pout (and qout) on the off-diagonal en-tries. Define ratio ε1 = pout/pin and ε2 = qout/qin, bothof which could be viewed as the (inverse) signal-to-noiseratio; for all synthetic networks, we use ε1 = ε2 = 0.5 asinitial values for BPGCN. While, for BP we use the rightinitial values which are used to generate correspondinggraphs, naturally obtaining asymptotically optimal re-sults. The initial values of ε1 and ε2 for BPGCN couldbe instead determined in a soft manner by the validationset; however, results show that this treatment is not verynecessary when compared to BP.

Extensive numerical experiments have been carried outon large synthetic graphs with varying average degreesand varying ε1,2 (Fig. 4; see also Appendix E). First, it isconfirmed that BPGCN results are perfectly aligned withthe (asymptotically) optimal BP results , outstanding allother GCNs. In Figure 4(a), c1 is fixed to 4, and c2 in-creases from 3 to 20. We can see that all other algorithms(GCN, APPNP, GAT, and SGCN) work much worse thanBPGCN, even when c2 is quite large. In Figure 4(b), c2is fixed to 4 and c1 is varied. Similar to Figure 4(a), itshows that when c1 is small, GCN, APPNP, GAT, andSGCN perform far worse than BP and BPGCN. Theseresults on synthetic graphs clearly show the superiorityof BPGCN over existing popular GCN algorithms; thereason to account for the poor performance of the refer-ence GCN algorithms is due to the sparsity of the graph(see below).

In Figure 4(c), ε2 is fixed to 0.1; the average degrees ofthe synthetic graph are fixed as c1 = 10 and c2 = 10; ε1is varied. Quite surprisingly, results show that BPGCNworks perfectly in the whole range of ε1, even when ε1is close to 1, i.e., the P matrix is homogeneous, whereasconventional GCNs quickly fail when ε1 increases. Thisphenomenon indicates that conventional GCNs underdiscussion have great difficulties in extracting the infor-mation on group labels from features, when the groupstructures of the graph are noisy, as one would imag-ine; nevertheless, to an extraordinary extent, BPGCN isnot subject to this failure. Further check on the outputsof conventional GCNs demonstrate that these results allhave good overlap in the training set, but poor overlap in

the test set, indicating clear overfitting to training labels.In what follows, we discuss the two issues (the sparsityissue and the overfitting issue) emerged in the results andanalyze how they are successfully overcome by BPGCN.

The sparsity issue— Properties of the forward-propagation of GCNs are closely related to their linearconvolution kernels, hence the reason to account for thesparsity issue can be uncovered by studying the spec-trum of the linear kernels used in graph convolutions. InGCN, SGCN and APPNP, the linear convolution kernel

is a variant of the normalized adjacency matrix A (20).It has been established in e.g. [26, 32] that this typeof linear operators have localization problems on largesparse graphs, with leading eigenvectors only encodinglocal rather than global information on group structures(Appendix C). In contrast, our BPGCN is immune to thesparsity issue, because the convolution kernels of BPGCNare the non-backtracking matrices, which naturally over-come the localization problem on large sparse graphs [26].Therefore, inspired by BPGCN, a straightforward way ofovercoming the sparsity issue in classic GCNs might beto consider using a linear kernel that does not triggerthe localization problem in sparse graphs, such as thenon-backtracking matrix or the X-Laplacian [32].

The overfitting issue— From Eq. (19), (23) and(24), one could observe that in these models the linearfilters always operate directly on the weight matrices oron the hidden states. This implies an underlying as-sumption of conventional GCNs: the relational data (i.e.edges, or the adjacency matrix of the connectivity graph)must always contain information on item nodes’ groupstructures (i.e., information on nodes’ labels). This is anatural assumption, yet may not always be true. In anadvanced manner, BPGCN relaxes this assumption bylearning an affinity matrix P that essentially stores thelearned signal-to-noise ratio (ε1,2) indicating the distri-bution of edges; hence it may identify that there is notnecessarily information encoded in edge connectivities.

B. Results on real-world datasets

Next we apply BPGCN and reference algorithms onseveral well-known real-world networks with and with-out node features. The Karate Club network [33]and Political Blogs network [34] are classical net-work datasets containing community structures. Since inthese two networks there is no feature of nodes, canonicalGCNs commonly use the identity matrix as the featurematrix F [1]. Given their small sizes, for the KarateClub network we use 2 nodes per group as training nodesand 2 nodes per group as validation nodes, and for thePolitical Blogs network we use 10 nodes per groupfor training and 10 nodes per group for validation. TheCiteseer, Cora and Pubmed networks are standarddatasets for semi-supervised classification widely used ingraph neural network studies; we follow the splitting ruleof training, test and validation sets introduced in [1] on

10

4 8 12 16 20c2

0.50

0.75

1.00Ov

erlap

(a)

4 8 12 16 20c1

0.50

0.75

1.00

Overlap

BPGCNBPGCN

APPNPGATSGCN

(b)

0.00 0.25 0.50 0.75 1.00ε1

0.2

0.4

0.6

0.8

1.0

Overlap

(c)

FIG. 4. Comparing BP and BPGCN with other GCNs on synthetic networks. Networks are generated by JSBM with n = 10, 000item nodes, m = 10, 000 features, and κ = 5 groups. During classification, 5% randomly selected nodes are used as the trainingset, another 5% used as the validation set, and the rest of nodes are the test set (all synthetic nodes have ground-true grouplabels), on which the metric of overlap (14) is evaluated to indicate different algorithms’ performance. Each data point in thefigure is averaged over 10 instances. (a) ε1 = 0.1, ε2 = 0.2, c1 = 4, and c2 varies; (b) ε1 = 0.1, ε2 = 0.2, c2 = 4, with c1 varies;(c) c1 = c2 = 10, ε2 is fixed to 0.2, and ε1 ranges from 0 to 1.

these graphs. For all these networks, there are ground-true labels for nodes’ group membership, coming fromeither public information (for Karate club and Politi-cal blogs networks) or empirical expert knowledge (i.e.,research areas of articles in citation networks). Same asthe case for synthetic data, the performance of testedalgorithms is evaluated by the overlap between classifi-cation results and the ground-true labels in the test set.

For BPGCN, a tunable external field is adopted andcorresponding hyper parameters are introduced, so asto adjust the relative strength of the training label onnodes. Identical to the case in synthetic graphs, diag-onal matrices are used as initial values for P and Q,controlled by the two signal-to-noise ratios ε1 = pout/pin

and ε2 = qout/qin. Unlike on synthetic graphs where ini-tial ε1 and ε2 are fixed at 0.5, on real-world graphs wedid a coarse search using the validation set to determineproper values for ε1 and ε2 during the preprocessing step,together with the search for proper hyper parameters, in-cluding the strength of the external field strength and thenumber of layers of BPGCN (see Appendix D).

Classification results are shown in the Table I. On thetwo networks without node features, all tested GCN al-gorithms perform quite well, and label information issuccessfully extracted from edge connectivities. BPGCNoutperforms other GCNs for the Polical blogs net-work; the good performance may come from its non-backtracking convolution kernel, which has good spectralproperties on large sparse graphs such as the Politicalblogs network [26]. On networks with node features,the comparison of classification results is less straight-forward. On Citeseer and Cora, the performance ofBPGCN is certainly comparable to other GCNs, and inboth cases it is superior to the performance of at leastone reference algorithm. Nevertheless, BPGCN workspoorly on the Pubmed network, yielding the worse per-formance among all tested GCNs. The reason is thatthe Pubmed network contains 19717 item nodes, but

only 500 features; when features are densely connectedto item nodes, the attribute graph significantly deviatesfrom being a sparse random graph, a critical assumptionour BPGCN algorithm and the underlying JSBM rely on.Indeed, it is verified that, when completely ignoring fea-tures and only using edge connectivities in classification,BPGCN yields a classification accuracy at 71.0%, bet-ter than 70.0%. Since the Multilayer percetron (MLP)method which only uses the feature information alreadyachieves a high accuracy, a possible remedy for BPGCNin situations where the attribute graph is far from hav-ing a locally tree-like topology, is to use the classificationresults yielded by MLP in place of the original attributegraph, as the external field acting on the BPGCN (i.e.,adopting MLP as a pre-classification step). Tests con-firm that this pre-processed version of BPGCN greatlyenhances the classification accuracy on the Pubmed net-work, up to 81.7%, which is significantly better thanthe state-of-the-art results yielded by APPNP. This pre-processing step does not improve the results on Cite-seer and Cora, as expected, since the attribute graphsare sparse.

VII. CONCLUSIONS

In this study, we constructed the joint stochastic blockmodel (JSBM) that generates a two-fold graph which si-multaneously models item nodes and feature nodes inan aggregated network setting, based on the celebratedSBM. Utilizing the cavity method in statistical physicsand the corresponding belief propagation (BP) algorithmwhich is asymptotically exact in the thermodynamiclimit, theoretical results on the detectability phase tran-sition point and the phase diagram for JSBM are un-covered. Expectantly, JSBM could be used to generatebenchmark networks with continuously tunable parame-ters, which might be particularly useful in evaluating the

11

Karate Polblogs Citeseer Cora Pubmed# nodes 34 1490 3327 2078 19717# features 0 0 3703 1433 500# groups 2 2 6 7 3# training 4 20 120 140 60# validation 4 20 500 500 500# test 26 1450 1000 1000 1000c 4.6 22.4 2.78 3.89 4.49MLP [1] — — 58.4 52.2 72.7GCN [1] 96.5 86.2 71.1 81.5 79.0GAT [4] 87.9 88.7 70.8 83.1 78.5SGCN [31] 91.6 81.3 71.3 81.7 78.9APPNP [3] 96.3 87.5 71.8 83.5 80.1BPGCN 95.8 89.3 71.1 82.1 70.0

TABLE I. Classification performance of BPGCN and refer-ence GCNs on real-world networks. c denotes the averagedegree of the connectivity graph. For reference GCNs, resultson Karate Club and Polical blogs are carried out us-ing publically available implementations of these algorithms;results on Cora, Pubmed and Citeseer are adapted from[29]. For BPGCN, the reported accuracy values are based onthe vanilla version. The pre-processed version of the BPGCNgreatly enhances the classification accuracy on the Pubmednetwork, up to 81.7%(see text).

classification performance of graph neural networks.

Based on the BP equations established on JSBM, weproposed an algorithm for semi-supervised classification,adopting the graph convolution network structure, which

we termed as BPGCN. In contrast to most existing graphconvolution networks, the convolution kernel and acti-vation function of BPGCN are determined mathemat-ically from the Bayesian inference on the JSBM. Weshow that on synthetic networks generated by JSBM,BPGCN clearly outperforms several well-known existinggraph convolution networks, and obtains extraordinaryclassification accuracy in the parameter regime whereconventional GCNs fail to work; on real-world networks,BP also displays comparable performance to state-of-the-art GCNs. Compared with conventional GCNs, BPGCNis quite powerful in extracting label information fromedge connectivities; this advantage is rooted in its non-backtracking convolution kernel inherited from the BPalgorithm. The weakness of BPGCN is demonstrated bythe Pubmed dataset, in which case there are too few fea-tures for the attribute graph to be approximated by arandom bipartite graph; a remedy for applying BPGCNon such graphs is proposed, which uses MLP as a pre-processing step, and the corresponding advanced versionof BPGCN is demonstrated. Based on the fact thatBPGCN is immune to the sparsity issue and the overfit-ting issue exposed by conventional GCNs, we discussedpossible ways inspired by BPGCN to improve currentGCN techniques. It would be interesting to combinesuccessful features of BPGCN and state-of-the-art GCNsin greater depth and design new architectures for graphneural networks; we leave this idea for future work.

A pytorch and C++ implementation of our BPGCNand other GCNs on JSBM together with the real worlddatasets used in our experiments, are available at [35].

[1] Thomas N. Kipf and Max Welling, “Semi-supervised clas-sification with graph convolutional networks,” in Inter-national Conference on Learning Representations (2017).

[2] Ian Goodfellow, Yoshua Bengio, Aaron Courville, andYoshua Bengio, Deep learning, Vol. 1 (MIT press Cam-bridge, 2016).

[3] Johannes Klicpera, Aleksandar Bojchevski, and StephanGnnemann, “Combining neural networks with personal-ized pagerank for classification on graphs,” in Interna-tional Conference on Learning Representations (2019).

[4] Petar Velikovi, Guillem Cucurull, Arantxa Casanova,Adriana Romero, Pietro Li, and Yoshua Bengio, “Graphattention networks,” in International Conference onLearning Representations (2018).

[5] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley,Oriol Vinyals, and George E Dahl, “Neural message pass-ing for quantum chemistry,” in Proceedings of the 34thInternational Conference on Machine Learning-Volume70 (2017) pp. 1263–1272.

[6] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Al-varo Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Ma-linowski, Andrea Tacchetti, David Raposo, Adam San-toro, Ryan Faulkner, et al., “Relational inductive bi-ases, deep learning, and graph networks,” arXiv preprintarXiv:1806.01261 (2018).

[7] Paul W Holland, Kathryn Blackmond Laskey, and

Samuel Leinhardt, “Stochastic blockmodels: Firststeps,” Social networks 5, 109–137 (1983).

[8] Darko Hric, Tiago P Peixoto, and Santo Fortunato,“Network structure, metadata, and the prediction ofmissing nodes and annotations,” Physical Review X 6,031038 (2016).

[9] Laura Florescu and Will Perkins, “Spectral thresholds inthe bipartite stochastic block model,” in Conference onLearning Theory (2016) pp. 943–959.

[10] Jonathan S Yedidia, William T Freeman, and YairWeiss, “Understanding belief propagation and its gen-eralizations,” Exploring artificial intelligence in the newmillennium 8, 236–239 (2003).

[11] Aurelien Decelle, Florent Krzakala, Cristopher Moore,and Lenka Zdeborova, “Asymptotic analysis of thestochastic block model for modular networks and its al-gorithmic applications,” Physical Review E 84, 066106(2011).

[12] Pan Zhang, Cristopher Moore, and Lenka Zdeborova,“Phase transitions in semisupervised clustering of sparsenetworks,” Physical Review E 90, 052802 (2014).

[13] Simon Osindero and Geoffrey E Hinton, “Modeling im-age patches with a directed hierarchy of markov randomfields,” in Advances in neural information processing sys-tems (2008) pp. 1121–1128.

[14] Marc Mezard and Andrea Montanari, Information,

12

Physics and Computation (Oxford University press,2009).

[15] Hidetoshi Nishimori, “Exact results and critical prop-erties of the ising model with competing interactions,”Journal of Physics C: Solid State Physics 13, 4071 (1980).

[16] Yukito Iba, “The nishimori line and bayesian statistics,”Journal of Physics A: Mathematical and General 32,3875 (1999).

[17] H.A. Bethe, “Statistical theory of superlattices,” Proc.R. Soc. London A 150, 552–575 (1935).

[18] Elchanan Mossel, Joe Neeman, and Allan Sly, “A proofof the block model threshold conjecture,” Combinatorica38, 665–708 (2018).

[19] JRL De Almeida and David J Thouless, “Stability of thesherrington-kirkpatrick solution of a spin glass model,”Journal of Physics A: Mathematical and General 11, 983(1978).

[20] Harry Kesten and Bernt P Stigum, “Additional limittheorems for indecomposable multidimensional galton-watson processes,” The Annals of Mathematical Statis-tics 37, 1463–1481 (1966).

[21] Harry Kesten and Bernt P Stigum, “Limit theoremsfor decomposable multi-dimensional galton-watson pro-cesses,” Journal of Mathematical Analysis and Applica-tions 17, 309–338 (1967).

[22] Svante Janson, Elchanan Mossel, et al., “Robust recon-struction on trees is determined by the second eigen-value,” The Annals of Probability 32, 2630–2649 (2004).

[23] Leon Danon, Albert Diaz-Guilera, Jordi Duch, andAlex Arenas, “Comparing community structure identi-fication,” Journal of Statistical Mechanics: Theory andExperiment 2005, P09008 (2005).

[24] Pan Zhang, “Evaluating accuracy of community detec-tion using the relative normalized mutual information,”Journal of Statistical Mechanics: Theory and Experi-ment 2015, P11006 (2015).

[25] Arthur P Dempster, Nan M Laird, and Donald B Ru-bin, “Maximum likelihood from incomplete data via theem algorithm,” Journal of the Royal Statistical Society:Series B (Methodological) 39, 1–22 (1977).

[26] Florent Krzakala, Cristopher Moore, Elchanan Mossel,Joe Neeman, Allan Sly, Lenka Zdeborova, and PanZhang, “Spectral redemption in clustering sparse net-works,” Proceedings of the National Academy of Sciences110, 20935–20940 (2013).

[27] Diederik P Kingma and Jimmy Ba, “Adam: A methodfor stochastic optimization,” in International Conferenceon Learning Representations (2015).

[28] Pan Zhang and Cristopher Moore, “Scalable detection ofstatistically significant communities and hierarchies, us-ing message passing for modularity,” Proceedings of theNational Academy of Sciences 111, 18144–18149 (2014).

[29] Matthias Fey and Jan E. Lenssen, “Fast graph repre-sentation learning with PyTorch Geometric,” in ICLRWorkshop on Representation Learning on Graphs andManifolds (2019).

[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser,and Illia Polosukhin, “Attention is all you need,” in Ad-vances in neural information processing systems (2017)pp. 5998–6008.

[31] Felix Wu, Amauri Souza, Tianyi Zhang, ChristopherFifty, Tao Yu, and Kilian Weinberger, “Simplifying

graph convolutional networks,” in Proceedings of the 36thInternational Conference on Machine Learning, Vol. 97(PMLR, 2019) pp. 6861–6871.

[32] Pan Zhang, “Robust spectral detection of global struc-tures in the data by learning a regularization,” in Ad-vances In Neural Information Processing Systems (Cur-ran Associates, Inc., 2016) pp. 541–549.

[33] Wayne W Zachary, “An information flow model for con-flict and fission in small groups,” Journal of anthropolog-ical research 33, 452–473 (1977).

[34] Lada A Adamic and Natalie Glance, “The political blo-gosphere and the 2004 us election: divided they blog,”in Proceedings of the 3rd international workshop on Linkdiscovery (ACM, 2005) pp. 36–43.

[35] https://github.com/pengfzhou/BPGCN.[36] Yann LeCun, Yoshua Bengio, et al., “Convolutional net-

works for images, speech, and time series,” The handbookof brain theory and neural networks 3361, 1995 (1995).

[37] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,“Imagenet classification with deep convolutional neuralnetworks,” in Advances in neural information processingsystems (2012) pp. 1097–1105.

[38] Joan Bruna, Wojciech Zaremba, Arthur Szlam, andYann LeCun, “Spectral networks and locally connectednetworks on graphs,” in 2nd International Conference onLearning Representations (2014).

[39] David K Hammond, Pierre Vandergheynst, and RemiGribonval, “Wavelets on graphs via spectral graph the-ory,” Applied and Computational Harmonic Analysis 30,129–150 (2011).

[40] Jason Weston, Frederic Ratle, Hossein Mobahi, andRonan Collobert, “Deep learning via semi-supervisedembedding,” in Neural Networks: Tricks of the Trade(Springer, 2012) pp. 639–655.

[41] Zhilin Yang, William W Cohen, and Ruslan Salakhut-dinov, “Revisiting semi-supervised learning with graphembeddings,” in Proceedings of the 33rd InternationalConference on International Conference on MachineLearning-Volume 48 (JMLR. org, 2016) pp. 40–48.

[42] Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty,“Semi-supervised learning using gaussian fields and har-monic functions,” in Proceedings of the 20th Internationalconference on Machine learning (ICML-03) (2003) pp.912–919.

[43] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena,“Deepwalk: Online learning of social representations,” inProceedings of the 20th ACM SIGKDD international con-ference on Knowledge discovery and data mining (ACM,2014) pp. 701–710.

[44] Qing Lu and Lise Getoor, “Link-based classification,” inProceedings of the 20th International Conference on Ma-chine Learning (ICML-03) (2003) pp. 496–503.

Appendix A: Belief propagation equations

Combine the Boltzmann distribution Eq. (1), the like-lihood function Eq. (2), and the variational distributionEq. (7), by minimizing the Bethe free energy with respectto constraints subject to the normalizations of marginals,one arrives at the standard form of belief propagationequations [10] for JSBM:

13

ψi→jti =αtiZi→j

∏k∈∂i\j

∑tk

ptitkψk→itk

∏k/∈∂i\j

∑tk

(1− ptitk)ψk→itk.∏µ∈∂i

∑tµ

qtitµψµ→itµ

∏µ/∈∂i

∑tµ

(1− qtitµ)ψµ→itµ

ψi→µti =αtiZi→µ

∏k∈∂i

∑tk

ptitkψk→itk

∏k/∈∂i

∑tk

(1− ptitk)ψk→itk·∏

ν∈∂i\µ

∑tν

qtitνψν→itν

∏ν /∈∂i\µ

∑tν

(1− qtitν )ψν→itν

ψµ→itµ =βtµZµ→i

∏j /∈∂µ\i

∑tj

qtµtjψj→µtj

∏j∈∂µ\i

∑tj

(1− qtµtj )ψj→µtj . (A1)

where ψi→jti , ψi→µti , and ψµ→itµ are cavity probabilities.Marginal probabilities are estimated as a function of cav-ity probabilities as

ψiti =αtiZi→j

∏k∈∂i

∑tk

ptitkψk→itk

∏k/∈∂i

∑tk

(1− ptitk)ψk→itk

.∏µ∈∂i

∑tµ

qtitµψµ→itµ

∏µ/∈∂i

∑tµ

(1− qtitµ)ψµ→itµ

ψµtµ =βtµZµ→i

∏j /∈∂µ

∑tj

qtµtjψj→µtj

∏j∈∂µ

∑tj

(1− qtµtj )ψj→µtj .

(A2)

Since we have nonzero interactions between every pairof nodes, in Eq. (A1) we have in total n(n − 1) + 2mnmessages. This results to an algorithm where even a sin-gle update takes O(n2) time, making it suitable only fornetworks of up to a few thousand nodes. Fortunately,for large sparse networks, i.e., when n, m is large andptitj = qtitµ = O(1/n), we can neglect terms of sub-leading order in the equations. In this case, it is as-sumed that i or µ sends the same message to all its non-neighbors j, and these messages are viewed as represent-ing an external field. By this means, now we only need tokeep track of 2M messages, where M is the total numberof all edges and hyperedges, and each update step takesO(n+m) time.

Suppose that j /∈ ∂i, we have :

ψi→jti =αtiZi→j

∏k∈∂i

∑tk

ptitkψk→itk

∏k/∈∂i\j

(1−∑tk

ptitkψk→itk

)

.∏µ∈∂i

∑tµ

qtitµψµ→itµ

∏µ/∈∂i

∑tµ

(1− qtitµ)ψµ→itµ

= ψiti +O(1/n) (A3)

Similarly, we have ψµ→itµ = ψµtµ +O(1/m). To the leadingorder, the messages on non-edges do not depend on the

target node. For nodes with j ∈ ∂i, we have

ψi→jti =αtiZi→j

∏k∈∂i\j

∑tk

ptitkψk→itk

∏k/∈∂i

(1−∑tk

ptitkψk→itk

)

.∏µ∈∂i

∑tµ

qtitµψµ→itµ

∏µ/∈∂i

(1−∑tµ

qtitµψµ→itµ )

' αtie−hti

Zi→j

∏k∈∂i\j

∑tk

ptitkψk→itk

∏µ∈∂i

∑tµ

qtitµψµ→itµ

(A4)

where terms having O(1/n) and O(1/m) contribution to

ψi→jti combining to the definition of the auxiliary externalfield:

hti =∑k

∑tk

ptitkψktk

+∑µ

∑tµ

qtitµψµtµ (A5)

Applying the same approximations to the external fieldsacting on feature nodes, one finally arrives at BP equa-tions (3).

Appendix B: Detectability transition analysis usingnoise perturbations

On a graph generated by the JSBM with n item nodesand m feature nodes, the probability that an item nodehas k1 neighboring item nodes p1(k1) follows a Poissondistribution with average degree c1, and the probabilityof it being associated with k2 feature nodes p2(k2) followsa Poisson distribution with average degree c2. Similarly,the probability that a feature node has k3 neighboringitem nodes p3(k3) follows a Poisson distribution with av-erage degree c3 = nc2/m. Considering a branching pro-cess on a graph generated by JSBM with infinite size, theaverage branching ratio of the process is related to theexcess degree which is defined upon the average numberof neighbors. The average excess degree of an item nodeis computed as

c1 =

∑k1k1(k1 − 1)p1(k1)∑k1k1p1(k1)

= c1. (B1)

14

Similarly we have c2 = c2 as well, and the excess degreeof feature nodes is

c3 =

∑k2p2(k2)k2(k2 − 1)n/m∑

k2k2p2(k2)

= c2n/m = c3. (B2)

Consider the noise propagation process on a tree graphas depicted in Fig. 2, with the number of depth l→∞. Inthe tree, odd layers contain exclusively item nodes, andeven layers contain feature nodes (blue boxes) and edges(green boxes) which are connected to item nodes in oddlayers. Assume that on the leaves of the tree (nodes onthe l-th layer) the paramagnetic fixed point is perturbedas

ψvl→vl−1

tvl= αtvl + δ

vl→vl−1

tvl. (B3)

where vl represent an item node on the l layer, tvl is labelof vl, v0 corresponds to the root node i in Fig. 2. Now letus investigate the influence of the perturbation, from themessage on any leaf, to the message on the root node.For simplicity, first choose one path only containing itemnodes (i.e., not connected via any feature node) and lat-ter generalize to paths containing both item nodes andfeature nodes. We define the Jacobian matrix Ti→j withrespect to the message passing from an item node i toanother item node j along an edge (ij), and the Jaco-bian matrix Tµ→i corresponding to the message passingfrom a feature node µ to an item node i, and similarliythe matrix Ti→µ for the backward passage. Elements ofthese Jacobian matrices are formulated as

T i→jtitk=∂ψj→ktj

∂ψi→jti

∣∣αti

= αti(nptitkc1

− 1),

Tµ→ititµ =∂ψi→jti

∂ψµ→itµ

∣∣αti ,βtµ

= αti(mqtµtic2

− 1),

T i→µtitµ =∂ψµ→jtµ

∂ψi→µti

∣∣αti ,βtµ

= βtµ(mqtitµc2

− 1). (B4)

These matrices represent propagation strength (3) be-tween two messages in the vicinity of the paramagneticfixed point. We can see that, all three matrices are in-dependent of node indices, only depend on the type ofthe nodes. If the path only contains edges, the perturba-tion δv0→xtv0

on the root node induced by the perturbation

δvl→vl−1

tvlon the leaf node can be written as

δv0→xtv0=

∑tvd :d=1,...,l

l−1∏d=0

T i→jtvd ,tvd+1δvl→vl−1

tvl, (B5)

or in the vector form, δv0→x = (Ti→j)lδvl→vl−1 . Nowconsider the path contains both edges and hyper edges:every time a hyper edge is passed through, Ti→µ andTµ→i transmits to Ti→j ; therefore, the total weight act-ing on the path is

δv0→x = (Ti→j)s(Ti→µTµ→i)(l−s)δvl→vl−1

where s is the number of edges and l − s is the numberof feature nodes on this path.

For l→∞, (Ti→j)s(Ti→µTµ→i)(l−s) is dominated bythe product of the largest eigenvalues, λA, λF , and λF ′ ofthe three matrices; notice that λF = λF ′ . So the aboveequation can be written as

δv0→x = (λA)s(λF )2(l−s)δvl→vl−1 .

Then consider the collection of all perturbations on theroot node v0 from all leaves. Obviously the mean is 0, andthe variance on the t-th component of the perturbationvector is computed as⟨

(δv0→xt )2⟩

=

⟨ l∑s=0

(ls)∑1

(c1)s(c2c3)l−s∑vl=0

λAsλF

2(l−s)δvl→vl−1

2⟩

=

l∑s=0

(ls)∑1

(c1)s(c2c3)l−s∑vl=0

λA2sλF

4(l−s) ⟨(δvl→vl−1

t )2⟩

=

l∑s=0

(l

s

)(c1λA

2)s(c2c3λF4)l−s

⟨(δvl→vl−1

t )2⟩

= (c1λA2 + c2c3λF

4)l⟨(δvl→vl−1

t )2⟩. (B6)

Here, we have made use of the property that all pertur-bations on different leaves are independent. Obviously,the paramagnetic fixed point is locally unstable underrandom perturbations when (c1λA

2 + c2c3λF4)l > 1, so

the phase transition of detectablity locates at

c1λA2 + c2c3λF

4 = 1. (B7)

Appendix C: Graph convolution networks and thespectral localization problem of linear convolution

kernels

Convolution networks [36] have been proved to be oneof the most successful models for image classifications [37]and many other machine learning problems [2]. The suc-cess of CNN is credited to the convolution kernel whichis a linear operator defined on grid-like structures. How-ever, in recent years, a significant amount of attentionhas been paid to generalizing convolutional operations tographs, which do not have grid structures.

In [38], authors propose to define convolutional layerson graphs that operate on the spectrum of the graphLaplacian:

Xl+1 = σ(Λ ?Xl) = σ(VΛVTXl

), (C1)

where Xl is the state in the lth layer, V is the eigenvectormatrix of the graph Laplacian L = I−D−

12 AD−

12 (A is

adjacency matrix and D is the diagonal matrix on nodedegrees), and σ(·) is an element-wise non-linear activa-tion function, and Λ is a diagonal matrix representing

15

a kernel in the frequency (graph Fourier) domain. Usu-ally Λ contains only several non-zero elements, workingas cut-offs to the frequency in the Fourier domain, be-cause it is believed that using only a few eigenvectors ofthe graph Laplacian are sufficient for describing smoothstructures of the graph. Only computing leading eigen-vectors is also computationally efficient.

Even though, computing only a few eigenvectors of theLaplacian of large graphs could be quite cumbersome. In[39], authors proposed to parametrize the kernel in thefrequency domain through a truncated series expansion,using the Chebyshev polynomials up to K-th order:

Xl+1 = σ(Λ ?Xl) ≈ σ

[K∑k=0

ckTk

(2

λmaxL− I

)Xl

],

(C2)where ck denotes coefficients, TK(x) = 2xTk−1(x) −Tk−2(x) are recursively defined Chebyshev polynomials,and λmax is the largest eigenvalue. In [1] the authorslimited the convolution operation to K = 1, and approx-imate λmax = 2, then obtain

Xl+1 = σ(Λ ?Xl) ≈ θ1X − θ2D− 1

2 AD−12 ,

with two free parameters θ1 and θ2. Further, [1] re-stricted the two parameters by specifying θ = θ1 = −θ2,and introduced a normalization trick

I + D−12 AD−

12 → D−

12 AD−

12 ,

where A = A + I, and Di = 1 +∑j Aij . Finally, upon

generalizing to node features of κ components, that is,to the κ-channel signal X ∈ Rn×κ, one arrives at theGCN [1], with the forward model taking form of

Xl+1 = σ(AXlW), (C3)

where σ(·) is an activation function such as ReLU, Xl ∈Rn×κl is the network state at layer l (n denotes the num-ber of nodes in the graph, κl is the dimension of stateat layer l). Matrix W ∈ Rκl+1,κl is the trainable weight

matrix at the l-th layer, and the propagator A (20) isresponsible for convolving the neighborhood of a node,which isshared over the whole network.

It has been shown in [1] that GCN significantly out-performs related standard methods, including manifoldregularization (ManiReg) [40], Semi-supervised embed-ding (SemiEmb) [41], label propagation (LP) [42], skip-gram based graph embedding (DeepWalk) [43], the it-erative classification algorithm in conjunction with twoclassifiers taking care of both local node features and ag-gregations [44], and the recently proposed Planetoid [41].

For many further developments of GCNs in recentyears, almost all of them can be understood as an ef-fective object representation starting from the originalfeature vector and then projected forward by multiplyingfinite times of the linear convolution kernel. So it is recog-nized that the main principle of graph convolution kernels

is inspired by the spectral properties of linear operators,such as Laplacians, normalized Laplacians [1, 31], ran-dom walk matrix [3] etc. The underlying assumption isthat the eigevectors of the graph convolution kernel con-tain global information about group labels, which can berevealed during the forward propagation of GCNs.

However, it is known that this assumption may nothold on large sparse networks, because the eigenvectorsof conventional linear operators such as graph Lapla-cians are subject to the localization problem, induced bythe fluctuation of degrees or the local structures of thegraph [26, 32]. Even on random graphs where the Pois-son degree distribution is rather concentrated, the local-ization problem is still significant on large graphs. Forthe adjacency matrix, it is well known that the largesteigenvalue is bounded below by the squared root of the

largest node degree (which grows as log(n)log log(n) ), and di-

verges on large graphs when the number of nodes n→∞.Thus the corresponding eigenvectors only report informa-tion of the largest degrees. For normalized matrices suchas the normalized Laplacian, there are many eigenvaluesthat are very close to 0, with corresponding eigenvectorsreporting information about local dangling sub-graphsrather than global structures related to the group labels.We refer to [26, 32] for detailed analysis of spectrum lo-calizations of graph Laplacians, and for the comparisonbetween spectral algorithms using different operators onlarge sparse graphs. Unfortunately, real-world networksare usually sparse, as we can see from Table I that (al-though they might not be large enough) the citation net-works we used for experiments all have an average degreearound 4, which is extremely low.

On large synthetic networks the sparsity issue for con-ventional GCNs is revealed more clearly (see main text).We carried out extensive experiments to verify our anal-ysis (in Fig. 6).

Appendix D: Belief Propagation Graph ConvolutionNetwork(BPGCN)

On a graph generated by JSBM with n item nodes, mfeatures nodes, nc1/2 edges and nc2 hyper edges, i.e. theaverage degree of item nodes in the connectivity graph isc1 and the average degree of the attribute graph is c2. Wedefine matrices storing cavity messages Ψi→j

l ∈ Rnc1×κ,

Ψi→µl ∈ Rnc2×κ and Ψµ→i

l ∈ Rnc2×κ, and marginal ma-trices Ψi

l ∈ Rn×κ and Ψµl ∈ Rm×κ, where l indicates

the cavity messages after the l-step of iterations, TheBP equations (3) can be written in the form of matrix

16

multiplications:

Ψµl+1 = Softmax

[UI log(Ψi→µ

l P)]

Ψil+1 = Softmax

[UII log(ΨI→i

l Q) + UIII log(Ψi→jl P)

]Ψi→jl+1 = Softmax

[BI log(Ψµ→i

l Q) + BII log(Ψi→jl P)

]Ψi→µl+1 = Softmax

[BIII log(Ψi→j

l P) + BIV log(Ψµ→il Q)

]Ψµ→il+1 = Softmax

[BV log(Ψi→µQ)

](D1)

where P, Q are affinity matrices of size κ× κ.Kernel matrices UI to UIII, BI to BV are non-

backtracking matrices [26] that encode adjacency infor-mation of cavity messages and marginals, which are de-fined as:

Uµ,i→νI = δµν ∈ 0, 1m×nc2

Ui,µ→jII = δij ∈ 0, 1n×nc2

Ui,k→lIII = δil ∈ 0, 1n×nc1

Bi→j,µ→lI = δil(1− δµj) ∈ 0, 1nc1×nc2

Bi→j,k→lII = δil(1− δkj) ∈ 0, 1nc1×nc1

Bi→µ,j→lIII = δil(1− δµj) ∈ 0, 1nc2×nc1

Bi→µ,ν→lIV = δil(1− δµν) ∈ 0, 1nc2×nc2

Bµ→i,j→νV = δµν(1− δij) ∈ 0, 1nc2×nc2 . (D2)

Therefore, the matrix multiplication form of the parallel-updated BP equations can be viewed as a forward modelof a neural network. When the above equations propa-gate, and are truncated after L steps of iterations, theresulting algorithm scheme is termed as BPGCN, withL layers. States in the last layer are extracted as theoutput of BPGCN for computing the cross-entropy losson training labels (18). Matrices Q and P are train-able parameters of the BPGCN and are updated usingthe back propagation algorihtm. Input of the neural net-work are probability-normalized random initial cavitiesand marginals. In order to accelerate the training processof BPGCN, we use an adjustable external field strengthγ as a hyper parameter. For example, suppose node iis in the training set, i.e. we have label ti for node i, aterm γ log(0.1, 0.1, 0.9, ....1) is added to (D1) inside theSoftmax function. When γ approaches infinity, node i ispinned to label ti [12], and is used as an hyper parameteradjusted by the validation set. For nodes in the trainingset, cavity probabilities and marginals are initialized as(0, 1, ..., 0). Thus the parameters of BPGCN are P andQ, and the hyper parameters are ε1, ε2, and γ. The hy-per parameters we used in the experiments on real-worlddatasets (Table. I are: for Citeseer and Cora, L = 5and external strength γ = 2.0; on Cora we set ε1 = 0.1,ε2 = 0.6; on Citeseer we set ε1 = 0.1, ε2 = 0.5. ForPubmed, L = 2, external strength γ = 1.5, ε1 = 0.3, andε2 = 0.8; when the attribute graph is pre-processed witha MLP to offer BPGCN as a prior or an external field,

0 10 20 30 40 50

epoch

0

0.2

0.4

0.6

0.8

1

Ove

rla

p

BPGCN-train

BPGCN-test

GCN-train

GCN-test

MLP-train

MLP-test

FIG. 5. Training and test overlap (Eq. (14)) as a functionof epoch on a synthetic graph generated by JSBM, with n =m = 10000, c1 = c2 = 10, ε1 = 1, and ε2 = 0.1.

we set L = 12, external strength γ = 0.5, ε1 = 0.1. Forboth Karate Club and Political Blogs networks,L = 5, external strength γ = 0.5, and ε1 = 0.1 (ε2 is notapplicable).

Appendix E: More comparisons on syntheticnetworks

We carried out extensive numerical experiments onsynthetic networks generated by JSBM with various pa-rameters. The overlap of BP, BPGCN and reference al-gorithms BPGCN, GCN, APPNP, GAT and SGCN arecompared in Fig.6. In the top row, the average degree ofthe connectivity graph is fixed to c1 = 4, 6, and 10 fromleft to right. Due to the spectrum localization problemof graph Laplacians, as we have discussed earlier, whenc1 is fixed to 4, GCN, APPNP, GAT, and SGCN workmuch worse than BPGCN, even when c2 is large. Wealso see that when c1 is large, most of these tested GCNswork well, even for a very low c2. In the second row, theaverage degree of the attribute graph c2 is fixed and c1varies. Performances of reference GCNs approaches theperformance of BPGCN, when c1 gradually increases.

In the third row of the figure, c1, c2 and ε1 are fixed andε2 varies. On the left, c1 = c2 = 4, ε1 = 0.1. The graphsare in the sparse regime so conventional GCNs do notwork well due to the sparsity issue. In the middle, c1 =c2 = 10, ε1 = 0.4. Graphs are not sparse, but the edgescontains relatively little information about group labels.Results show that SGCN has trouble in this regime evenwhen ε2 is very small. On the right c1 = c2 = 10, andε1 = 0.1. The graphs are not in the sparse regime, andedges contain enough information about labels. We seethat all GCNs except APPNP perform reasonably well;APPNP has trouble with low ε2, probably because annon-optimal ω parameter was learned.

In the last row of the figure, c1, c2, and ε2 are fixed,and the inverse signal-to-noise ratio ε1 varies. On the

17

left, c1 = c2 = 4, ε2 = 0.1. We see that BPGCN startsto have a large variance, but the overlap is on averagemuch higher than other GCNs, due to the sparsity of thegraphs. In the middle, c1 = c2 = 10, ε2 = 0.4, meaningthat the hyper edges in the attribute graph contain littleinformation of labels, and the major information comesfrom the connectivity graph which is not sparse. In thiscase we see that the performances of reference GCNs arecomparable to BP and BPGCN, since there is no spar-sity issue. On the right, c1 = c2 = 10, and ε2 = 0.1.There is no sparsity issue for conventional GCNs, andthe attribute graph contains enough information aboutthe labels. However, the result is quite surprising: whileBP and BPGCN gives almost 100% classification accu-racy in the full range of ε1, conventional GCNs yield verylow accuracy with ε1 larger than 0.5. This phenomenonimplies that, when the edges of the graph are quite noisy,information from the features is not well-extracted byconventional GCNs.

To understand the reason for this observation, in Fig. 5we plot the training process of GCN, MLP and BPGCN,on a graph generated with c1 = c2 = 10, ε1 = 1 andε2 = 0.1. In this case it means that the connectivitygraph is a pure random graph, with edges containingno information of the label at all (ε1 = 1), while theattribute graph contains adequit informatino about the

ground-truth (ε2 = 0.1). We can see that BPGCN yieldsvery high accuracy on the training set and the test set,even from the beginning, because ε2 is so small that theinitial values for BPGCN are already good enough. Atthe begining of the training process, MLP has low train-ing accuracy as well as low test accuracy. But after sev-eral epochs of training, its training accuracy goes to 1,and it also generalizes very well to the test set on whichthe accuracy gradually increases to around 0.6. However,this generalization does not happen for GCN, as we cansee that although GCN fits very well to the training set,the overlap for the test set is only slighly above 0.2, i.e.,result of a random guess. Obviously, GCN overfits to thetraining set; the reason is that GCN assumes that edgesalways contain information of the labels, even when theunderlying graph is purely random. Moreover, we candeduct directly from the formula of APPNP (23) that,since APPNP uses ratios ω and 1− ω to weight the con-tributions of edges and features, the best possible resultgiven by APPNP on graphs of the above type, whereedges contain no information and features contain all theinformation, is almost equal to the result of MLP. In-deed, we have tested with ε1 = 1, ε2 = 0.1, c1 = c2 = 10;APPNP achieves an overlap around 0.6, while MLP fallsaround 0.66.

18

4 8 12 16 20c2

0.50

0.75

1.00Ov

erlap

c1=4, ε1=0.1, ε2=0.2

BPGCNBPGCN

APPNPGATSGCN

4 8 12 16 20c2

0.50

0.75

1.00

Overlap

c1=6, ε1=0.1, ε2=0.2

BPGCNBPGCN

APPNPGATSGCN

4 8 12 16 20c2

0.8

0.9

1.0

Overlap

c1=10, ε1=0.1, ε2=0.2

BPGCNBPGCN

APPNPGATSGCN

4 8 12 16 20c1

0.50

0.75

1.00

Overlap

c2=4, ε1=0.1, ε2=0.2

BPGCNBPGCN

APPNPGATSGCN

4 8 12 16 20c1

0.50

0.75

1.00

Overlap

c2=6, ε1=0.1, ε2=0.2

BPGCNBPGCN

APPNPGATSGCN

4 8 12 16 20c1

0.50

0.75

1.00

Overlap

c2=10, ε1=0.1, ε2=0.2

BPGCNBPGCN

APPNPGATSGCN

0.00 0.25 0.50 0.75 1.00ε2

0.4

0.6

0.8

1.0

Overlap

c1= c2=4, ε1=0.1

0.00 0.25 0.50 0.75 1.00ε2

0.2

0.4

0.6

0.8

1.0

Overlap

c1= c2=10, ε1=0.4

0.00 0.25 0.50 0.75 1.00ε2

0.7

0.8

0.9

1.0

Overlap

c1= c2=10, ε1=0.1

0.00 0.25 0.50 0.75 1.00ε1

0.2

0.4

0.6

0.8

1.0

Overlap

c1= c2=4, ε2=0.1

0.00 0.25 0.50 0.75 1.00ε1

0.2

0.4

0.6

0.8

1.0

Overlap

c1= c2=10, ε2=0.4

0.00 0.25 0.50 0.75 1.00ε1

0.2

0.4

0.6

0.8

1.0

Overlap

c1= c2=10, ε2=0.1

FIG. 6. Overlap (Eq. (14)) obtained by BP, BPGCN and reference GCNs on synthetic networks generated by JSBM withdifferent parameters. All networks have n = m = 10000, and group numbers κ = 5. The fraction of training labels is fixed atρ = 0.05. Each data point is averaged over 10 random instances. The other parameters are printed above each subfigure.