Comments on ”Fast Transforms: BandedMatrices with Banded Inverses”

The Narrator

November 08, 2015

Theory behind Banded Matrices with Banded Inverses

In general, the aim of a communication engineer is to be able to get the information ascorrect as possible by using the noisy (distorted) signal as it is shown in Fig.1. Sometimes,the original signal is expressed using different basis (domain) to make use of properties of thetransformed signal. An important property, e.g., can be the increased number of zeros in thetransformed signal vector so that less number of bits is sent. We can introduce a transformby using the equation y = Ax where A ∈ Rn∗n and y, x ∈ Rn. Obviously, such a transformshould be invertible to be able to reverse back the transformation. For this reason, A isnon-singular. Thus, the inverse transform can be written as x = A−1y, and this one-to-onemapping does not introduce any information loss.

According to the properties of the applications and signals, one may need different transfor-mations. Some applications (e.g. HD Broadcasting) need fast forward and inverse transfor-mation to be able to send or decode (compressed) data on time. This idea makes the bandedmatrices with banded inverses very special class of the matrices as they provide very fastforward and inverse transformation thanks to their special structure.

The question, now, is how to generalize an identifying property about the banded matriceswith banded inverses. According to the theorem in the paper, these matrices can be expressedas a product of N block-diagonal matrices, namely A = F1F2...FN , where Fi is block-diagonaland non-singular for any i = 1, 2, .., N . At this point, one may argue that the product of

Source X A Black Box A−1 x = A−1yx y = Ax y

Figure 1: An example of a typical transformation scheme

1

block-diagonal matrices gives another block-diagonal matrix. That seems true, but the ideais arranging the matrices in a way that the blocks in one of the matrices in the productshould be offset from the blocks in others (this will be clearer with the example below). Inaddition, each block-diagonal matrix composed of 2 by 2 and 1 by 1 blocks (not always aconstant size).1 Once this idea is understood, one can easily deduce that the inverse of thisbanded matrix will also be banded, leading to a fast inverse transformation.2

Another very important property of this kind of matrices is as follows: Let A be a bandedmatrix with a banded inverse A−1. The bandwidths are w and W, respectively. Then,the number of block-diagonal matrices can be upper bounded by a function C(w3 + W 3),independent of the matrix dimension n. This independence of n implies that the block-diagonal matrices with invertible blocks are generator matrices for the singly infinite bandedmatrices with banded inverses as n goes to infinity.

Example 1

F1 =

1

1 21 1

2 11 2

, F2 =

1 11 2

3 12 1

2

− > A = F1F2 =

1 1 01 2 6 21 2 3 1 0

0 4 2 22 1 4

A−1 =

2 1 −2−1 −1 2 00 1 −1 −2

313

−2 2 2 −10 −1

613

As it is shown in the example, the inverse of the banded matrix that is constructed in theway explained above is also banded (offset is coming from the scalar in the first entry ofF1). If we chose a random banded matrix, with high probability the inverse would be afull matrix instead of banded. Therefore, we can make a more general conclusion regardingthis properties: If one can design a banded transformation matrix by using the product ofnon-singular block-diagonal matrices (or equivalently, one can show that a banded matricescan be factorized into a product non-singular block-diagonal matrices), then s/he makessure that the inverse transformation will also be requires less calculation, hence, quickerexecution.3

1The reason for limiting block-diagonal matrices to be 2 by 2 or 1 by 1 matrices is to reduce the bandwidthof matrix A.

2Note that the inverse of a block-diagonal matrix is block-diagonal, and A−1 = F−1N F−1

N−1...F−11 .

3According to Prof. Strang, for a band matrix, ordinary elimination requires w2n instead of n3

3 for a fullmatrix, where n is the dimension of the matrix and w is the bandwidth of banded matrix.

2

Practical Implications

In this part, we will give a few practical examples, in which banded matrices with bandedinverses can be of great importance. The first example can be from Polar Codes. Theidea of polar coding is to introduce a one-to-one mapping between the source symbols andthe channel input symbols, which polarizes the channel.4 By channel polarizing, we meanforming generic channels between input and output symbols by using this transformation.Polar codes achieve capacity with a decoding complexity of O(nlogn). The idea of using abanded matrix as a transformation matrix can be of great benefit in decoding complexitybecause of the less number of calculations. However, one should be careful at choosing asuitable transformation to polarize the channel.

Another example is broadcasting in High Definition TV (HD TV).5 In this scheme, if one isdealing with problems offline, then delay is not very important. However, the main challengecomes into the picture when the high quality content is broadcasted online (e.g., live HDbroadcasting). Assume we have very long sequence of video stream x to be transmitted. Aswe do not have enough resources, e.g. time, we need to compress data. If we can find abanded matrix with a banded inverse and if the resulted matrix y = Ax has less number ofdata bits to be transmitted (i.e., more zeros in the vector y) with the same entropy, thenthe process would be faster, resulting in smaller delay. As the mapping is one-to-one we donot lose any information by doing this transformation.

Lastly, banded matrices can be useful for the Big Data or Machine Learning applications,where one has huge data matrices and there is no obvious structure to be used. For example,in deep learning algorithms, we have very big data matrices without structure. Then, wecan try to find a one-to-one mapping by introducing a banded transformation matrix witha banded inverse. Thanks to this invertible transformation, we may need less number ofcomputations if the dimensions are very big, and the bandwidths of the banded matrices aresmall.

4Note that one can also think about source polarization.5In Image and Video Compression algorithms, this matrices can be of the great importance.

3