1

Two-Dimensional DOA Estimation for L-shapedNested Array via Tensor ModelingFeng Xu, Student Member, IEEE, and Sergiy A. Vorobyov, Fellow, IEEE

Abstract—The problem of two-dimensional (2-D) direction-of-arrival (DOA) estimation for L-shaped nested array is consid-ered. Typically, the multi-dimensional structure of the receivedsignal in co-array domain is ignored in the problem considered.Moreover, the cross term generated by the correlated signal andnoise components degrades the 2-D DOA estimation performanceseriously. To tackle these issues, an iterative 2-D DOA estimationapproach based on tensor modeling is proposed. To develop suchapproach, a higher-order tensor is constructed, whose factormatrices contain the target azimuth and elevation information.By exploiting the Vandermonde structure of the factor matrix, acomputationally efficient tensor decomposition method is thendeveloped to estimate the targets DOA information in eachdimension independently. Then, an eigenvalue-based approachthat exploits a natural coupling of the 2-D spatial parametersis proposed to pair the azimuth and elevation angles. Finally,an iterative method is designed to improve the DOA estimationperformance. Specifically, the cross term is estimated and re-moved in the next step of such iterative procedure on the basisof the DOA estimates originated from the tensor decompositionin the previous step. Consequently, the DOA estimation withbetter accuracy and higher resolution is obtained. The proposediterative 2-D DOA estimation method for L-shaped nested arraycan resolve more targets than the number of real elements, evenwhen the azimuth or elevation angles are identical, which is supe-rior to conventional approaches. Simulation results validate theperformance improvement of the proposed 2-D DOA estimationmethod as compared to existing state-of-the-art DOA estimationtechniques for L-shaped nested array.

Index Terms—2-D DOA estimation, L-shaped nested array,Tensor modeling, Vandermonde factor matrix

I. INTRODUCTION

THE problem of two-dimensional (2-D) direction-of-arrival (DOA) estimation of multiple signals impinging

on an antenna array has attracted considerable attention inseveral applications such as wireless communications, radar,sonar and others [1]–[3]. In these applications, several 2-Darray structures such as uniform rectangular array (URA), cir-cular array and L-shaped array have been considered [4]. Forexample, URA is a widely used array geometry for airborneor spaceborne arrays, where DOA estimation methods such asmultiple signal classification (MUSIC) [5] and estimation of

This work was supported in part by the Academy of Finland under Grant319822 and in part by the China Scholarship Council. This work wasconducted while Feng Xu was a visiting doctoral student with the Departmentof Signal Processing and Acoustics, Aalto University. (Corresponding author:Sergiy A. Vorobyov.)

Feng Xu is with the School of Information and Electronics, Beijing Instituteof Technology, Beijing 100081, China, and also with the Department of SignalProcessing and Acoustics, Aalto University, Espoo 02150, Finland. (e-mail:fengxu@bit.edu.cn, feng.xu@aalto.fi).

Sergiy A. Vorobyov is with the Department of Signal Processing andAcoustics, Aalto University, Espoo 02150, Finland. (e-mail: svor@ieee.org).

signal parameters via rotational invariance technique (ESPRIT)[6], [7] can be straightforwardly conducted. However, it hasbeen established that L-shaped array is superior to URA[8] since the corresponding Carmer-Rao bound (CRB) issignificantly (37%) lower than that of the URA. This propertymeans that L-shaped array can provide a higher accuracy for2-D DOA estimation. Therefore, the study of high resolution2-D DOA estimation methods for L-shaped array has been thefocus of array processing research over the past two decades[9]–[23].

In general, L-shaped array can be divided into two linearsubarrays. Thanks to it, the 2-D DOA estimation problem canbe regarded as two one-dimensional (1-D) DOA estimationproblems, and 1-D high resolution DOA estimation methodscan be generalized to the case of L-shaped array conveniently[9]–[13]. For example, the MUSIC algorithm for solvingtwo 1-D spectrum searching problems is much easier thanconducting a complex 2-D spectrum search. To further reducethe computational complexity, a modified propagator method(PM) has been proposed [9], which avoids the use of matrixsingular value decomposition (SVD). In [10], the azimuthand elevation angles are independently estimated using a 1-D subspace-based method without eigendecomposition. Thestudy of DOA estimation in the presence of mutual couplinghas also been considered for L-shaped array [11]. Note thatthe independent sets of azimuth and elevation angles must beproperly paired after using the 1-D DOA estimation methods.Several approaches have been developed in the literature forthis purpose [12], [13]. To obtain the correct azimuth andelevation pairs, a Toeplitz matrix has been built by exploitingthe cross-correlation matrix (CCM) of the signal receivedby L-shaped array in [12]. The common structure of theabove reviewed approaches for L-shaped array-based 2-DDOA estimation consists of two parts: 1) 1-D DOA estimationfor each subarray, and 2) a proper pair-matching of the twoestimated angle sets. Let us categorize these approaches in thefirst category.

The second category of approaches for L-shaped array-based 2-D DOA estimation [14]–[23], on the contrary, aimsat achieving an automatic pairing during the joint 2-D DOAestimation. By exploiting the fact that the noise component canbe fully eliminated in the CCM, a joint SVD method [14]–[16] has been proposed to improve the 2-D DOA estimationperformance. The auto-pairing of two estimated angle setsis then also achieved by using the eigenvalue decomposition(EVD) of the submatrices constructed from the CCM. In [17],an angle estimation method has been introduced to split thejoint steering vector into two steering vectors in order to fulfill

arX

iv:2

104.

0679

9v1

[ee

ss.S

P] 1

4 A

pr 2

021

2

the L-shaped DOA estimation without pairing. However, thismethod requires one 1-D spectrum search and suffers fromthe angle ambiguity. To reduce the computational burden, asignal subspace-based algorithm has been proposed in [18].This method estimates the noise subspace by rearranging theelements of three vectors, i.e., the first column, the first rowand the diagonal entries of the CCM. A generalization of thePM using the CCM has been developed in [19], where onlylinear operations on the signal matrix have been required.Furthermore, the conjugate symmetry property of the arraymanifold has also been exploited in [20]–[22] to raise thearray aperture and the number of snapshots simultaneously.Recently, a tensor-based approach [23] aiming at increasingthe system degree of freedom (DOF) has been suggested forL-shaped array-based DOA estimation, where the subarrayson both axes are divided into several overlapped subarraysof smaller size. A more general case has been studied in[24], where the authors illustrate that any centrosymmetricarray processing problem can be interpreted in terms of acoupled canonical polyadic decomposition (CPD) problem.By using tensor modeling, the multi-dimensional structure ofthe received signal is exploited and, hence, the estimationperformance can be improved [25], [26].

Meanwhile, a special array geometry named as nested arrayhas been widely investigated [27], [28], largely thanks to thefact that it can be used to detect more sources than the numberof real antenna elements due to the increased DOF in co-array domain. When uniform linear array (ULA) is replacedby linear nested array for L-shaped array, the correspondingarray is named as L-shaped nested array [29]–[33]. L-shapednested array also may enjoy an increased DOF. In [29], [33],the authors consider the 2-D DOA estimation problem astwo independent 1-D DOA estimation problems. For each1-D problem, the spatial smoothing (SS) (see [27]) is useddirectly, while the fourth-order difference co-array is exploitedin [30]. The azimuth and elevation angles are matched bypairing the source powers estimated by two nested subarraysseparately. The signal subspace joint diagonalisation (SSJD)technique [31] is used to conduct 2-D DOA estimation andpair-matching, which can be regarded as a generalization of[20], [21]. Possible holes in the cross-difference co-arrayfor L-shaped nested array can be filled by using obliqueprojection [32], and a virtual CCM with larger aperture canbe constructed to fulfill 2-D DOA estimation with betterperformance.

It is important to dress, however, that the existed methodsfor L-shaped nested array DOA estimation ignore the multi-dimensional structure of the received signal, especially afterSS is applied on both directions. Besides, in the co-arraydomain [33], the signal and noise terms become correlatedand the cross term between them cannot be ignored. Thisunexpected component can degrade the DOA estimation per-formance significantly. To tackle these problems, in this paper,an iterative 2-D DOA estimation method via tensor modelingis developed for L-shaped nested array. The contributions ofthis paper are the following.• Unlike the conventional techniques for L-shaped nested

array which average the received signals of all subarrays

in co-array domain by applying SS, a higher-order tensormodel is designed here to store those received signals inorder to exploit the multi-dimensional structure inherentin the signals. The cross term between the correlatedsignal and noise terms is also considered.

• The parameter identifiability of the designed tensor modelis studied. Based on this study, the number of subarraysfor SS is optimized to maximize the system DOF.

• A computationally efficient tensor decomposition methodis proposed for 2-D DOA estimation. The azimuth andelevation angles are paired by the joint target spatialinformation in the fifth dimension of the designed tensor,which originates from the exploitation of the conjugatesymmetry property.

• An iterative DOA estimation method for for L-shapednested array is proposed. The essence of the iterativemethod is that the cross term is estimated and removedin the next step based on the DOA estimates obtained atthe previous step via tensor decomposition. The estimatedreceived signal is then modified and can be used as aninput for more accurate DOA estimation. Thus, the DOAestimation performance can be improved gradually overiterations.

• Analytical expression for CRB associated with our pro-posed received signal model is derived.

The remainder of this paper is organized as follows. Somepreliminaries about tensors and signal model for L-shapednested array are introduced in Section II. A novel higher-order tensor model for the signal received by L-shaped nestedarray is developed in Section III. The parameter identifiabilityfor this tensor model is studied in the same section with thepurpose to demonstrate advantages of the proposed model,while the optimization of DOF is also presented. In Section IV,an iterative 2-D DOA estimation method for L-shaped nestedarray is proposed via decomposition of the designed signaltensor. Numerical results are presented in Section V in orderto verify the effectiveness of the proposed method. Finally,Section VI draws our conclusion.

Notation: Scalars, vectors, matrices and tensors are repre-sented by lower-case, boldface lower-case, boldface upper-case, and calligraphic letters, e.g., r, r, R, and R, respec-tively. The transposition, Hermitian transposition, inversion,pseudo-inversion, conjugation, outer product, Kronecker prod-uct and Khatri-Rao (KR) product operations are denoted by(·)T , (·)H , (·)−1

, (·)†, ∗, ,⊗, and , respectively, while theoperator vec (·) stacks the elements of a matrix/tensor oneby one to a long vector. The operation denoted as diag(y)returns a diagonal matrix built out of its vector argument, while‖R‖F and ‖R‖ stand for the Frobenius norm and Euclideannorm of R, respectively. Moreover, IM and 0M×N denote theidentity matrix of dimension M ×M and the all-zero matrixof size M × N , respectively, while JM ∈ RM×M standsfor the exchange matrix with ones on the anti-diagonal andzeros elsewhere. For R ∈ CM×N , the n-th column vector and(m,n)-th element are denoted by rn and rmn, respectively,while the m-th element of r ∈ CM is given by rm. Theestimates of R and r are denoted as R and r, respectively,

3

while the noise-free versions of R are r are written as R orr.

II. PRELIMINARIES ON TENSORS AND SIGNAL MODEL

In this section, we introduce some preliminaries about tensor[34]–[36], which will be heavily used later in the paper. Then,the signal model for L-shaped nested array is given.

A. Preliminaries on Tensors

Fact 1 (Kruskal Form Tensor): An N -th order tensorR ∈ CI1×I2×···×IN is presented in Kruskal form if

R =

K∑k=1

λkα(1)k α(2)

k · · · α(N)k , [[λ;A(1),A(2), · · · ,A(N)]]

(1)

where λ , [λ1, λ2, · · · , λK ]T , α(n)k is the k-th column of

A(n) with A(n) ∈ CIn×K being the n-th factor matrix, and Kis the tensor rank. Following this type of tensor presentation,the parallel factor (PARAFAC) decomposition of a rank Ktensor R consists of finding R = [[λ;A(1),A(2), · · · ,A(N)]]so that ||R − R||2F is minimized.

Fact 2 (Tensor Reshape): For an N -th order tensorR ∈ CI1×I2×···×IN , the tensor reshape operator generatesan M -th order tensor T ∈ CJ1×J2×···×JM that satisfiesvec(R) = vec(T ) and

∏Nn=1In =

∏Mm=1Jm. In particular,

consider the set A = 1, 2, · · · , N and M subsetsAm, m = 1, 2, · · · ,M , as partitions of A such thatA1 ∪ · · · ∪ AM = A and Ai ∩ Aj = ∅,∀i 6= j. Then, thetensor reshape operator can be expressed as

T , reshape(R, [A1,A2, · · · ,AM ]). (2)

For example, taking a 5-th order tensor R ∈ CI1×I2×···×I5 ,let A1 = 1, 3, A2 = 2, 4 and A3 = 5. The reshaped3-rd order tensor T is of dimension CI1I3×I2I4×I5 , i.e., J1 =I1 × I3, J2 = I2 × I4, and J3 = I5.Fact 3 (Tensor Unfolding): For an N -th order tensor R =[[λ;A(1),A(2), · · · ,A(N)]] and Λ = diag(λ), the unfoldingof R from the n-th dimension returns a matrix R(n) ∈CI1···In−1In+1···IN×In such that

R(n) =(A(N) · · · A(n+1) A(n−1) · · · A(1)

)Λ(A(n)

)T. (3)

It is worth noting that tensor unfolding can be regardedas a special case of tensor reshape where only two subsetsB1 = 1, 2, · · · , n − 1, n + 1, · · · , N and B2 = nexist. Hence, we define the unfolding operator as R(n) ,unfolding(R, [B1,B2]).

B. Signal Model

Consider an L-shaped nested array that consists of twolinear subarrays arranged along the x-axis and z-axis, asshown in Fig. 1. Each subarray is a two-level nested arraywith N (N is even) elements wherein the element at theorigin belongs to both subarrays. The first level has N/2elements with spacing d1 and the second level has another

x

y

z

k

01

2

1

2

...

...

1N −

1N−(0,0, )

N

( ,0,0)N

targets, 1,...,k K=

( 2,0,0)k

Fig. 1. L-shaped nested array configuration.

N/2 elements with spacing d2. Without loss of generality,let d1 = λ/2 and d2 = (N/2 + 1)d1 [27], where λis the working signal wavelength. Assume that K spatial-temporal uncorrelated narrowband far-field sources are im-pinging on the L-shaped nested array with azimuth and eleva-tion angles (θk, φk)Kk=1. The steering vectors of both subar-rays are given as ax(θ) , [1, e−jπ cos θξ2 , · · · , e−jπ cos θξN ]T

and az(φ) , [1, e−jπ cosφξ2 , · · · , e−jπ cosφξN ]T , respectively,where ξn is the normalized distance between the origin andthe n-th element. Accordingly, the steering matrices can bewritten as

Ax , [ax(θ1),ax(θ2), · · · ,ax(θK)]

Az , [az(φ1),az(φ2), · · · ,az(φK)](4)

respectively.Then, the received signal of the L-shaped nested array at

the time instance t can be expressed as

x(t) = Axs(t) + nx(t)

z(t) = Azs(t) + nz(t)(5)

where t = 1, 2, · · · , Ts with Ts being the signal time du-ration, i.e., the number of snapshots after sampling, s(t) ,[s1(t), s2(t), · · · , sK(t)]T is the signal vector, nx(t) and nz(t)are the additional Gaussian white noise vectors on x-axis andz-axis, respectively.

Taking first x(t) in (5), the auto-correlation matrixof the received signal can be expressed as Rxx =Ex(t)xH(t) = AxRsA

Hx + σ2

nIN , where Rs = diag(p)is the auto-correlation matrix of the source signals, p ,[σ2

1 , σ22 , · · · , σ2

K ]T , and σ2k denotes the power of the k-th

source. Vectorizing Rxx, we have

yxx = vec(Rxx) = (A∗x Ax)p + σ2ne (6)

where e is the result of the identity matrix vectorization. Theterm A∗x Ax can be regarded as the steering matrix of avirtual ULA with larger aperture and more elements in co-array domain.

After removing the repeated rows in yxx and sorting theother remaining rows, the received signal can be written as

yxx = Axp + σ2ne (7)

4

where Ax , [ax(θ1), ax(θ2), · · · , ax(θK)] ∈ C(2S−1)×K

is the steering matrix of the difference co-array, ax(θ) ,[e−jπ cos θ(−S+1), · · · , 1, · · · , e−jπ cos θ(S−1)]T ∈ C2S−1 isthe steering vector in direction θ, e , [0(S−1)×1, 1,0(S−1)×1],and S = N/2(N/2 + 1).

The received signal in co-array domain for z-axis can beanalogously expressed by replacing x and θ in (7) with z andφ, respectively.

III. PROPOSED HIGHER-ORDER TENSOR MODEL FORL-SHAPED NESTED ARRAY

It can be observed that the signal model (7) has onlyone snapshot. Like in the linear nested array case [27], theSS technique can be introduced to increase the number ofsnapshots. In particular, the number of subarrays M and thenumber of elements within one single subarray Q can beselected both equal to S such that the average of the signalcovariance matrices for all subarrays is expressed by the squareof the signal covariance matrix for a ULA subarray. How-ever, the multi-dimensional structure of the received signalin co-array domain is lost after applying the SS techniquebecause of the averaging of the signal covariance matricesfor all subarrays. The targets azimuth and elevation pairinginformation contained in the CCM is also ignored. Because ofthese problems, the DOA estimation performance for L-shapednested array degrades.

To tackle the above mentioned problems, a higher-ordertensor signal model is designed in this section in order to beable to exploit the multi-dimensional structure of the receivedsignal for all subarrays instead of averaging them as in the SStechnique. The conjugate symmetry property can be then usedto achieve a natural pair-matching.

A. High-Order Cross-Correlation Tensor ModelingIn general, the difference co-array can be divided into M

overlapped subarrays, each of them containing Q elements,i.e., M + Q = 2S for M = Q = S. For m-th subarray(here m = 1, 2, · · · ,M ), the received signal in x-axis can beexpressed as

y(m)xx = A(m)

x p + σ2nw(m) (8)

where A(m)x ∈ CQ×K is the steering matrix of the m-th

subarray and w(m) ∈ CQ contains the elements from m-thto (m + Q)-th row of e. Since the difference co-array is aULA, the steering matrix for each subarray is a Vandermondematrix, which satisfies the property that

A(m)x = A(1)

x Ω(m)x (9)

where A(1)x ∈ CQ×K is the reference steering matrix corre-

sponding to the submatrix of Ax with first Q rows, Ω(m)x =

diag(κm−1x ), and κx , [e−jπ cos θ1 , · · · , e−jπ cos θK ]T ∈ CK .

Then, let X , [y(1)xx , y

(2)xx , · · · , y(M)

xx ] ∈ CQ×M denote theconcatenation of the received signals for all M subarrays. Incompact form, X can be written as

X = A(1)x RsK

Tx + σ2

nW

=

K∑k=1

σ2ka

(1)x (θk)k

Tx (θk) + σ2

nW(10)

where Kx , [κ0x,κ

1x, · · · ,κM−1

x ]T ∈ CM×K denotesthe phase rotation matrix between different subarrays,a

(1)x (θk) , [e−jπ cos θk(−S+1), · · · , e−jπ cos θk(−S+Q)]T ∈ CQ

and kx(θk) , [1, e−jπ cos θk , · · · , e−jπ cos θk(M−1)]T ∈ CMare the k-th columns of the matrices A

(1)x and Kx, respec-

tively, and W , [w(1),w(2), · · · ,w(M)] is the noise matrix,which can also be expressed as

W =

[0M×(S−M), IM ,0M×(S−M)

]T, M ≤ S[

0Q×(S−Q), IQ,0Q×(S−Q)

], M > S

(11)

It can be observed that the rank of the noise matrix W isequal to min(M,Q).

Similarly, the matrix form of the received signal for allsubarrays in z-axis can be modeled as

Z = A(1)z RsK

Tz + σ2

nW

=

K∑k=1

σ2ka

(1)z (φk)k

Tz (φk) + σ2

nW(12)

where A(1)z , Kz , a

(1)z (φk) and kz(φk) are defined analogously

to their counterparts in x-axis.To fully exploit the multi-dimensional structure of the

received signal for the subarrays in x-axis and z-axis, the outerproduct of the received signal also need to be considered. Itis given by Rxz = EX Z∗, or equivalently, by

Rxz = [[p2; A(1)x ,Kx, A

(1)∗z ,K∗z]] + σ4

nW+ σ2

n(X W) + σ2n(W Z∗)

(13)

where the first two terms correspond to the received signal ofmultiple targets and noise, and the last two terms represent theundesirable cross products between the correlated signal andnoise components. Then the last two terms of (13) together arenamed as cross term. As for any 4-order tensor, the elements inthe cross-correlation tensor Rxz ∈ CQ×M×Q×M in (13) canbe uniquely found in the matrix X⊗Z∗. Therefore, by usingthe operator reshape(Rxz, [1, 3, 2, 4]), the noise term andcross term in the matrix form can be denoted by σ4

n(W⊗W)and σ2

n(X⊗W + W ⊗ Z∗), respectively.Accordingly, the other cross-correlation tensor Rzx =

EZ X∗ can be written as

Rzx = [[p2; A(1)∗x ,K∗x, A

(1)z ,Kz]] + σ4

nW+ σ2

n(Z W) + σ2n(W X∗).

(14)

Using both cross-correlation tensorsRxz andRzx, a higher-order tensor model that fully exploits the multi-dimensionalstructure shared by all subarrays in L-shaped nested array aswell as the joint target spatial information can be constructed.To demonstrate this, let us consider only the signal term atfirst. Each factor matrix in Rzx is a Vandermonde matrix.Thus, the conjugate symmetric property can be utilized. It isgiven by

JQA(1)∗x = A(1)

x Φx, JMK∗x = KxΠx

JQA(1)∗z = A(1)

z Φz, JMK∗z = KzΠz

(15)

where JQ and JM are the exchange matrices. The diagonalmatrices Φx, Φz , Πx and Πz are given by

Φx = diag(κ−Q−1x ), Πx = diag(κ1−M

x )

Φz = diag(κ−Q−1z ), Πz = diag(κ1−M

z )(16)

5

with κz , [e−jπ cosφ1 , · · · , e−jπ cosφK ]T ∈ CK .Inserting (15) into Rzx and concatenating both Rxz andRzx in a new dimension, the following 5-order tensor can beconstructed

R = [[p2; A(1)x ,Kx, A

(1)∗z ,K∗z,G]] (17)

where G ∈ C2×K represents the joint target spatial informa-tion in the 5-th dimension, given by

G =

[1 · · · 1

ejπ2S(cos θ1−cosφ1) · · · ejπ2S(cos θK−cosφK)

].

(18)Thus, by exploiting the conjugate symmetry property, the

effective array aperture is increased and the 5-th factor matrixG is built, which pairs the azimuth and elevation angles of alltargets naturally. The factor matrices of R contain the targets’2-D DOA information in two dimensions jointly or separately.However, if conventional tensor decomposition methods likealternating least square (ALS) algorithm or higher-order SVD(HOSVD) are directly utilized to conduct 2-D DOA estimation[34], [36], the computational complexity may be extremelyhigh. To reduce the computational complexity, we use tensorreshape operator to obtain a 3-order tensor.

Note that different reshapes are not equivalent fromthe parameter identifiability point of view [35]. We re-shape R into a new 3-order tensor such that the sys-tem DOF is maximized, and denote this reshape as T =reshape(R, [1, 3, 2, 4, 5]), or equivalently, as

T = [[p2; (A(1)x A(1)∗

z ), (Kx K∗z),G]] (19)

where A(1)x A

(1)∗z , Kx K∗z and G are the first, second

and third factor matrices of T , respectively. In matrix form,the received signal can be expressed by the tensor unfolding,i.e, T(2) = unfolding(T , [1, 3, 2]), or equivalently, by

T(2) = [(A(1)x A(1)∗

z )G]R2s(Kx K∗z)

T . (20)

Then the 2-D DOA estimation problem for L-shaped nestedarray consists of finding (θk, φk)Kk=1 from the observationof T .

B. Parameter Identifiability

Based on T , it can be shown that the parameter identi-fiability for our model is related to the tensor rank, whoseupper bound is restricted by the uniqueness condition oftensor decomposition. For tensors with arbitrary factor matrix,conventional ALS algorithm can be used and the uniquenesscondition is determined by the sum of the Kruskal ranks of allfactor matrices [34]. For tensors with structured factor matrixlike Vandermonde matrix, a computationally efficient tensordecomposition method and a better uniqueness condition havebeen discussed [37], [38]. In our case, the uniqueness condi-tion can be given by

min(2(Q2 − 1),M2) ≥ K. (21)

To further explain the essence of (21) and to demonstratethe distinct tensor reshape (19), rewrite T(2) as

T(2) =

K∑k=1

σ4k[(a

(1)x (θk)a(1)∗

z (φk))gk](kx(θk)k∗z(φk))T

(22)where gk is the k-th column of the factor matrix G. Theexpression a

(1)x (θk) a

(1)∗z (φk) gk in (22) can be regarded

as a steering vector in direction (θk, φk), which correspondsto a virtual co-array that consists of two centrally symmetricURAs. The structure of each URA merely depends on themanifold of a

(1)x (θ) a

(1)∗z (φ), since G is generated by

exploiting the conjugate symmetry property. Consequently,the maximum number of targets that can be resolved by thevirtual co-array is 2(Q2 − 1) if the number of snapshots islarge enough. However, the expression kx(θk) k∗z(φk) in(22) implies that the number of snapshots is M2 and it iscompariable with Q2. In this case, the maximum numberof targets that can be resolved by the virtual co-array ismin(2(Q2 − 1),M2), which is identical to (21).

It is also worth noting that the aperture of the virtualco-array rises with the increase of Q while the number ofefficient snapshots declines. It is typically determined in theconventional nested array that the selection of Q = M = Sis optimal in terms of the trade-off between robustness andspatial resolution. In our tensor model, however, the followingoptimization problem is built to maximize the system DOF

maxQ

min(2(Q2 − 1),M2)

s.t. Q+M = 2S(23)

whose optimal solution and optimal value are Q =√8S2 + 2− 2S and 24S2 − 8S

√8S2 + 2 + 2, respectively.

Using this result, approximately 1.37S2 + 2 targets can beresolved with only 2N − 1 physical elements based on theabove designed tensor model. It is superior to the conventionalapproaches that treat the received signals for different subar-rays in co-array domain separately. Moreover, the exploitationof the conjugate symmetry property increases the effectivearray aperture and enables a natural pairing of the azimuthand elevation angles for all targets. The factor matrices thatcontain the target 2-D spatial information can be utilized toconduct 2-D DOA estimation for L-shaped nested array.

IV. PROPOSED TWO-STEP 2-D DOA ESTIMATIONMETHOD FOR L-SHAPED NESTED ARRAY

In the previous section, a higher-order tensor signal modelhas been constructed and a special type of tensor reshape hasbeen utilized to improve the parameter identifiability of thecontracted tensor model. Note that the first two factor matricesof T are the KR product of two Vandermonde matrices, whosevectors of generators contain the target angular information.We can thus exploit the shift-invariance between differentsubarrays in two axes to conduct 2-D DOA estimation.

However, in the co-array domain, the signal and noise termsbecome correlated, and the cross term between them cannotbe ignored (see, for example, [28]). Using the same operationsas we did for deriving (19) and (20), the cross term between

6

signal and spatially correlated noise as well as the noise termcan be also expressed in the matrix form as

H = σ2n

[D⊗WW ⊗D∗

], N = σ4

n

[W ⊗WW ⊗W

](24)

where D , X + Z∗. Hence, the unfolding of the designedtensor that contains the signal term, noise term and cross termcomponents can be written as

T(2) = T(2) + H + N. (25)

Compared with the conventional technique based on av-eraging the signal covariance matrices of all subarrays, thespatially correlated cross term H degrades the DOA estimationperformance. Note that both H and N are sparse matrices,i.e., most of their elements are zeros. To resolve the afore-mentioned problem of performance degradation caused by thecross term, an iterative DOA estimation method is proposednext. The main idea of the method is to modify the receivedsignal at every next step of the estimation procedure based onthe DOA estimation results obtained in the previous step [39].Thus, the DOA estimation performance can be improved byestimating and removing the cross term in the received signaliteratively.

A. Step 1: DOA Estimation via Tensor DecompositionGiven the received signal matrices X and Z, the 3-order

tensor T can be constructed, whose factor matrices are Vander-monde matrices in the noise-less case. Since we assume thatall targets are spatially distinct, both A

(1)x A

(1)∗z and KxK∗z

are column full rank. Hence, a computationally efficient tensordecomposition method can be designed [37], [38].

Specifically, consider the matrix in (20). Denote the trun-cated SVD (tSVD)1 of this matrix as T(2) = UΛVH ,where U ∈ C2Q2×K , Λ ∈ CK×K , and V ∈ CM2×K . SinceA

(1)x A

(1)∗z and Kx K∗z are column full rank, for a

nonsingular matrix Ξ ∈ CK×K , it can be found that

UΞ = (A(1)x A(1)∗

z )G. (26)

Considering the KR product, the following relationshipshold

Ux1Ξ = (A(1)x1 A(1)∗

z )G

Ux2Ξ = (A(1)x2 A(1)∗

z )G

Uz1Ξ = (A(1)x A

(1)∗z1 )G

Uz2Ξ = (A(1)x A

(1)∗z2 )G

(27)

where A(1)x2 and A

(1)z2 denote the submatrices of A

(1)x and

A(1)z without the first row, respectively, A

(1)x1 and A

(1)z1 denote

the submatrices of A(1)x and A

(1)z without the last row,

respectively, Ux1, Ux2, Uz1 and Uz2 are the submatrices ofthe left singular matrix U, given by

Ux1 = [I2Q(Q−1),02Q(Q−1)×2Q]U

Ux2 = [02Q(Q−1)×2Q, I2Q(Q−1)]U

Uz1 = (IQ ⊗ [I2(Q−1),02(Q−1)×2])U

Uz2 = (IQ ⊗ [02(Q−1)×2, I2(Q−1)])U.

(28)

1The tSVD returns the dominant singular vectors and the associated singularvalues of a matrix.

Using the shift-invariance property of Vandermonde matrix,that is, A

(1)x2 = A

(1)x1 Ω

(2)x , A

(1)z2 = A

(1)z1 Ω

(2)z , we can write

Ux2Ξ = Ux1ΞΩ(2)x , Uz2Ξ = Uz1ΞΩ(2)∗

z (29)

or equivalently,

U†x1Ux2 = ΞΩ(2)x Ξ−1, U†z1Uz2 = ΞΩ(2)∗

z Ξ−1. (30)

Since Ω(2)x and Ω

(2)∗z are diagonal matrices generated by κx

and κ∗z , the eigenvalues of the matrices U†x1Ux2 and U†z1Uz2

can be regarded as the estimations of the diagonal elementsof Ω

(2)x and Ω

(2)∗z , respectively. After estimating κx and κz ,

the azimuth and elevation angles can be computed by

θk = arccos (j ln κx,k/π) , φk = arccos (j ln κz,k/π) (31)

where κx,k and κz,k are the k-th elements of the correspondingvectors. The aforementioned estimates are independent, whichmeans that the pair-matching of elevation and azimuth anglesin two dimensions is required.

Let us consider the factor matrix G, which contains the jointtarget elevation and azimuth information that can be used topair (θk, φk). Since G is also a Vandermonde matrix, we cangeneralize (27) to obtain the following relationship

Ug1Ξ = A(1)x A(1)

z , Ug2Ξ = A(1)x A(1)

z γ (32)

where γ is the second row of G, Ug1 and Ug2 are submatricesof U that consist of odd rows and even rows, respectively.Denote Γ = diag(γT ), we have

Ug2Ξ = Ug1ΞΓ⇔ U†g1Ug2 = ΞΓΞ−1. (33)

Similarly, the collection of eigenvalues of U†g1Ug2 canbe regarded as the estimation of γ. By exploiting γk, theconnection between cosβk , cos θk − cosφk and the pair(cos θk, cosφk) is established. The method of [12] can be thenmodified to pair the azimuth and elevation estimates (31).

Instead of matching the angles θkKk=1, φkKk=1 andβkKk=1 directly, we minimize the errors among K pairs ofeigenvalues computed by (30) and (33) to match the elevationand azimuth angles for each target, since the pairing orderis reserved in the collections of eigenvalues. In other words,there must exist two sets ik and jk, k = 1, 2, · · · ,K, suchthat

γk = (κx,ik κz,jk)−2S (34)

where ik and jk are the permutations of set 1, 2, · · · ,K.The pairing process is then conducted by repeatedly solving

the following minimization problem

ik, jk = mini,j

f(i, j|k) (35)

where f(i, j|k) ,∣∣∣∣∣∣γk − (κx,iκy,j)

−2S∣∣∣∣∣∣2.

This eigenvalue-based pair-matching approach avoids theangle ambiguity problem, since the exponent term 2S in γmay cause grating lobes problem for βkKk=1. Note alsothat some targets may have identical θk or φk angle2 andthe elements in κx, κy can be repeated while the elements

2This is a challenging scenario in the 2-D DOA estimation problem, sincethere could be harmonics with the same value in one dimension.

7

in γ must be distinct. To fulfill the one-to-one mapping, ifik, jk = min f(i, j|k), then for the other K − 1 targets,im, jm(m 6= k) should be chosen from the elements in thevectors κx, κy after removing κx,ik and κy,jk to avoid theoverlap of ik, jk.

Finally, the pair (θk, φk) for the k-th target can be given by

θk = θik , φk = φjk . (36)

B. Step 2: Cross Term Estimation and Elimination

In Step 2, the DOA estimation results from the previousstep(36) can be used to construct a scaled version of theundesirable cross term H. The renewed tensor T <l> afterremoving the estimated H can be then used as an input forStep 1 again to obtain the target DOA estimates with a lowererror. First, let us build the steering matrices of two differenceco-arrays, i.e.,

A(1),<l>x = [ax(θi1), ax(θi2), · · · , ax(θiK )]

A(1),<l>z = [az(φj1), az(φj2), · · · , az(φjK )]

(37)

where < l > denotes the current iteration.In the following, we drop the superscript < l > for

notational simplicity. The vector of the source powers can beestimated by minimizing the differences between the observa-tions in (7) and their estimates, which can be given as

p = argminp||yxx − A(1)

x p||2. (38)

Using least-square (LS) method, the solution of (38) is givenby

p =(A(1)Hx A(1)

x

)−1

A(1)x yxx. (39)

Then, the received signal of all subarrays in two axes canbe estimated as

X(1) = A(1)x RsK

(1)Tx , Z(1) = A(1)

z RsK(1)Tz (40)

where Rs = diag(p) and the reconstructed matrices K(1)x and

K(1)z for Step 2 are given by

K(1)x = [κ0

x,κ1x, · · · ,κM−1

x ]T

K(1)z = [κ0

z,κ1z, · · · ,κM−1

z ]T(41)

with κx = [e−jπ cos θi1 , · · · , e−jπ cos θiK ]T ∈ CK and κz =

[e−jπ cos φj1 , · · · , e−jπ cos φjK ]T ∈ CK . Therefore, the crossterm between the signal and the spatially correlated noise isobtained to be

H = σ2n

[D⊗W

W ⊗ D∗

](42)

where D = X(1) + Z(1)∗.Inserting (42) to (25), the updated received signal in Step 2

in the matrix form is given by

T(2) ← T(2) − µH (43)

where µ is a real number between zero and one, that is,a reliability factor to the estimates in Step 1. Once µ isdetermined, the modified received signal T(2) can be updatedand the DOA estimation with smaller error can be conductedvia the tensor decomposition approach used in Step 1. These

Algorithm 1: Proposed Iterative 2-D DOA EstimationMethod for L-shaped Nested ArrayInput: Observations of x(t) and z(t)Output: (θk, φk)k=K

k=1

1 Rxx,Rzz ← Ex(t)xH(t), Ez(t)zH(t);2 yxx, yzz ← vec(Rxx), vec(Rzz);3 Spatial smoothing, M is determined by (23);4 X,Z← [y

(1)xx , y

(2)xx , · · · , y(M)

xx ], [y(1)zz , y

(2)zz , · · · , y(M)

zz ];5 Rxz,Rzx ← EX Z∗, EZ X∗;6 Concatenate Rxz,Rzx to form the 5-th order tensor R;7 T ← reshape(R, [1, 3, 2, 4, 5]);8 T(2) ← unfolding(T , [1, 3, 2]);9 ε(0) =

∣∣∣∣T(2)

∣∣∣∣2F, δ = 10−5;

10 while ||ε(l) − ε(l−1)|| ≤ δ and l ≤ L do11 Step 1 begin12 (U,Λ,V)← tSVD(T<l−1>

(2) );13 Ux1,Ux2,Uz1,Uz2,Ug1,Ug2 ←(28) and

(32);14 κx,κ

∗z,γ ← Compute the EVD of the three

matrices U†x1Ux2, U†z1Uz2 and U†g1Ug2;15 for k = 1, 2, · · · ,K do16 ik, jk ← mini,j f(i, j|k), where

f(i, j|k) ,∣∣∣∣∣∣γk − (κx,iκy,j)

−2S∣∣∣∣∣∣2;

17 (θk, φk)← (31) and (36);18 end19 end20 Step 2 begin21 A<l>

x , A<l>z ,K<l>

x ,K<l>x ← (37) and (41);

22 p<l> ← (39);23 X<l>, Z<l> ← (40);24 H<l> ← (42);25 T<l>

(2) ← (43);26 ε(l) ← ||T(2) −T<l>

(2) ||2F ;

27 l← l + 1;28 end29 end

two steps can be repeated consequently several times until theconvergence or until the desired estimation error is achieved.

The scaling factor µ represents the reliability of the esti-mates H, i.e., if µ takes a value close to one, we believethat the estimation error of the cross term is negligible, whilea small value of µ implies that the estimates are erroneous.If µ = 1, it means that the cross term can be preciselyestimated and removed. In practice, however, estimation errorsare unavoidable. One can find a practical method based on themaximum likelihood (ML) criterion to determine the optimalvalue of µ [39].

An outline of the proposed iterative DOA estimation methodfor L-shaped nested array is summarized in Algorithm 1.

C. Computational Complexity

We analyze here the computational complexity of the pro-posed iterative 2-D DOA estimation method. The initial inputs

8

of the proposed method are the designed tensor T and its ma-trix unfolding T(2). The complexity of obtaining these inputsis O(2N2Ts+2Q2M2), where Ts is the number of snapshots.In Step 1, the proposed method mainly contains three parts,i.e., the tSVD of T(2), the EVD of three matrices and the pair-matching procedure. If (23) is satisfied, then T(2) ∈ C2Q2×M2

is a tall matrix, and the complexity of tSVD in this case isO(2Q2M4). While computing κx or κz , the number of flopsrequired is O(8Q3(Q−1)K+4Q(Q−1)K2+K3). Similarly,the complexity of estimating γ is O(4Q3K +2Q2K2 +K3).The pair-matching requires at most O(K(K − 1)/2) flops.In Step 2, the construction of the estimated cross term Hneeds O(2S(S − 1)K + 2Q2M2 + 2QK(M + K)) flops.Then, the computations in Step 1 are conducted again with theupdated inputs. For simplicity, let us assume that Q ≈M ≈ Sand consider only one iteration3 of cross term mitigation, i.e.,L = 1. Then, the number of flops required is approximatelyO(2N2Ts+4S4(S2+1)+2SK(2S−1)(4S+3K+1)+K(K−1)). It can be observed that the tSVD operator consumes themost of the computational complexity.

D. CRB for the Proposed Tensor ModelIt is also worth deriving analytical expression for the CRB

for the proposed tensor model to see that the improvementcomes from the proposed received signal model and also checkwhether the proposed algorithm achieves the statistical bound.

The tensor model (17) is used to conduct the 2-D DOA esti-mation. The received signal spatial covariance matrix withoutthe cross term can be written as

R =

K∑k=1

σ4kcx(θk)c

Hz (φk) (44)

where cx(θk) , a(1)x (θk) ⊗ kx(θk) ⊗ gx(θk), cz(φk) ,

a(1)z (φk) ⊗ kz(φk) ⊗ gz(φk), gx(θk) = [1, ejπ2S cos θk ]T and

gz(φk) = [1, ejπ2S cosφk ]T . Vectorizing (44), the receivedsignal vector can be written as

r =

K∑k=1

σ4kcx(θk)⊗ c∗z(φk). (45)

Let us collect all unknown but deterministic entities to a3K × 1 vector

ψ , [θ1, · · · , θK , φ1, · · · , φK , σ21 , · · · , σ2

K ]T . (46)

Using the Slepian-Bangs (SB) formula [40], the (i, j)-thentry of the Fisher information matrix (FIM) can be found as

[J(ψ)]i,j = Tstr

(R−1 ∂R

∂[ψ]iR−1 ∂R

∂[ψ]j

)

= Ts

(∂r

∂ψ

)H (RT ⊗R

)( ∂r

∂ψ

) (47)

where∂r

∂ψ=

[∂r

∂θ1, · · · , ∂r

∂θK,∂r

∂φ1, · · · , ∂r

∂φK,∂r

∂σ21

, · · · , ∂r

∂σ2K

].

(48)

3The proposed method can remove the most of the cross term based on theinitial estimates. This will be verified by the simulation examples in Section V.

To compute the derivatives (48), we only need to considertwo submatrices sequentially, i.e.,

∂r

∂ψ=

(C′x + Cz)Rs, (C′z + Cx)Rs︸ ︷︷ ︸

C′

,CxzRs︸ ︷︷ ︸C

(49)

where Cxz = 2(CxC∗z), Cx , [cx(θ1), · · · , cx(θK)], Cz ,[cz(φ1), · · · , cz(φK)], and

C′x∆=

[∂cx(θ1)

∂θ1, · · · , ∂cx(θK)

∂θK

]C′z

∆=

[∂cz(φ1)

∂φ1, · · · , ∂cz(φK)

∂φK

].

(50)

Then, the FIM can be obtained as

J =

[JH1 J1 JH1 J2

JH2 J1 JH2 J2

](51)

where J1 =(RT ⊗R

)−1/2C′ and J2 =

(RT ⊗R

)−1/2C.

Considering the inverse of a 2× 2 block matrix, the CRB canbe derived as

CRB(ψ) =1

Ts

(JH1 Π⊥J2

J1

)−1(52)

where Π⊥J2= I4Q2M2 − J2(J

H2 J2)

−1JH2 .

V. SIMULATION RESULTS

In this section, several simulation examples are presentedin order to evaluate the performance of the proposed iterative2-D DOA estimation method. Throughout the simulations, anL-shaped nested array that consists of two nested subarrayswith N = 6 elements along x-axis and z-axis is considered.For each nested subarray, the inner ULA consists of N/2 =6/2 = 3 elements with spacing d = λ/2, while the outerULA consists of the other three elements with spacing 2λ.Note that the element at the origin is used twice. The sourcesare modeled as random Gaussian processes and the noise isassumed to be spatially and temporally white. Consequently,the nested array forms a difference co-array with DOF 2S −1 = 2 ·12−1 = 23. The optimal number of elements for eachsubarray is Q ≈ 10, which is different from the conventionalnested array (Q = M = S = 12) [27]. In our examples, weassume that K = 3 sources are impinging on the L-shapednested array from distinct directions θk = [10, 20, 30] andφk = [45, 40, 35], and Ts = 512. The number of MonteCarlo trials is P = 500, while the scaling (reliability) factorin the proposed algorithm is µ = 0.9. The designed tensormodel (17) is used. The SNR is computed as

SNR [dB] = 20 log

∥∥∥T(2)

∥∥∥F

‖τN‖F. (53)

For comparison, the conventional ALS algorithm [34] andthe DOA estimation method of [27] are also used. Note thatthe proposed eigenvalue-based pair-matching approach is usedfor all methods except for the ALS-based method.

9

5 10 15 20 25 30 35Azimuth[Deg]

34

36

38

40

42

44

46

48E

leva

tio

n[D

eg]

TrueDirectionl=1l=2l=3

Fig. 2. DOA estimation results, three targets, L = 3, SNR = -3 dB.

0 2 4 6 8 10 12 14 16 18 20Number of Iterations

10-4

10-3

10-2

10-1

100

101

102

RM

SE

[Deg

]

Azimuth(SNR=-4dB)Azimuth(SNR=0dB)Azimuth(SNR=4dB)Azimuth(SNR=6dB)Elevation(SNR=-4dB)Elevation(SNR=0dB)Elevation(SNR=4dB)Elevation(SNR=6dB)

Fig. 3. RMSEs versus number of iteration, three targets, 500 trials.

A. Example 1: Effect of the Proposed Two-Step 2-D DOAEstimation Method

In our first example, we aim at studying how many iterationsare required for the proposed DOA estimation algorithm.The DOA estimates during each iteration are also shown toillustrate the cross term mitigation process and the validity ofthe proposed method.

In Fig. 2, the DOA estimates for three targets obtained bythe proposed algorithm with L = 3 iterations are shown. TheSNRs are -3 dB. When l = 1, the DOA estimation is based onthe initial received signal matrix T(2) where cross term H ispresent. Note that DOAs of all three targets are not correctlyestimated. However, using the proposed iterative estimationmethod in Algorithm 1, it is possible to gradually eliminatethe cross term. It can be observed in the second iteration (l =2) that the DOA estimates are more accurate after a propermitigation of the cross term, which in turn provides a betterdescription of H for the final iteration (l = 3). Hence, all threetargets are resolved successfully in the third iteration.

Then we also evaluate the root mean square error (RMSE)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 110-3

10-2

10-1

100

101

102

RM

SE

[Deg

]

Azimuth(SNR=-6dB)Azimuth(SNR=-3dB)Azimuth(SNR=0dB)Azimuth(SNR=3dB)Elevation(SNR=-6dB)Elevation(SNR=-3dB)Elevation(SNR=0dB)Elevation(SNR=3dB)

Fig. 4. RMSEs versus µ, three targets, 500 trials.

versus the number of iterations for several values of SNR. Letus set the maximum number of iterations to L = 20. Theazimuth RMSE is computed by

RMSE =

√√√√ 1

2PK

K∑k=1

P∑p=1

(θk(p)− θk(p)

)2

(54)

while the elevation RMSE is obtained similar by replacingθk with φk. It can be seen in Fig. 3 that the proposedmethod converges after several iterations for both azimuthand elevation estimates. It is worth noting that the number ofiterations required by the proposed method increases graduallywith the decrease of the SNR, since the estimation of His more reliable at high SNR. The number of iterations Lrequired for convergence is no more than 4 when the SNR isabove 0 dB. In some cases, only one iteration is sufficient.

Note also that if SNR is too low, the first DOA estimationresult obtained in Step 1 of the proposed algorithm barely con-tains any target spatial information. Under this circumstance,some other DOA estimation methods that perform well at lowSNR can be used to initialize the proposed iterative algorithm.

B. Example 2: The Optimal Value of µ

In this example, we evaluate the optimal value of thereliability factor µ and show the RMSE performance of theproposed algorithm versus different values of µ. Four caseswith different SNRs (-6 dB, -3 dB, 0 dB and 3 dB) areconsidered. Although it is unnecessary when SNR is largeenough, the maximum number of iterations is set as L = 20 foreach case. The reliability factor µ varies from zero to one witha fixed step size 0.05, while other parameters are unchangedas compared to the previous example.

It can be seen in Fig. 4 that the elevation and azimuthRMSEs are quite poor when SNR is -6 dB. This is becausethe signal component is interfered by the noise, and thereconstructed cross term H in Step 2 of the proposed iterativealgorithm contains no target information but noise. Thus, theaccuracy of the proposed iterative 2-D DOA estimation method

10

is limited by the accuracy of the initial Step 1 when SNR islow, and as mentioned above, some other DOA estimationmethod that perform well at low SNR should be used toinitialize the proposed iterative algorithm. From the other threecases, it can be seen that the final RMSEs for both elevationand azimuth estimates decline steadily when µ raises fromzero to 0.9. A turning point can be observed when µ is nearby0.9. Thus, we can set µ = 0.9 as the suboptimal value duringour simulations. Although deriving the close-form expressionfor computing the optimal reliability factor is not feasible, weuse this example to demonstrate that µ can be determined inpractice, and the algorithm is not very sensitive if a suboptimalvalue is selected. Indeed, the proposed method is valid witha suboptimal µ. The only possible price is that a suboptimalscaling (reliability) factor may degrade the convergence speed,which means that more iterations may be required.

C. Example 3: The Validity of the Enhanced DOF via TensorModeling

This example is used to verify the advantage of thedesigned tensor model in terms of the enhanced DOF.Instead of three targets, we assume that totally K =36 targets are impinging on the L-shaped nested arrayfrom distinct directions determined by a 6 × 6 grid gen-erated by θ = [10, 20, 30, 40, 50, 60] and φ =[5, 15, 25, 35, 45, 55]. The powers of all sources areidentical, i.e., let the SNRs be 13 dB. The maximum numberof iterations of the proposed algorithm is set as L = 20 andµ = 0.9. Note that the DOF of the conventional L-shapednested array model for our simulation setup is 2S − 1 =2 · 12 − 1 = 23. Thus, the method of [27] cannot resolveK = 36 targets simultaneously.

It can be observed in Fig. 5 that all target DOAs areestimated and paired by our proposed method correctly. There-fore, the proposed tensor model can be used to resolve moretargets than the conventional L-shaped nested array signalmodel. Note that some estimates fall out of the grid, whichis caused by the cross term residue in Step 2. With theincrease of the number of targets, the precise estimation of Hbecomes more difficult as well as the determination of a properreliability factor µ. Under this circumstance, the cross termis unavoidable and the DOA estimation accuracy is thereforedegraded slightly. Fig. 5 also proves that the pair-matchingmethod proposed in this paper is effective even in some verychallenging scenario for 2-D DOA estimation, where sometargets share identical azimuth or elevation angles.

D. Example 4: RMSE Performance versus SNR

Our fourth example aims to illustrate the DOA estimationperformance of the proposed iterative algorithm in terms ofRMSE. Three targets are placed at θk = [15, 25, 35] andφk = [50, 40, 30]. To ensure the validity of the proposedalgorithm, the maximum number of iterations is set as L = 20and µ = 0.9. The conventional nested array signal model isused to conduct 2-D DOA estimation via the SS technique,wherein an extra pair-matching procedure is demanded. TheALS algorithm based on the tensor model (19) is also used. In

0 10 20 30 40 50 60Azimuth[Deg]

10

20

30

40

50

60

70

Ele

vati

on

[Deg

]

TrueDirectionEstimates

Fig. 5. Resolution of K = 36 targets, SNR = 13 dB, 500 trials.

the proposed algorithm, the initial DOA estimation result be-fore mitigating the cross term is named as Elevation/Azimuth-Step 1 while the final DOA estimation result after eliminatingthe cross term is referred to as Elevation/Azimuth-Step 2.

The elevation and azimuth RMSEs of the algorithms testedare shown in Fig. 6. The SS-based DOA estimation canentirely eliminate the cross term and the signal covariancematrix is positive semidefinite for any finite number of snap-shots. However, this method averages the signal covariancematrices of all subarrays in the co-array domain on both x-axis and z-axis. Thus, the SS-based DOA estimation methodignores the multi-dimensional structure between those sub-arrays. Therefore, the corresponding RMSEs are relativelypoor (high). Moreover, the DOF in this case is only 2S − 1since the azimuth and elevation estimations are conductedseparately. The DOA estimates in Step 1 of the proposedalgorithm are also relatively poor because the cross termdegrades the performance significantly. It can be shown thatthe ALS algorithm that uses the same tensor model slightlyimproves the RMSEs as compared to the results of Step 1 ofthe proposed algorithm. It is because the ALS algorithm dealswith the designed higher-order tensor directly. The cross termin tensor form is a sparse tensor, which has less influenceon the estimation results obtained by tensor decomposition.However, the use of ALS algorithm demands much morecomputation resources, and the method is unstable especiallywhen the number of targets is unknown. The proposed methodafter cross term mitigation surpasses the other methods withthe lowest RMSE threshold. It is because it exploits the multi-dimensional structure of the received signal for all subarraysand removes the cross term efficiently. Indeed, a computation-ally efficient tensor decomposition method is used to conduct2-D DOA estimation, while the proposed iterative method isused to remove the cross term.

E. Example 5: Probability of Resolution versus SNR

Finally, we evaluate the methods tested in terms of theprobability of resolution for two closely spaced targets. We

11

-15 -10 -5 0 5 10 15SNR[dB]

10-4

10-3

10-2

10-1

100

101

102

RM

SE

[Deg

]

Azimuth-Step 1Azimuth-SSAzimuth-ALSAzimuth-Step 2Azimuth-CRBElevation-Step 1Elevation-SSElevation-ALSElevation-Step 2Elevation-CRB

Fig. 6. RMSE versus SNR, three targets, 500 trials.

assume only two sources in this example at θk = [15, 16]and φk = [30, 31]. The other parameters are unchangedas compared to previous example. These two targets areconsidered to be resolved if∥∥∥θk − θk∥∥∥ ≤ ‖θ1 − θ2‖ /2∥∥∥φk − φk∥∥∥ ≤ ‖φ1 − φ2‖ /2, k = 1, 2

(55)

holds true.It can be seen in Fig. 7 that all methods tested achieve

perfect resolution at high SNR. For each method, the elevationresolution threshold is substantially smaller than its counter-part in azimuth, which is reasonable since the RMSEs ofelevation estimation are always better than those of azimuth es-timation as shown in previous examples. The proposed methodenables the lowest threshold for both elevation and azimuthresolution. Note that it is possible for the ALS algorithmto use the designed tensor model in order to resolve twoclosely spaced targets at extremely low SNR. The threshold,however, is worse than that of the proposed method due tothe destructive influence of the cross term. Indeed, the crossterm is masked by the noise when SNR is low, and the ALS-based algorithm can capture only the structure of the signalterm, while the structure of the cross term is different andcannot be captured by the ALS-based method. In fact, thecross term for the ALS-based method works as an additionalnoise component, and the cross term degrades the performanceof the ALS-based method even when SNR is high. With theincrease of SNR, the DOA estimation accuracy improvementprovided by the ALS algorithm degrades since the cross termcannot be ignored. This is the main reason behind the fact thatthe curves (for azimuth and elevation) for the ALS algorithmin Fig. 7 are relatively flat compared to the curves for theother methods tested. Finally, note that the proposed iterative2-D DOA estimation method can effectively mitigate the crossterm and, hence, achieves a better resolution performance.

-15 -10 -5 0 5 10 15 20 25 30SNR[dB]

0

0.2

0.4

0.6

0.8

1

Elevation-Step 1Elevation-SSElevation-ALSElevation-Step 2

-15 -10 -5 0 5 10 15 20 25 30SNR[dB]

0

0.2

0.4

0.6

0.8

1

Pro

bab

ility

of

Res

olu

tio

n

Azimuth-Step 1Azimuth-SSAzimuth-ALSAzimuth-Step 2

Fig. 7. Probability of resolution versus SNR, two targets, 500 trials.

VI. CONCLUSION

An iterative 2-D DOA estimation method via tensor mod-eling has been proposed for L-shaped nested array. In theproposed method, a higher-order tensor has been designed toexploit the multi-dimensional structure of the received signalfor all subarrays in co-array domain. The designed tensormodel significantly improves the system DOF by optimizingthe number of subarrays for SS technique. A computationallyefficient tensor decomposition method has been then devel-oped to decompose the Vandermonde factor matrices, whosevectors of generators provide the target spatial information.An eigenvalue-based approach has been proposed to pairthe azimuth and elevation angles, which is effective even insome challenging 2-D DOA estimation scenarios. The crossterm caused by the correlated signal and noise componentsof the received signal in co-array domain is estimated andremoved in the second step of our methods based on theDOA estimates obtained at the first step, and then stepsare repeated iteratively to achieve a better DOA estimationperformance. Therefore, the received signal can be modifiedgradually during iterations. Comparing with existed DOAestimation methods for L-shaped nested array, the proposedmethod can take advantage of the multi-dimensional structureof the received signal, it is also capable of mitigating thecross term. The parameter identifiability of the designed tensormodel has been significantly improved. Simulation results haveverified that the proposed method achieves a better accuracyand higher resolution in the problem of 2-D DOA estimationfor L-shaped nested array as compared to existed techniques.

REFERENCES

[1] A. Hassanien and S. A. Vorobyov, “Transmit energy focusing for DOAestimation in MIMO radar with colocated antennas,” IEEE Trans. SignalProcess., vol. 59, no. 6, pp. 2669–2682, Jun. 2011.

[2] H. Van Trees, Detection, estimation, modulation theory: Part I: de-tection, estimation, and linear modulation theory. New York: WileyIntersci, 2004.

[3] H. Van Trees, Optimum array processing: Part IV of detection, estima-tion, and modulation theory. New York: Wiley Intersci, 2004.

12

[4] A. L. Swindlehurst and T. Kailath, “Azimuth/elevation direction findingusing regular array geometries,” IEEE Trans. Aerosp. Electron. Syst.,vol. 29, no. 1, pp. 145–156, Jan. 1993.

[5] R. Schmidt, “Multiple emitter location and signal parameter estimation,”IEEE Trans. Antennas Propag., vol. 34, no. 3, pp. 276–280, Mar. 1986.

[6] R. Roy and T. Kailath, “ESPRIT-estimation of signal parameters viarotational invariance techniques,” IEEE Trans. Acoust., Speech, SignalProcess., vol. 37, no. 7, pp. 984–995, Jul. 1989.

[7] M. D. Zoltowski, M. Haardt, and C. P. Mathews, “Closed-form 2-Dangle estimation with rectangular arrays in element space or beamspacevia unitary ESPRIT,” IEEE Trans. Signal Process., vol. 44, no. 2, pp.316–328, Feb. 1996.

[8] Y. Hua, T. K. Sarkar, and D. D. Weiner, “An L-shaped array for esti-mating 2-D directions of wave arrival,” IEEE Trans. Antennas Propag.,vol. 39, no. 2, pp. 143–146, Feb. 1991.

[9] N. Tayem and H. M. Kwon, “L-shape 2-dimensional arrival angleestimation with propagator method,” IEEE Trans. Antennas Propag.,vol. 53, no. 5, pp. 1622–1630, May 2005.

[10] G. Wang, J. Xin, N. Zheng, and A. Sano, “Computationally efficientsubspace-based method for two-dimensional direction estimation withL-shaped array,” IEEE Trans. Signal Process., vol. 59, no. 7, pp. 3197–3212, Jul. 2011.

[11] J. Liang, X. Zeng, W. Wang, and H. Chen, “L-shaped array-basedelevation and azimuth direction finding in the presence of mutualcoupling,” Signal Process., vol. 91, no. 5, pp. 1319 – 1328, May 2011.

[12] S. Kikuchi, H. Tsuji, and A. Sano, “Pair-matching method for estimating2-D angle of arrival with a cross-correlation matrix,” IEEE AntennasWireless Propag. Lett., vol. 5, pp. 35–40, Mar. 2006.

[13] Y. Wei and X. Guo, “Pair-matching method by signal covariancematrices for 2D-DOA estimation,” IEEE Antennas Wireless Propag.Lett., vol. 13, pp. 1199–1202, Jun. 2014.

[14] J. Gu and P. Wei, “Joint SVD of two cross-correlation matrices to achieveautomatic pairing in 2-D angle estimation problems,” IEEE AntennasWireless Propag. Lett., vol. 6, pp. 553–556, Nov. 2007.

[15] J. Gu, W. Zhu, and M. N. S. Swamy, “Joint 2-D DOA estimation viasparse L-shaped array,” IEEE Trans. Signal Process., vol. 63, no. 5, pp.1171–1182, Mar. 2015.

[16] S. O. Al-Jazzar, D. McLernon, and M. A. Smadi, “SVD-based jointazimuth/elevation estimation with automatic pairing,” Signal Process.,vol. 90, no. 5, pp. 1669 – 1675, May 2010.

[17] J. Liang and D. Liu, “Joint elevation and azimuth direction finding usingL-shaped array,” IEEE Trans. Antennas Propag., vol. 58, no. 6, pp. 2136–2141, Jun. 2010.

[18] N. Xi and L. Liping, “A computationally efficient subspace algorithmfor 2-D DOA estimation with L-shaped array,” IEEE Signal Process.Lett., vol. 21, no. 8, pp. 971–974, Aug. 2014.

[19] N. Tayem, K. Majeed, and A. A. Hussain, “Two-dimensional DOAestimation using cross-correlation matrix with L-shaped array,” IEEEAntennas Wireless Propag. Lett., vol. 15, pp. 1077–1080, Dec. 2016.

[20] Y. Dong, C. Dong, J. Xu, and G. Zhao, “Computationally efficient 2-D DOA estimation for L-shaped array with automatic pairing,” IEEEAntennas Wireless Propag. Lett., vol. 15, pp. 1669–1672, Jan. 2016.

[21] Y. Dong, C. Dong, W. Liu, H. Chen, and G. Zhao, “2-D DOA esti-mation for L-shaped array with array aperture and snapshots extensiontechniques,” IEEE Signal Process. Lett., vol. 24, no. 4, pp. 495–499,Apr. 2017.

[27] P. Pal and P. P. Vaidyanathan, “Nested arrays: A novel approach toarray processing with enhanced degrees of freedom,” IEEE Trans. SignalProcess., vol. 58, no. 8, pp. 4167–4181, Aug. 2010.

[22] R. Wu and Z. Zhang, “Convex optimization-based 2-D DOA estimationwith enhanced virtual aperture and virtual snapshots extension for L-shaped array,” IEEE Trans. Veh. Technol., vol. 69, no. 6, pp. 6473–6484,Jun. 2020.

[23] W. Rao, D. Li, and J. Q. Zhang, “A tensor-based approach to L-shapedarrays processing with enhanced degrees of freedom,” IEEE SignalProcess. Lett., vol. 25, no. 2, pp. 1–5, Feb. 2018.

[24] M. Sørensen, I. Domanov, and L. De Lathauwer, “Coupled canonicalpolyadic decompositions and multiple shift invariance in array process-ing,” IEEE Trans. Signal Process., vol. 66, no. 14, pp. 3665–3680, Jul.2018.

[25] A. Cichocki, D. Mandic, L. De Lathauwer, G. Zhou, Q. Zhao, C. Caiafa,and H. A. Phan, “Tensor decompositions for signal processing applica-tions: From two-way to multiway component analysis,” IEEE SignalProcess. Mag., vol. 32, no. 2, pp. 145–163, Mar. 2015.

[26] F. Xu, S. A. Vorobyov, and X. Yang, “Joint DOD and DOA estimationin slow-time MIMO radar via PARAFAC decomposition,” IEEE SignalProcess. Lett., vol. 27, pp. 1495–1499, Aug. 2020.

[28] P. Pal and P. P. Vaidyanathan, “Nested arrays in two dimensions, part II:Application in two dimensional array processing,” IEEE Trans. SignalProcess., vol. 60, no. 9, pp. 4706–4718, Sep. 2012.

[29] C. Niu, Y. Zhang, and J. Guo, “Interlaced double-precision 2-D angle es-timation algorithm using L-shaped nested arrays,” IEEE Signal Process.Lett., vol. 23, no. 4, pp. 522–526, Apr. 2016.

[30] Y. Dong, C. Dong, Y. Zhu, G. Zhao, and S. Liu, “Two-dimensionaldoa estimation for L-shaped array with nested subarrays without pairmatching,” IET Signal Process., vol. 10, pp. 1112–1117, Dec. 2016.

[31] X. Li, S. Ren, J. Liu, and W. Wang, “Augmented L-shaped nested arraybased on the fourth-order difference co-array concept,” in Proc. IEEE10th Sensor Array Multichannel Signal Process. Workshop, Sheffield,U.K., Jul. 2018, pp. 31–35.

[32] Y. Yang, X. Mao, Y. Hou, and G. Jiang, “2-D DOA estimation viacorrelation matrix reconstruction for nested L-shaped array,” DigitalSignal Process., vol. 98, no. 102623, Nov. 2020.

[33] Z. Zheng and S. Mu, “2-D direction finding with pair-matching operationfor L-shaped nested array,” IEEE Commun. Lett., vol. 25, no. 3, pp. 975–979, Mar. 2021.

[34] T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,”SIAM Rev., vol. 51, no. 3, pp. 455–500, 2009.

[35] A. Phan, P. Tichavsky, and A. Cichocki, “CANDECOMP/PARAFACdecomposition of high-order tensors through tensor reshaping,” IEEETrans. Signal Process., vol. 61, no. 19, pp. 4847–4860, Oct. 2013.

[36] N. D. Sidiropoulos, L. De Lathauwer et al., “Tensor decomposition forsignal processing and machine learning,” IEEE Trans. Signal Process.,vol. 65, no. 13, pp. 3551–3582, Jul. 2017.

[37] F. Xu, M. W. Morency, and S. A. Vorobyov, “DOA estimation fortransmit beamspace MIMO radar via tensor decomposition with Van-dermonde factor matrix,” arXiv: 2101.12666, Jan. 2021.

[38] M. Sørensen and L. De Lathauwer, “Blind signal separation via tensordecomposition with vandermonde factor: Canonical polyadic decompo-sition,” IEEE Trans. Signal Process., vol. 61, no. 22, pp. 5507–5519,Nov. 2013.

[39] M. Shaghaghi and S. A. Vorobyov, “Subspace leakage analysis andimproved DOA estimation with small sample size,” IEEE Trans. SignalProcess., vol. 63, no. 12, pp. 3251–3265, Jun. 2015.

[40] P. Stoica and R. Moses, Spectral analysis of signals. New York: PrenticeHall, Inc, 2005.