2. Signal Space Concepts

R.G. Gallager

The signal-space viewpoint is one of the foundations of modern digital communications.Credit for popularizing this viewpoint is often given to the classic text of Wozencraft andJacobs (1965).

The basic idea is to view waveforms, i.e., finite-energy functions, as vectors in a certainvector space that we call signal-space. The set of waveforms in this space is the set offinite-energy complex functions u(t) mapping each t on the real time axis into a complexvalue. We will refer to such a function u(t) as u : R→ C. Recall that the energy in u isdefined as

||u ||2 =

∫ ∞

−∞|u(t)|2dt. (1)

The set of functions that we are considering is the set for which this integral exists andis finite. This set of waveforms is called the space of finite-energy complex functions, or,more briefly, the space of complex waveforms, or, more briefly yet, L2.

Most of the waveforms that we deal with will be real, and pictures are certainly easier todraw for the real case. However, most of the results are the same for the real and complexcase, so long as complex conjugates are included; since the complex conjugate of a realnumber is just the number itself, the complex conjugates simply provide a minor irritantfor real functions. The Fouier transform and Fourier series for a waveform is usuallycomplex, whether the waveform is real or complex, and this provides an additional reasonfor treating the complex case. Finally, when we treat band-pass functions and their base-band equivalents, it will be necessary to consider the complex case.

CAUTION: Vector spaces are defined over a set of scalars. The appropriate set of scalarsfor a vector space of real functions is R, whereas the appropriate set of scalars for complexfunctions is C. In a number of situations, this difference is important. In vector spacenotation, the set of real functions is a subset of the complex functions, but definitely not avector subspace. We will try to flag the cases where this is important, but ‘caveat emptor.’

1 Vector spaces

A vector space is a set V of elements defined on a scalar field and satisfying the axiomsbelow. The scalar field of primary interest here is C, in which case the vector space iscalled a complex vector space. Alternatively, if the scalar field is R, the vector space

1

is called a real vector space. In either case, the scalars can be added, multiplied, etc.according to the well known rules of C or R. Neither C nor R include ∞. The axiomsbelow do not depend on whether the scalar field is R, C, or some other field. In readingthe axioms below, try to keep the set of pairs of real numbers,viewed as points on a plane,in mind and observe that the axioms are saying very natural and obvious statementsabout addition and scaling of such points. Then reread the axioms thinking about theset of pairs of complex numbers. If you read the axioms very carefully, you will observethat they say nothing about the important geometric ideas of length or angle. We remedythat shortly. The axioms of a vector space are listed below:

(a) Addition: For each v ∈ V and u ∈ V , there is a vector v + u ∈ V called the sum ofv and u satisfying

(i) Commutativity: v + u = u + v ,

(ii) Associativity: v + (u + w) = (v + u) + w ,

(iii) There is a unique 0 ∈ V such that v + 0 = v for all v ∈ V ,

(iv) For each v ∈ V , there is a unique −v such that v + (−v) = 0.

(b) Scalar multiplication: For each scalar α and each v ∈ V there is a vector αv ∈ Vcalled the product of α and v satisfying

(i) Scalar associativity: α(βv) = (αβ)v for all scalars, α, β, and all v ∈ V ,

(ii) Unit multiplication: for the unit scalar 1, 1v = v for all v ∈ V .

(c) Distributive laws:

(i) For all scalars α and all v ,u ∈ V , α(v + u) = αv + αu .

(ii) For all scalars α, β and all v ∈ V , (α + β)v = αv + βv .

The space of finite-energy complex functions is a complex vector space. When we view awaveform v(t) as a vector, we denote it by v . There are two reasons for doing this: first,it reminds us that we are viewing the waveform as a vector; second, v(t) refers both to thefunction as a totality and also to the value corresponding to some particular t. Denotingthe function as v avoids any possible confusion.

The vector sum, v +u is the function mapping each t into v(t)+u(t). The scalar productαv is the function mapping each t into αv(t). The axioms are pretty easily verified forthis space of functions with the possible exception of showing that v + u is a finite-energy function if v and u are. Recall that for any complex numbers v and u, we have|v + u|2 ≤ 2|v|2 + 2|u|2. Thus, if v and u are finite-energy functions,

∫ ∞

−∞|v(t) + u(t)|2 dt ≤

∫ ∞

−∞2|v(t)|2 dt +

∫ ∞

−∞2|u(t)|2 dt < ∞ (2)

Also, if v is finite-energy, then αv has α2 times the energy of v , which is still finite. Weconclude that the space of finite-energy complex functions forms a vector space using Cas the scalars. Similarly, the space of finite-energy real functions forms a vector spaceusing R as the scalars.

2

2 Inner product vector spaces

As mentioned above, a vector space does not in itself contain any notion of length or angle,although such notions are clearly present for vectors viewed as two or three dimensionaltuples of complex or real numbers. The missing ingredient is an inner product.

An inner-product on a complex vector space V is a complex valued function of two vectors,v ,u ∈ V , denoted by 〈v ,u〉, that has the following properties:

(a) Hermitian symmetry: 〈v ,u〉 = 〈u , v〉∗:(b) Hermitian bilinearity: 〈αv + βu ,w〉 = α〈v ,w〉+ β〈u ,w〉

(and consequently 〈v , αu + βw〉 = α∗〈v ,u〉+ β∗〈v ,w〉);(c) Strict positivity: 〈v , v〉 ≥ 0, with equality if and only if v = 0.

A vector space equipped with an inner product is called an inner product space.

An inner product on a real vector space is the same, except that the complex conjugatesabove are redundant.

The norm ||v ||, or the length of a vector v is defined as

||v || =√〈v , v〉.

Two vectors v and u are defined to be orthogonal if 〈v,u〉 = 0.

Thus we see that the important geometric notions of length and orthogonality are bothdefined in terms of the inner product.

3 The inner product space R2

For the familiar vector space Rn of real n-tuples, the inner product of vectors v =(v1, . . . vn) and u = (u1, . . . , un) is usually defined as

〈v ,u〉 =n∑

j=1

vjuj.

You should take the time to verify that this definition satisfies the inner product axiomsabove. In the special case R2 (see Figure 1), a vector v = (v1, v2) is a point in the plane,with v1 and v2 representing its horizontal and vertical location. Alternatively, v can beviewed as the directed line from 0 to the point (v1, v2).

The inner product determines geometric quantities such as length and relative orientationof vectors. In particular, by plane geometry,

• the distance from 0 to u is ||u || =√〈u ,u〉,

3

0©©©©©©©©©©©©©*

©©©©©*AA

AAK

¢¢¢¢¢¢¢̧ 6

-v1

u = (u1, u2) v = (v1, v2)

v2

u |v

u⊥v

Figure 1: Two vectors, u = (u1, u2) and v = (v1, v2) in R2. Note that ||v ||2 =〈v , v〉 = v2

1 + v22 is the squared length of v (viewed as a directed line).

• the distance from v to u (and from u to v) is ||v − u ||

• cos(∠(v − u)) = 〈v ,u〉||v || ||u ||

4 One Dimensional Projections

Using the cosine formula above, or using direct calculation, we can view the vector uas being the sum of two vectors, u = u |v + u⊥v , where u |v is collinear with v andu⊥v = u − u |v is orthogonal to v (see Figure 1). We can view this as dropping aperpendicular from u to v . The word ‘perpendicular’ is a synonym1 for orthogonal andis commonly used for R2 and R3. In particular,

u |v =〈u , v〉||v ||2 v (3)

This vector u |v is called the projection of u onto v . Rather surprisingly, (3) is valid forany inner product space.

Theorem 4.1 (One Dimensional Projection Theorem) Let u and v be vectors inan inner product space (either on C or R). Then u = u|v + u⊥v where u|v = αv (i.e.,u|v is collinear with v) and 〈u⊥v, v〉 = 0 (i.e., u⊥v is orthogonal to v). The scalar α isuniquely given by α = 〈u, v〉/||v||2.

The vector u |v is called the projection of u on v .

Proof: We must show that u⊥v = u−u |v is orthogonal to v .

〈u⊥v , v〉 = 〈u−u |v , v〉 = 〈u , v〉 − 〈u |v , v〉= 〈u , v〉 − α〈v , v〉 = 〈u , v〉 − α||v ||2

which is 0 if and only if α = 〈u , v〉/||v ||2 ¤1Two vectors are called perpendicular if the cosine of the angle between them is 0. This is equivalent

to the inner product being 0, which is the definition of orthogonal.

4

Another well known result in R2 that carries over to any inner product space is thePythagorean theorem: If u and v are orthogonal, then

||u + v ||2 = ||u ||2 + ||v ||2 (4)

To see this, note that

〈u + v ,u + v〉 = 〈u ,u〉+ 〈u , v〉+ 〈v ,u〉+ 〈v , v〉.

The cross terms disappear by orthogonality, yielding (4).

There is an important corollary to Theorem 4.1 called the Schwartz inequality.

Corollary 1 (Schwartz inequality) Let u and v be vectors in an inner product space(either on C or R). Then

|〈u, v〉| ≤ ||u|| ||v|| (5)

Proof: Since u |v and u⊥v are orthogonal, we use (4) to get

||u ||2 = ||u |v ||2 + ||u⊥v ||2.

Since each term above is non-negative,

||u ||2 ≥ ||u |v ||2 = |α|2||v ||2

=

∣∣∣∣〈u , v〉||v ||2

∣∣∣∣2

||v ||2 =|〈u , v〉|2||v ||2

This is equivalent to (5). ¤

5 The inner product space of finite-energy complex

functions

The inner product of two vectors u and v in a vector space V of finite-energy complexfunctions is defined as

〈u , v〉 =

∫ ∞

−∞u(t)v∗(t)dt (6)

Since the vectors are finite energy, 〈v , v〉 < ∞. It is shown in problem 6.3 that thisintegral must be finite. Note that the Schwartz inequality proven above cannot be usedfor this, since we have not yet shown that this is an inner product space. However, withthis finiteness, the first two axioms of an inner product space follow immediately.

The final axiom of an inner product space is that 〈v , v〉 ≥ 0 with equality iff v = 0. Wehave already seen that if a function v(t) is zero for all but a finite number of values oft (or even at all but a countable number), then its energy is 0, but the function is not

5

identically 0. This is a nit-picking issue at some level, but it comes up when we look atthe difference between functions and expansions of those functions.

In very mathematical treatments, this problem is avoided by defining equivalence classesof functions, where two functions are in the same equivalence class if their difference haszero energy. In this type of very careful treatment, a vector is an equivalence class offunctions. However, we don’t have the mathematical tools to describe these equivalenceclasses precisely. As a more engineering point of view, we continue to view functions asvectors, but say that two functions are L2 equivalent if their difference has zero energy.We then ignore the value of functions at discontinuities, but do not feel the need to specifythe class of functions that are L2 equivalent. Thus, for example, we refer to a function andthe inverse Fourier transform of its Fourier transform as L2 equivalent. We all recognizethat it is unimportant what value a function has at a point of discontinuity, and the notionof L2 equivalence lets us ignore this issue.

We will come back to this question of L2 equivalence several times, but for now, we simplyrecognize that the set of finite-energy complex functions is an inner product space subjectto L2 equivalence; this space is called L2 for brevity.

At this point, we can use all the machinery of inner product spaces, including projectionand the Schwartz inequality.

6 Subspaces of inner product spaces

A subspace S of a vector space V is a subset of the vectors in V which form a vector spacein their own right (over the same set of scalars as used by V).

Example: Consider the vector space R3, i.e., the vector space of real 3-tuples. Geomet-rically, we regard this as a volume where there are three orthogonal coordinate directions,and where u1, u2, u3 specify the length of u in each of those directions. We can definethree unit vectors, e1, e2, e3 in these three coordinate directions respectively, so thatu = u1e1 + u2e2 + u3e3.

Consider the subspace of R3 composed of all linear combinations of u and v where, forexample u = (1, 0, 1) and v = (0, 1, 1). Geometrically, the subspace is then the set ofvectors of the form αu + βv for arbitrary scalars α and β. Geometrically, this subspaceis a plane going through the points 0,u , and v . In this plane (as in the original vectorspace) u and v each have length

√2 and the cosine of the angle between u and v is 1/2.

The projection of u on v is u |v = (0, 1/2, 1/2) and u⊥v = (1, 1/2,−1/2). We can viewvectors in this subspace, pictorially and geometrically, in just the same way as we viewedvectors in R2 before.

6

7 Linear independence and dimension

A set of vectors, v 1, . . . , vn in a vector space V is linearly dependent if∑n

j=1 αjv j = 0 forsome set of scalars that are not all equal to 0. Equivalently, the set is linearly dependentif one of those vectors, say vn, is a linear combination of the others, i.e., if for some setof scalars,

vn =n−1∑j=1

αjv j

A set of vectors v 1, . . . , vn ∈ V is linearly independent if it is not linearly dependent, i.e.,if

∑nj=1 αjv j = 0 only if each scalar is 0. An equivalent condition is that no vector in the

set is a linear combination of the others. Often, instead of saying that a set v 1, . . . , vn islinearly dependent or independent, we say that the vectors v 1, . . . , vn are dependent orindependent.

For the example above, u and v are independent. In contrast, u |v , v are dependent. Asa more peculiar example, v and 0 are dependent (α0 + 0v for all α). Also the singletonset {0} is a linearly dependent set.

The dimension of a vector space is the number of vectors in the largest linearly indepen-dent set in that vector space. Similarly, the dimension of a subspace is the number ofvectors in the largest linearly independent set in that subspace. For the example above,the dimension of R3 is 3 and the dimension of the subspace is 2.2 We will see shortly thatthe dimension of L2 is infinity, i.e., there is no largest finite set of linearly independentvectors representing waveforms.

For a finite dimensional vector space of dimension n, we say that any linearly independentset of n vectors in that space is a basis for the space. Since a subspace is a vector spacein its own right, any linearly independent set of n vectors in a subspace of dimension n isa basis for that subspace.

Theorem 7.1 Let V be an n-dimensional vector space and let v1, . . . , vn ∈ V be a basisfor V. Then every vector u ∈ V can be represented as a unique linear combination ofv1, . . . , vn.

Proof: Assume that v 1, . . . , vn ∈ V is a basis for V . Since V has dimension n, every setof n + 1 vectors is linearly dependent. Thus, for any u ∈ V ,

βu +∑

j

αjv j = 0

for some choice of scalars not all 0. Since v 1, . . . , vn ∈ V are independent, β above mustbe non-zero, and thus

u =n∑

j=1

−αj

βv j (7)

2You should be complimented if you stop at this point and say “yes, but how does that follow fromthe definitions.” We give the answer to that question very shortly.

7

If this representation is not unique, then u can also be expressed as u =∑

j γjv j whereγj 6= −αj/β for some j. Subtracting one representation from the other, we get

0 =∑

j

(γj + αj/β)v j

which violates the assumption that {v j} is an independent set. ¤A set of vectors v 1, . . . , vn ∈ V is said to span V if every vector u ∈ V is a linearcombination of v 1, . . . , vn ∈ V . Thus the theorem above can be more compactly statedas: if v 1, . . . , vn ∈ V is a basis for V , then it spans V .

Let us summarize where we are. We have seen that any n-dimensional space must havea basis consisting of n independent vectors and that any set of n linearly independentvectors is a basis. We have also seen that any basis for V spans V . We have also seen thatthe space contains no set of more than n linearly independent vectors. What still remainsto be shown is the important fact that no set of fewer than n vectors spans V . Beforeshowing this, we illustrate the wide range of choice in choosing a basis by the followingresult:

Theorem 7.2 Let u ∈ V be an arbitrary non-zero vector and let v1, . . . , vn be a basis forV. Then u combined with n−1 of the original basis vectors v1, . . . , vn forms another basisof V.

Proof: From Theorem (7.1), we can uniquely represent u as∑

j γjv j where not all {γj}are 0. Let i be an index for which γi 6= 0. Then

v i =1

γi

u −∑

j 6=i

γj

γi

v j (8)

Since the {γj} are unique, this representation of v i is unique. The set of vectorsu , v 1, . . . , v i−1, v i+1, . . . , vn must then be linearly independent. Otherwise a linear com-bination of them, with not all zero coefficients, would sum to 0; this linear combinationcould be added to the right hand side of (8), contradicting its uniqueness. Thus this setof vectors forms a basis. ¤

Theorem 7.3 For an n-dimensional vector space V, any set of m < n linearly inde-pendent vectors do not span V, but can be extended by n −m additional vectors to spanV.

Proof: Let w 1, . . . ,wm be a set of m < n linearly independent vectors and let v 1, . . . , vn

be a basis of V . By Theorem 7.2, we can replace one of the basis vectors {v j} by w 1

and still have a basis. Repeating, we can replace one of the elements of this new basisby w 2 and have another basis. This process is continued for m steps. At each step, weremove one of the original basis elements, which is always possible because w 1, . . . ,wm arelinearly independent. Finally, we have a basis containing w 1, . . . ,wm and n−m vectors

8

of the original basis. This shows that w 1, . . . ,wm can be extended into a basis. Anyvector, say v i, remaining from the original basis is linearly independent of w 1, . . . ,wm.Thus v i ∈ V is not a linear combination of w 1, . . . ,wm, which therefore do not span V .¤We can summarize the results of this section about any n-dimensional vector space (orsubspace) V as follows:

• Every set of n linearly independent vectors forms a basis and spans V .

• Every set of n vectors that spans V is a linearly independent set and is a basis of V .

• Every set of m < n independent vectors can be extended to a basis.

• If a set of n independent vectors in a space U spans U , then U is n-dimensional.

These results can now be applied to the example of the previous section. R3 is a 3-dimensional space according to the definitions here, since the three unit co-ordinate vectorsare independent and span the space. Similarly, the subspace S is a 2-dimenional space,since u and y are independent and, by definition, span the subspace.

For inner product spaces, almost everything we do will be simpler if we consider orthonor-mal bases, and we turn to this in the next section.

8 Orthonormal bases and the projection theorem

In an inner product space, a set of vectors φ1, φ2, . . . is orthonormal if

〈φj,φk〉 =

{0 for j 6= k1 for j = k

(9)

In other words, an orthonormal set is a set of orthogonal vectors where each vector isnormalized in the sense of having unit length. It can be seen that if a set of vectorsv 1, v 2, . . . is orthogonal, then the set

φj =1

‖v j‖v j

is orthonormal. Note that if two vectors are orthogonal, then any scaling (includingnormalization) of each vector maintains the orthogonality.

If we project a vector u onto a normalized vector φ, then, from Theorem 4.1, the projec-tion has the simplified form

u |� = 〈u ,φ〉φ (10)

We now generalize the Projection Theorem to the projection of a vector u ∈ V onto afinite dimensional subspace S of V .

9

8.1 Finite-dimensional projections

Recall that the one dimensional projection of a vector u on a normalized vector φ is givenby (10). The vector u⊥� = u − u |� is orthogonal to φ.

More generally, if S is a subspace of an inner product space V , and u ∈ V , the projectionof u on S is defined to be a vector u |S ∈ S such that u − v |S is orthogonal to all vectorsin S.

Note that the earlier definition of projection is a special case of that here if we take thesubspace S to be the one dimensional subspace spanned by φ. In other words, sayingthat u |� is collinear with φ is the same as saying it is in the one dimensional subspacespanned by φ.

Theorem 8.1 (Projection theorem) Let S be an n-dimensional subspace of an innerproduct space V and assume that φ1,φ2, . . . , φn is an orthonormal basis for S. Then anyu ∈ V may be decomposed as u = u|S +u⊥S where u|S ∈ S and 〈u⊥S , s〉 = 0 for all s ∈ S.Furthermore, u|S is uniquely determined by

u|S =n∑

j=1

〈u,φj〉φj (11)

Note that the theorem above assumes that S has a set of orthonormal vectors as a basis.We will show later that any finite dimensional inner product space has such an orthonomalbasis, so that the assumption does not restrict the generality of the theorem.

Proof: Let v =∑n

i=1 αiφi be an arbitrary vector in S. We first find the conditions on vunder which u − v is orthogonal to all vectors s ∈ S. We know that u − v is orthogonalto all s ∈ S if and only if

〈u − v ,φj〉 = 0, for all j,

or equivalently if and only if

〈u , φj〉 = 〈v ,φj〉, for all j. (12)

Now since v =∑n

i=1 αiφi, we have

〈v ,φj〉 =n∑

i=1

αi〈φi,φj〉 = αj, for all j, (13)

This along with (12) uniquely determines the coefficients {αj}. Thus u − v is orthogonalto all vectors in S if and only if v =

∑j〈u , φj〉φj. Thus u |S in (11) is the unique vector

in S for which u⊥S = u − u |S is orthogonal to all s ∈ S. ¤As we have seen a number of times, if v =

∑j αjφj for orthonormal functions {φj}, then

‖v‖2 = 〈v ,

n∑j=1

αjφj〉 =n∑

j=1

α∗j〈v ,φj〉 =n∑

j=1

|αj|2

10

where we have used (13) in the last step. Thus, for the projection u |S , we have theimportant relation

‖u |S‖2 =n∑

j=1

|〈u , φj〉|2 (14)

8.2 Corollaries of the projection theorem

The projection theorem in the previous subsection assumes the existence of an orthonor-mal basis for the subspace. We will show that such a basis always exists in the nextsubsection, and simply assume that here. Thus, for any n-dimensional subspace S of aninner product space V and for any u ∈ V , there is a unique decomposition of u intotwo orthogonal components, one, u |S in S, and the other, u⊥S , orthogonal to all vec-tors in S (including, of course, u |S). Letting φ1, . . . , φn be an orthonormal basis for S,u |S =

∑j〈u ,φj〉φj.

From the Pythagorean Theorem,

‖u‖2 = ‖u |S‖2 + ‖u⊥S‖2. (15)

This gives us the corollary

Corollary 2 (norm bounds)

0 ≤ ‖u|S‖2 ≤ ‖u‖2. (16)

with equality on the right if and only if u ∈ S and equality on the left if and only if u isorthogonal to all vectors in S.

Recall from (14) that ‖u |S‖2 =∑

j |〈u ,φj〉|2. Substituting this in (16), we get Bessel’sinequality. Bessel’s inequality is very useful in understanding the convergence of orthonor-mal expansions.

Corollary 3 (Bessel’s inequality)

0 ≤n∑

j=1

|〈u,φj〉|2 ≤ ‖u‖2.

with equality on the right if and only if u ∈ S and equality on the left if and only if u isorthogonal to all vectors in S.

Another useful characterization of the projection u |S is that it is the vector in S that isclosest to u . In other words, it is the vector s ∈ S that yields the least squared error(LSE) ‖u − s‖2:

11

Corollary 4 (LSE property) The projection u|S is the unique closest vector s ∈ S tou; i.e., for all s ∈ S,

‖u− u|S‖ ≤ ‖u− s‖with equality if and only if s = u|S .

Proof: By the projection theorem, u − s = u |S + u⊥S − s. Since both u |S and s are inS, u |S − s is in S, so by Pythagoras,

‖u − s‖2 = ‖u |S − s‖2 + ‖u⊥S‖2.

By strict positivity, ‖u |S − s‖2 ≥ 0, with equality if and only if s = u |S . ¤

8.3 Gram-Schmidt orthonormalization

In our development of the Projection Theorem, we assumed an orthonormal basisφ1, . . . , φn for any given n-dimensional subspace of V . The use of orthnormal basessimplifies almost everything we do, and when we get to infinite dimensional expansions,orthonormal bases are even more useful. In this section, we give the Gram-Schmidt pro-cedure, which, starting from an arbitrary basis s1, . . . , sn for an n-dimensional subspace,generates an orthonormal basis.

The procedure is not only useful in actually finding orthonormal bases, but is also usefultheoretically, since it shows that such bases always exist. In particular, this shows thatthe Projection Theorem did not lack generality. The procedure is almost obvious givenwhat we have already done, since, with a given basis, s1, . . . , sn, we start with a subspacegenerated by s1, use the projection theorem on that space to find a linear combination ofs1 and s2 that is orthogonal to s1, and work ourselves up gradually to the entire subspace.

More specifically, let φ1 = s1/‖s1‖. Thus φ1 is an orthonormal basis for the subspace S1

of vectors generated by s1. Applying the projection theorem, s2 = 〈s1,φ1〉φ1 + (s2)⊥S1 ,where (s2)⊥S1 is orthogonal to s1 and thus to φ1.

(s2)⊥S1 = s2 − 〈s1, φ1〉φ1

Normalizing this,φ2 = (s2)⊥S1/‖u⊥S1‖

Since s1 and s2 are both linear combinations of φ1 and φ2, the subspace S2 generated bys1 and s2 is also generated by the orthonormal basis φ1 and φ2.

Now suppose, using induction, we have generated an orthonormal set φ1, . . . , φk to gen-erate the subspace Sk generated by s1, . . . , sk. We then have

(sk+1)⊥Sk= sk+1 −

k∑j=1

〈sk+1, φj〉φj

12

Normalizing this,

φk+1 =(sk+1)⊥Sk

‖( sk+1)⊥Sk‖

Note that sk+1 is a linear combination of φ1, . . . , φk+1. By induction, s1, . . . , sk are alllinear combinations of φ1, . . . , φk. Thus the set of orthonormal vectors {φ1, . . . , φk+1}generates the subspace Sk+1 generated by s1, . . . , sk+1.

In summary, we have shown that the Gram-Schmidt orthogonalization procedure producesan orthonormal basis {φj} for an arbitrary n dimensional subspace S with the originalbasis s1, . . . , sn.

If we start with a set of vectors that are not independent, then the algorithm will find anyvector s i that is a linear combination of previous vectors, will eliminate it, and proceedto the next vector.

9 Orthonormal expansions in L2

We now have the background to understand the orthogonal expansions such as the Sam-pling Theorem and the Fourier series that we have been using to convert waveforms intosequences of real or complex numbers. The Projection Theorem of the last section wasrestricted to finite dimensional subspaces, but the underlying inner product space wasarbitrary. Thus it applies to vectors in L2, i.e., to waveforms.

As an example, consider the Fourier series again, where for each integer k,−∞ < k < ∞,we defined θk(t) as e2πikt/T for τ ≤ t ≤ τ + T and zero elsewhere. Here it is convenient tonormalize these functions, i.e.,

φk(t) =

{ √1/T e2πikt/T for τ ≤ t ≤ τ + T

0 otherwise

The Fourier series expansion then becomes

u(t) =∞∑

k=−∞αkφk(t) where (17)

αk =

∫ ∞

−∞u(t)φ∗k(t) dt (18)

As vectors, u =∑

k αkφk where αk = 〈u ,φk〉. If the infinite sum here means anything,we can interpret it as some kind of limit of the partial sums,

u (n) =n∑

k=−n

〈u ,φk〉φk

We recognize this as the projection of u on the subspace Sn spanned by φk for−n ≤ k ≤ n.In other words, u (n) = u |Sn . From Corollary 3, u |Sn is the closest point in Sn to u . Thisapproximation minimizes the energy between u and any vector in Sn.

13

We also have an expression for the energy in the projection u |Sn . From (14), allowing forthe different indexing here, we have

‖u |Sn‖2 =n∑

k=−n

|〈u , φk〉|2 (19)

This quantity is non-decreasing with n, but from Bessel’s inequality, it is bounded by ‖u‖2,which is finite. Thus, as n increases, the terms ‖〈u ,φn〉‖2 + ‖〈u ,φ−n〉‖2 are approaching0. In particular, if we look at the difference between u |Sm and u |Sn , we see that for n < m,

limn,m→∞

∑

j:n<|k|≤m

‖u |Sm − u |Sn‖2 = limn,m→∞

∑

k:n<|k|≤m

|〈u , φk〉|2 = 0

This says that the sequence of projections {u |Sn} are approaching each other in terms ofenergy difference. A sequence for which the terms approach each other is called a Cauchysequence. The problem is that when funny things like functions and vectors get close toeach other, it doesn’t necessarily mean that they have any limit. Fortunately, the Riesz-Fischer theorem says that indeed these vectors do have a limit (i.e., that Cauchy sequencesfor L2 converge to actual vectors). This property that Cauchy sequences converge tosomething is called completeness. A complete inner product space is called a Hilbertspace.

So what have we learned from this rather long sojourn into abstract mathematics? Wealready knew the Fourier series was a relatively friendly way to turn waveforms intosequences, and we have the intuition that truncating a fourier series simply cuts off thehigh frequency components of the waveform.

Probably the most important outcome is that all orthonormal expansions may be lookedat in a common framework and are all subject to the same types of results. The Fourierseries is simply one example.

An equally important outcome is that as additional terms are added to a truncatedorthonormal expansion, the result is changing (in an energy sense) by increasingly smallamounts. One reason for restricting ourselves to finite energy waveforms (aside fromphysical reality) is that as their energy gets used up in different degrees of freedom (i.e.,components of the expansion), there is less to be used in other degrees of freedom, sothat in an energy sense, we expect some sort of convergence. The exploration of thesemathematical results simply attempts to make this intuition a little more precise.

Another outcome is the realization that these truncated sums can be treated simply withno sophisticated mathematical issues such as equivalence etc. How the truncated sumsconverge, of course, is tricky mathematically, but the truncated sums themselves are verysimple and friendly.

14