The Orbit Method for Nilpotent Lie Groups.

Jethro van Ekeren

1 Introduction.

The orbit method refers to a collection of related procedures by which unitary representationsof a Lie group G can be constructed from certain geometric objects associated to G calledcoadjoint orbits. The method can be made to work perfectly for the class of simply connectednilpotent Lie groups (the case we shall deal with), but modifications of the method work formany other classes of Lie groups.

More generally, the orbit method can be seen as a unifying concept, or philosophy, thatexplains some of the kinds of representations G ought to have, and aids in their classification.The method is quite mysterious in this sense, because there seems to be no explanation yetfor why the method should fail to work in general, and yet still provide important informationfor such a wide variety of groups [5].

Each coadjoint orbit is a symplectic manifold and behaves somewhat like the classical phasespace of a physical system having G as a symmetry group. On the other hand, unitaryrepresentations of G appear naturally in the description of a quantum system with symmetrygroup G. The orbit method is thus closely related to the geometric quantisation program inmathematical physics.

In this document we shall discuss only nilpotent groups and will not tackle the subtletiesinvolved in extending the method to more general groups, nor the relations of the orbitmethod to quantisation.

2 Definitions of relevant objects.

2.1 Nilpotent Lie groups.

A connected Lie group G is said to be nilpotent if its Lie algebra g is nilpotent. A Liealgebra is nilpotent if its lower central series eventually vanishes, i.e., we define g1 = [g, g],g2 = [g, g1] and gk+1 = [g, gk] in general, g is nilpotent if gn = 0 for some n ∈ N.

Although there are non-simply connected nilpotent Lie groups (S1 for example) wheneverwe talk about a nilpotent Lie group we will always assume it is simply connected.

1

Simply connected nilpotent Lie groups have several nice properties that will be used through-out the proofs of the theorems to follow: Every connected Lie subgroup of G is also simplyconnected and nilpotent. G is unimodular (meaning that a left Haar measure is also right-invariant). The exponential map exp : g → G is a diffeomorphism and it carries the Lebesguemeasure on g to a Haar measure on G. See [1] for more details.

2.2 Coadjoint orbits.

Let G be a simply connected nilpotent Lie group and let g be its Lie algebra, so g is nilpotentalso. Every finite-dimensional representation of G gives rise to a representation of g on thesame space. For simply connected Lie groups the converse is also true – there is a bijectivecorrespondence between the finite-dimensional representations of G and those of g.

G acts upon itself via conjugation, i.e., for each g ∈ G there is a smooth map sg : G → Ggiven by σg(x) = gxg−1. This induces a linear map dx(sg) : TxG→ Tgxg−1G on each tangentspace. In particular (setting x = e) it induces a linear map de(σg) : g → g. It is well knownthat

Ad : g �→ Ad(g) = de(σg)

is a representation of G on g, it is called the adjoint representation.

But this representation of G gives rise to a representation of g, also on the vector space g.This is also referred to as the adjoint representation and is denoted ad : g → End (g). It iseasily shown that

ad(X) : Y �→ [X, Y ].

The symmetric bilinear form called the Killing form B : g × g → R, may now be de-fined by B(X, Y ) = Tr (ad(X)ad(Y )). It is a fact that B is invariant, i.e., B([X, Y ], Z) =B(X, [Y, Z]). Note that B is, in general, degenerate (only for semisimple Lie algebras is Bnondegenerate).

We can define a representation K of G on g∗, the dual of g. We let 〈·, ·〉 : g∗× g → R denotethe natural pairing (so 〈λ,X〉 is also often written as λ(X)). K(g)λ is defined by

〈K(g)λ, Y 〉 =⟨λ,Ad(g−1)Y

⟩,

K : G → Aut (g∗) is known as the coadjoint representation. As always, this gives rise to arepresentation k of g which can be shown to be

〈k(X)λ, Y 〉 = 〈λ,−[X, Y ]〉 .

Under the action of G, the vector space g∗ splits up into a number of orbits, called coadjointorbits. These will turn out to be basic objects of study, they are very important. It also turnsout that each coadjoint orbit is a symplectic manifold in a natural way. This is importantin general, but is not necessary for us to know about.

2

2.3 Unitary representations.

Let G be a topological group. A representation ofG is a pair (ρ, V ) where V is a Hilbert spaceand ρ : G → Aut (V ) is a group homomorphism (i.e., ρ(g1g2) = ρ(g1)ρ(g2) and ρ(1) = 1V )that is continuous. Continuity is defined with respect to the strong operator topology onAut (V ) i.e., for each x ∈ V , g �→ ρ(g)x is a continuous function on G.

A representation is unitary if ρ(g) is a unitary operator on V for each g ∈ G. Recall that anoperator A on the Hilbert space V is unitary if

〈Ax, y〉 = 〈x,Ay〉 for all x, y ∈ V .

2.4 Induction of representations.

If G is a Lie group and H a closed subgroup then induction is a means of producing arepresentation of G given a representation (ρ,W ) of H . The construction is as follows: LetV ⊆ L2(G,W ) be the subspace of functions satisfying the covariance relation

f(hx) = ρ(h)f(x) for all h ∈ H , x ∈ G.

Then we have π – the right regular representation of G – acting on V ,

π(g)f(x) = f(xg),

(π, V ) is easily seen to be a representation.

Two important properties of the induction procedure (which we will not prove) are thefollowing:

Lemma 1. Let K be a closed subgroup of H, which is itself a closed subgroup of G a Liegroup, let ρ be a representation of K. Then

Ind GKρ

∼= Ind GH(Ind H

Kρ).

Lemma 2. Let G be a unimodular Lie group. Suppose ρ is a unitary representation of Ha closed subgroup of G. Then Ind G

Hρ is a unitary representation of G. In particular this istrue for all nilpotent groups since they are unimodular [1].

For practical purposes it is often desirable to have a more concrete description of an inducedrepresentation, we will now give such a description. Let G be a simply connected nilpotentLie group andH a closed subgroup. Let (ρ,W ) be a (unitary) representation ofH . X = H\Gis a homogeneous space, it is diffeomorphic to Rn−k (if dimG = n and dimH = k), we canalso find a smooth function (a section) s : X → G such that x = Hs(x) for all x ∈ X (see[1] for details).

Any g ∈ G can now be written uniquely as g = hs(Hg). Consequently we can write

s(x)g = h(x, g)s(xg) (1)

3

and it can be seen that h(x, g) satisfies the equation

h(x, g1g2) = h(x, g1)h(xg1, g2). (2)

Note that this equation holds independently of the choice of section s.

Let V = L2(X,W ) – the vector space of L2 functions f : X → W – we can define arepresentation π of G on V as follows: If f ∈ V set

[π(g)f ](x) = ρ(h(x, g))f(xg).

The fact that this is a representation follows quickly from equation (2), this construction isequivalent to the more abstract construction given initially, the resulting representations ofG are equivalent.

2.5 The correspondence.

We now describe how to construct a unitary representation of G given an element of g∗.We will prove that the representation obtained depends only on the coadjoint orbit Ω thatF lies in. Later we will prove that the resulting representations are irreducible, that theydo not depend on certain choices made in the construction and that all unitary irreduciblerepresentations are obtained in this way.

Let F ∈ g∗ and let h ⊆ g be a subalgebra of maximal dimension that is subordinate to F(this means that F |[h,h] ≡ 0). We may define a 1-dimensional representation of H = exp h

by

ρF,H(expX) = e2πi〈F,X〉.

The fact that this is a representation is easily proved using the Baker-Campbell-Hausdorffformula which states that for X, Y ∈ g there exists Z ∈ g such that expX expY = expZand Z is given by the series,

Z = X + Y + 12[X, Y ] + 1

12[X, [X, Y ]] + 1

12[Y, [X, Y ]] + · · ·

where the higher order terms are quite complicated, but unnecessary for our purposes.

ForX, Y ∈ h all terms in the sum with brackets are annihilated by F , so 〈F, Z〉 = 〈F,X + Y 〉.Thus

ρF,H(expX exp Y ) = ρF,H(expZ) = e2πi〈F,Z〉

= e2πi〈F,X+Y 〉 = ρF,H(expX)ρF,H(expY ).

Now we simply induce a representation of G from ρF,H , i.e., we set πΩ = Ind GHρF,H the

unitary irreducible representation associated to Ω.

4

Now, let F, F ′ ∈ Ω a single orbit under the coadjoint action. Indeed, let F ′ = K(x)F . Leth be a maximal subalgebra subordinate to F and define h′ = Ad(x)h. If X ′, Y ′ ∈ h′, letX ′ = Ad(x)X and Y ′ = Ad(x)Y ,

〈F ′, [X ′, Y ′]〉 = 〈F ′,Ad(x)[X, Y ]〉=

⟨K(x−1)F ′, [X, Y ]

⟩= 〈F, [X, Y ]〉 = 0.

So h′ is subordinate to F ′ and it must be maximal for if there were a larger one h′0 then

we could let h0 = Ad(g−1)h′0 which would have larger dimension than h (since Ad(x) and

Ad(x−1) are automorphisms). Note that

ρF ′,H′(expX) = e2πi〈F ′,X〉 = e2πi〈F,Ad(x−1)X〉= ρF,H ◦ exp ◦Ad(x−1)X = [ρF,H ◦ σx−1 ](expX)

We will now prove a general statement regarding induced representations. Let H be a closedsubgroup of a Lie group G, let σ be an automorphism of G (so σ−1(H) is also a closedsubgroup of G) and let (π, V ) be a representation of H . π ◦ σ is a representation σ−1(H).Also

Lemma 3. ρσ = Ind Gσ−1(H)(π ◦ σ) is equivalent to ρ = (Ind G

Hπ) ◦ σ.

Proof. Let the spaces the two induced representations act on be denoted Wσ and W respec-tively. W and Wσ consist of functions f from G to V such that

f(hg) = π(h)f(g) for all h ∈ H , g ∈ G

and f(hg) = π(σ(h))f(g) for all h ∈ σ−1(H), g ∈ G

respectively.

An intertwiner between these representations is A : W → Wσ defined by [Af ](g) = f(σ(g))(i.e., A ◦ ρ(g) = ρσ ◦ A for all g ∈ G),

[A ◦ ρ(g)]f(x) = A(Lσ(g)f(x)) = Lσ(g)f(σ(x))

= f(σ(x)σ(g)) = f(σ(xg)) = Af(xg)

= [ρσ(g) ◦A]f(x)

as required.

Now let (π, V ) be a unitary representation of G and let x ∈ G, then π and πx = π ◦ σx areequivalent representations. For all g ∈ G, πx(g) = π(xgx−1) = π(x)π(g)π(x)−1, π(x) is aunitary operator, so the representations are unitarily equivalent.

Applying this fact and the lemma, we have

π′ = Ind GH′ρF ′,H′ = Ind G

H′ [ρF,H ◦ σx−1 ]

∼= [Ind GHρF,H ] ◦ σx−1

∼= Ind GHρF,H = π.

so indeed, the representation does not depend on F , only on Ω. We have yet to show thatπ does not depend on h, this is somewhat more involved and will be proved later.

5

3 Two Examples.

Before giving the proofs we will demonstrate the orbit method for two particular Lie groups.First the unique simply connected, 1-dimensional, nilpotent Lie group R×

+. Second theHeisenberg group,

H3 =

⎧⎨⎩ga,b,c =

⎛⎝ 1 a c

0 1 b0 0 1

⎞⎠

∣∣∣∣∣∣ a, b, c ∈ R

⎫⎬⎭ .

The orbit method applied to R×+ is almost trivial. The Lie algebra of R×

+ is R and forany F ∈ R∗ = R, the maximal subordinate subalgebra is all of R. So the correspondingrepresentation is ρ(expX) = e2πiFX , or rather ρ(t) = e2πiF t. It is known that these areprecisely the unitary irreducible representations of R×

+.

The proof that the orbit method works for a general simply connected nilpotent Lie groupwill proceed by induction, R×

+ will be the base case. So of course it is not valid to say wehave used the orbit method to produce the representations of R×

+, this must be proved insome other manner. The same comment applies to the following ‘derivation’ of the unitaryirreducible representations of H3, the fact that these are the representations of H3 will beused in the proof of the orbit method. However using the orbit method to produce themserves the dual purpose of showing how the method works in the simplest nontrivial case,and of displaying the representations of H3, so that we may refer to them later on.

The Lie algebra of G = H3 is

g =

⎧⎨⎩(x, y, z) =

⎛⎝ 0 x z

0 0 y0 0 0

⎞⎠

∣∣∣∣∣∣ x, y, z ∈ R

⎫⎬⎭ .

The exponential map is easily calculated to be

exp

⎛⎝ 0 x z

0 0 y0 0 0

⎞⎠ =

⎛⎝ 0 x z + 1

2xy

0 0 y0 0 0

⎞⎠

so in our chosen coordinates, exp : (x, y, z) �→ (x, y, z + 12xy).

We will now briefly introduce the so-called canonical coordinates of the second kind on G,these are

φ : xP + yQ+ zZ �→ exp(xP ) exp(zZ) exp(yQ).

where P = E12, Q = E23 and Z = E13. For H3 we compute

φ

⎛⎝ 0 x z

0 0 y0 0 0

⎞⎠ =

⎛⎝ 1 x z

0 1 y0 0 1

⎞⎠

6

so in these coordinates φ takes the very simple form (x, y, z) �→ (x, y, z), in other words, thecanonical coordinates (of the second kind) of ga,b,c are (a, b, c).

The group law is

ga,b,c · ga′,b′,c′ = ga+a′,b+b′,c+c′+ab′ .

and the inversion is

g−1a,b,c = g−a,−b,−c+ab.

3.1 Coadjoint orbits.

The adjoint action of G on g is Ad(g) : X �→ gXg−1, so

Ad(ga,b,c) : (x, y, z) �→ (x, y, z − bx+ ay).

We can identify the dual g∗ with the space of strictly lower-triangular matrices via theHilbert-Schmidt inner product. The linear functional

(λ, ν, μ) : (x, y, z) �→ λx+ μy + νz

is represented as

⎛⎝ 0 0 0

λ 0 0ν μ 0

⎞⎠ .

In this description, the coadjoint action of G on g∗ is K(g) : X �→ p(g−1Xg) where p(A)denotes the strictly lower-triangular part of the matrix A. A simple computation shows that

K(ga,b,c)(λ, μ, ν) = (λ+ bν, μ− aν, ν).

We can now easily deduce the coadjoint orbits of G. If ν �= 0 then we can obviously chooseappropriate a and b to map (λ, μ, ν) to (0, 0, ν). If ν = 0 then (λ, μ, 0) is G-stable. So wehave two classes of orbits, a 1-parameter family Ων where ν ∈ R\{0} and a 2-parameterfamily Ωλ,μ where λ, μ ∈ R.

3.2 Corresponding unitary irreducible representations.

Consider F = (λ, μ, 0) ∈ Ωλ,μ, the kernel of F is RZ ⊆ g. So we need to find a maximalh ⊆ g such that [h, h] ⊆ RZ. But h = g will do the trick. The corresponding unitaryrepresentation of H = G is 1-dimensional,

π(exp(X)) = e2πi〈F,X〉 = e2πi(λx+μy),

7

this becomes

π(ga,b,c) = e2πi(λa+μb).

Next consider F = (0, 0, ν) ∈ Ων , the kernel of F is R 〈X, Y 〉 ⊆ g. h cannot be all of g, soh = R 〈X,Z〉 is a possible choice, [h, h] = 0. The corresponding unitary representation ofH = {ga,0,c} is

ρ(exp(X)) = e2πi〈F,X〉 = e2πiνz ,

this becomes

ρ(ga,0,c) = e2πiνc.

Now we need to induce this up to a representation of G. The induced representation will acton the space of L2 functions on X = H\G. Now, {exp(tY )}t∈R is a full set of representativesof the cosets, so let s([ga,b,c]) = g0,b,0, we now want to solve equation (1),

s(x)g = h(x, g)s(xg)⎛⎝ 1 0 0

0 1 t0 0 1

⎞⎠

⎛⎝ 1 a c

0 1 b0 0 1

⎞⎠ =

⎛⎝ 1 a c

0 1 00 0 1

⎞⎠

⎛⎝ 1 0 0

0 1 b+ t0 0 1

⎞⎠

⎛⎝ 1 a c

0 1 b+ t0 0 1

⎞⎠ =

⎛⎝ 1 a a(b+ t) + c

0 1 b+ t0 0 1

⎞⎠

so a = a and c = c− ab− at.

Then the induced representation π = Ind GHρ is given by

[π(ga,b,c)f ](t) = ρ(h(x, g))f(tg)

= e2πiν(c−ab−at)f(t+ b).

These are all the unitary irreducible representations of the Heisenberg group (up to unitaryequivalence).

4 Proof of the method in general.

Throughout this section, C will denote a central subgroup of G (that is, a subgroup ofZ(G)). Before starting with the proofs we will record three structural lemmas pertainingto nilpotent Lie algebras with one-dimensional centre, two regarding the structure of thesealgebras themselves, the other regarding representations of the corresponding Lie groups.

8

Lemma 4 (Kirillov’s lemma). Let g be a nilpotent Lie algebra with a 1-dimensional centrez. Then g has a basis {P,Q, Z,W1, . . .Wk} such that

z = RZ, [P,Q] = Z and [Wi, Q] = 0 (1 ≤ i ≤ k).

Proof. Let Z ∈ z be nonzero. Consider the Lie algebra g = g/z. It is nilpotent, hence hasnonzero centre z, choose a nonzero Q ∈ z and let Q be a preimage of Q under the projection,Q is not in z but is central modulo z (i.e., [Q, g] ⊆ z). Therefore there exists an element P ∈ g

such that [P,Q] �= 0, but [P,Q] ∈ z, without loss of generality we may assume [P,Q] = Z.

Since Q is central modulo z, the centraliser of Q has codimension 1 in g, so we can findelements W1,W2, . . .Wk of the centraliser that together with Q and Z form a basis.

Let g0 denote the centraliser of Q, i.e., g0 = R 〈Q,Z,W1, . . .Wk〉, so g = g0 ⊕ RP .

Lemma 5. g0 is an ideal in g.

Proof. As the centraliser of Q, g0 is a subalgebra of g. Let X ∈ g0 and suppose [X,P ] =αP +X ′ where X ′ ∈ g0. Then [X, [X,P ]] = α2P +X ′′ where X ′′ ∈ g0 too. We can continuethis process, but since g is nilpotent we must have α = 0. So [X,P ] ∈ g0 and g0 is anideal.

We put G0 = exp g0, which is a normal subgroup of G. The centre Z(G) of G is a subgroupof G0.

Lemma 6. Let G be a simply connected, nilpotent Lie group whose centre is 1-dimensional,let g be its Lie algebra and let (π, V ) be a unitary irreducible representation of G. Then oneof the following must occur,

• π|Z(G) = 1V .

• π is induced from a unitary irreducible representation (ρ,W ) of G0 and ρ|Z(G) �= 1W .

Proof. Let P , Q, Z, etc. be as in lemma (4) and put a = RQ⊕ RZ, an abelian subalgebraof g. Let A = exp a, A is diffeomorphic to R2 and exp is a diffeomorphism.

A multiplicative character of a group G is a continuous group homomorphism χ : G → C×.Let M = A, the Pontrjagin dual of A (the space of multiplicative characters of A). M ishomeomorphic to the Pontrjagin dual of R2 which is just R2. The elements of M are

χμ,λ : exp(yQ+ zZ) �→ e2πi(μy+λz)

and giving χμ,λ the coordinates (μ, λ) makes M a manifold.

We can compute the coadjoint action of G on M to be the following,

K(g0 exp tP ) : (μ, λ) �→ (μ− tλ, λ) where g0 ∈ G0.

9

The adjoint action (and therefore coadjoint action) of G0 on A is trivial because [g0, a] = 0.The action of exp tP can be computed explicitly. The G-action has a certain property thatwe will not define – it is tame, this will soon be necessary for the application of a technicallemma.

The orbits of M under this action of G are the lines Ωλ = {(t, λ)|t ∈ R} for each λ �= 0 andthe point orbits Ωμ = {(μ, 0)} for any μ ∈ R. The stabiliser subgroup of Ωλ is G0 and thestabiliser subgroup of Ωμ is G itself.

We must make a general definition before proceeding (see [3] for details and references): A∗-representation Π of a smooth G-manifold M on a Hilbert space V is an associative algebrahomomorphism Π : A(M) → End (V ) such that

Π(f) = Π(f)∗ for all f ∈ A(M).

(A(M) is the space of compactly supported smooth functions on M , it is an associative alge-bra under pointwise multiplication). Π is said to be nondegenerate if {Π(f)v|f ∈ A(M), v ∈V } is dense in V .

Π is compatible with a representation (π, V ) of G if:

π(g) ◦ Π(f) ◦ π(g)−1 = Π(R(g)f) for all g ∈ G, f ∈ A(M).

(R(g) is the right regular representation here).

Nondegenerate ∗-representations Π of M are all of the form

Π(f) =

∫M

f(m)dμ(m)

where μ is a projector-valued measure. If B(M) is the σ-algebra of Borel subsets of M andP(V ) is the set of projection operators P : V → V (i.e., those satisfying P 2 = P = P ∗) thena projector-valued measure on M is a map μ : B(M) → P(V ) that is countably additive,such that μ(M) = 1V .

The important theorem, which we will by no means prove, is the following:

Theorem 7. Let G be a Lie group and let M be a smooth manifold with a tame G-action.Let (π, V ) be a unitary irreducible representation of G and let Π be a nondegenerate ∗-representation of M compatible with π. Then

• The projector-valued measure corresponding to Π is concentrated on a single G-orbitΩ ⊆M .

• π is induced from a unitary irreducible representation ρ of the stabiliser of any m ∈ Ω.

Now, let (π, V ) be a unitary irreducible representation of G. We wish to construct a non-degenerate ∗-representation Π of A(M) on V compatible with π. Let f ∈ A(M), then we

10

have the inverse Fourier transform

f(y, z) =

∫M

f(μ, λ)e2πi(μy+λz)dμdλ

= F−1 [ψ(y, λ) : λ→ z]

where ψ(y, λ) = F−1[f(μ, λ) : μ→ y]

we put

Π(f) =

∫A

f(y, z)π(exp(yQ+ zZ))dydz

which can be shown to do the job.

By theorem (7), Π must correspond to a projector-valued measure concentrated on a singleorbit of M and π is induced from a unitary irreducible representation of the stabiliser sub-group for that orbit, there are now two cases. If the orbit is Ωλ then π is induced from arepresentation of G0 and we are done.

On the other hand, if the orbit is Ωμ then the stabiliser is G itself and we have not learnedanything. However Ωμ = {(μ, 0)} is a single point. So we can see that the correspondingprojector-valued measure must actually be

m(E) =

{1V if (μ, 0) ∈ E0 otherwise.

and so Π(f) = f(μ, 0)1V .

So

f(μ, 0)1V =

∫A

f(y, z)π(exp(yQ))π(exp(zZ))dydz

=

∫ ∞

−∞

(∫ ∞

−∞f(y, z)e−2πiqzdz

)π(exp(yQ))dy

=

∫ ∞

−∞ψ(y, q)π(exp(yQ))dy

here we have used that exp(zZ) commutes with all operators π(g) and hence by Schur’slemma is a multiple of the identity. Indeed we must then have π(exp(zZ)) = e−2πiqz forsome q ∈ R.

If we replace f(μ, λ) by f(μ, kλ) in the equations above then the LHS remains the same, butψ(y, q) in the RHS becomes ψ(y, kq). We can now conclude that q = 0. For if q �= 0 then kqranges over R as k does, we deduce that ψ(y, λ) (and therefore f(μ, λ)) does not depend onλ, but we clearly could have chosen f that does depend on λ to obtain a contradiction.

Consequently π(exp(zZ)) = 1V and we are done.

11

We need one further result of a more general nature regarding induction of representations.Let G be a nilpotent Lie group and C ⊆ Z(G) a subgroup, let p : G → G = G/C bethe canonical projection. Representations (π, V ) of G with the property π|C = 1V andrepresentations (π, V ) of G are in bijective correspondence via π = π ◦ p. The bijectionpreserves unitarity and irreducibility. We also have:

Lemma 8. If π = Ind GHρF,H where h is maximal subordinate to F and

ρF,H(expX) = e2πi〈F,X〉

then π = Ind GHρF ,H where H = p(H) is maximal subordinate to F = F and

ρF ,H(expX) = e2πi〈F ,X〉.

The converse also holds.

Proof. There is an isomorphism ϕ : g∗ → c⊥ ⊆ g∗ defined by

〈ϕ(λ), X〉 = 〈λ, p∗(X)〉 .

For any g ∈ G, X ∈ c and f ∈ V ,

f(g) = π(expX)f(g) = f(g expX) = f(expXg) = e2πi〈F,X〉f(g),

which can only be achieved if F |c = 0 (conversely if F |c = 0 then π|C = 1V ). So F mustactually ‘live’ in g∗. I will refer to the canonical image of F in g∗ as F for notational nicety.

Clearly⟨F , p∗(X)

⟩= 〈F,X〉 for all X ∈ g.

Any subalgebra h of g subordinate to F and of maximal dimension must contain c (indeed allof z(g)). Thus h ⊆ g is maximal subordinate if and only if h = p∗(h) is maximal subordinatefor F .

Set σ = Ind GHρF ,H , where

ρF ,H(exp(X)) = e2πi〈F ,X〉 (for X ∈ h)

and set σ = σ ◦ p. We wish to confirm that σ is equivalent to our original π.

π and σ act on the spaces

V = {f ∈ L2(G,C)|f(hg) = ρF,H(h)f(g) for all h ∈ H},V = {f ∈ L2(G,C)|f(hg) = ρF ,H(h)f(g) for all h ∈ H}

respectively and σ also acts on V . Define ψ : V → V by [ψ(f)](g) = f(p(g)), we need tocheck that it is an intertwiner for the representations σ and π. ψ is certainly an isomorphismbetween the spaces L2(G) and L2(G), but does it map V onto V ?

12

Let h = expX ∈ H , then

[ψ(f)](hg) = f(p(hg)) = f(p(h)p(g))

= ρF ,H(p(h))f(p(g)) = e2πi〈F ,p∗(X)〉f(p(g)) = e2πi〈F,X〉f(p(g))

= ρF,H(h)[ψ(f)](g)

so ψ is an isomorphism between V and V .

It is clear that ψ is an intertwiner because

π(g)[ψ(f)](x) = [ψ(f)](xg) = f(p(xg))

= f(p(x)p(g)) = σ(p(g))f(p(x)) = σ(g)[ψ(f)](x)

as required.

Now we are finally ready to prove the main theorems:

Theorem 9. All unirreps of G are given by the coadjoint orbit procedure described earlier.

Proof. We have already noted that this is true for R×+. We will therefore proceed by induction

on dimG. Assume that dimG > 1 and that the result holds for all simply connected,nilpotent Lie groups of dimension less than dimG. We discern two cases:

Case I: dimZ(G) > 1.

Let k = dimZ(G) and let {Z1, . . . Zk} be a basis of z(g). Then for any unirrep (π, V ) of Gwe have

π(exp

(∑ziZi

))= e2πi

∑qizi

for some numbers q1, q2, . . . qk ∈ R. Let z0 = {∑ ziZi ∈ z(g)|∑ qizi = 0} and let Z0 =exp(z0), these have dimension at least 1. Clearly π|Z0 = 1V , so we define the factor repre-sentation π on G = G/Z0, it is a unirrep.

dim G < dimG, so by the inductive assumption, π ∼= Ind GHρF ,H where F ∈ g, h is a maximal

subalgebra of g subordinate to F and

ρF ,H(exp(X)) = e2πi〈F ,X〉 (for X ∈ h).

is a representation of H = exp(h). Lemma (8) immediately tells us that π ∼= Ind GHρF,H

where

ρF,H(expX) = e2πi〈F,X〉 (for X ∈ h)

(h = p−1∗ (h), H = p−1(H) and F is just F extended by 0 to all of g).

13

Case II: dimZ(G) = 1.

According to lemma (6) either π|Z(G) = 1V , in which case we reduce to the previous case, orelse π ∼= Ind G

G0ρ where ρ is a unirrep of G0 and ρ|Z(G) �= 1V .

By the inductive assumption, ρ is itself of the form Ind G0H0ρF0,H0 so we have π ∼= Ind G

H0ρF0,H0

(here F0 ∈ g∗0). We wish to show that there exists an F ∈ g∗ with maximal subordinate

subalgebra h0 and ρF,H0(h) = ρF0,H0(h) for all h ∈ H0.

ρ|Z(G) is nontrivial, let f ∈ Vρ and let Z(G) = RZ, then

ρ(exp(zZ))f(g) = f(g exp(zZ)) = f(exp(zZ)g)

= ρF0,H0(exp(zZ))f(g) = e2πiz〈F0,Z〉f(g)

so the action can only be nontrivial if 〈F0, Z〉 �= 0, so Z /∈ [h0, h0]. We will use this fact soon.

Note that g∗0 is isomorphic to g∗/RλP where λP is the linear functional on g that maps P to

1 and all other basis vectors to 0. Let p : g∗ → g∗0 be the projection, then if F ∈ p−1(F0), it

is clear that

〈F,X〉 = 〈F0, X〉for any X ∈ g0. Note that Q ∈ h0 because [Q, g0] = 0 so Q /∈ h0 would contradictmaximality of h0. h0 is also subordinate to F , but it is not immediately clear that it hasmaximal dimension.

Let h be some maximal subordinate subalgebra for F . Either h ⊆ g0 in which case h ismaximal subordinate for F0 as well and thus has dimension no larger than h0.

The other possibility is P +W ∈ h for some W ∈ g0, then Q /∈ h for otherwise [P +W,Q] =Z ∈ [h, h] which is not allowed since 〈F, Z〉 �= 0. Put h′ = (h∩ g0)⊕RQ which has the samedimension as h and is maximal subordinate for F0. It therefore has the same dimension ash0, which must therefore be maximal subordinate for F .

Thus

ρF,H(exp(X)) = e2πi〈F,X〉 = ρF0,H0(exp(X))

for all X ∈ h0.

We have shown that our method produces all unitary irreducible representations, we nowneed to show that it produces no others.

Theorem 10. All representations produced by the method are unirreps and do not dependon the maximal h we choose (for a fixed F ).

Proof. Once again, we know this for dimG = 1, so assume dimG > 1 now and that theresult holds for all groups of smaller dimension.

14

Let F ∈ g∗ be given and let h be any subalgebra of g of maximal dimension subordinate toh. Define

ρF,H(exp(X)) = e2πi〈F,X〉 (for X ∈ h).

as usual and set π = Ind GHρF,H (say the space of the representation is V ).

Case I: dimZ(G) > 1, or else Z(G) = exp RZ is 1-dimensional

and 〈F, Z〉 = 0.

The subspace z0 of z(g) on which F ≡ 0 is nontrivial. Letting Z0 = exp z0, we find thatπ|Z0 = 1V . So setting G = G/Z0 and p : G→ G as in lemma (8), π factors through p to givethe representation π of G and π = π ◦ p.By lemma (8), π = Ind G

HρF ,H , where

ρF ,H(exp(X)) = e2πi〈F ,X〉 (for X ∈ h).

By the inductive assumption π is a unirrep of G. Therefore, by lemma (8) again (or rather,the comments immediately preceding it), π is a unirrep too.

Say we have two maximal subalgebras h1 and h2 subordinate to F , and the two correspondingrepresentations π1 and π2. These factor to corresponding representations π1 and π2 of Gwhich are both attached to F . By the inductive assumption π1 and π2 are equivalent, henceso are π1 and π2.

Case II: dimZ(G) = 1 and 〈F, Z〉 �= 0.

We may assume without loss of generality that 〈F, Z〉 = 1 and 〈F,Q〉 = 0 (by makingappropriate choices in the proof of Kirillov’s lemma). We now further divide cases:

Case IIa: h ⊆ g0.

In this case we can write π = Ind GG0π0 where π0 = Ind G0

H ρF,H is a representation of G0 (andsay π0 acts on the space V0). h is clearly maximal subordinate to F as a subalgebra of g0,so by the inductive assumption, π0 is a unirrep of G0.

Schur’s lemma states that if (ρ, V ) is a unitary representation of a group G then ρ is irre-ducible if and only if the intertwiners of ρ (i.e., the bounded linear maps commuting withρ(g) for all g ∈ G) are the multiples of the identity λ1V . With this in mind, let A be anintertwiner for π.

15

Using the Baker-Campbell-Hausdorff formula, we obtain

π(exp xP )f(exp tP ) = f(exp (t+ x)P )

and π(exp yQ)f(exp tP ) = f(exp[tP + yQ+ 12tyZ])

= f(exp tyZ exp yQ exp tP )

= e2πityf(exp tP )

(3)

Recall lemma (5), which says g0 is an ideal in g, thus G0 is a normal subgroup of G. Thereforeif g0 ∈ G0,

π(g0)f(exp tP ) = f(exp tPg0) = f(exp tPg0 exp [−tP ] exp tP )

= π0(exp tPg0 exp [−tP ])f(exp tP )

which we will write as

π(g0)f(exp tP ) = πt(g0)f(exp tP )

where πt(g0) = π0(exp tPg0 exp [−tP ]) (by a fortuitous juxtaposition of events, the represen-tation πt with t = 0 does coincide with π0, so the notation is well-defined :). Each πt is arepresentation of G0 equivalent to π0, hence unitary and irreducible.

For any fixed y ∈ R, π(exp yQ) : f(t) �→ e2πityf(t). If Mφ and M ′φ are as in lemma (12) then

by the Stone-Weierstrass theorem, A ◦Mφ = M ′φ ◦ A for any φ ∈ C∞

c (R). Since this latterspace is weakly dense in L∞(R), we conclude the same holds for φ ∈ L∞(R).

We now want to apply lemma (12) to this situation. Take S = R and μ the Lebesguemeasure. Let K = K ′ = V0, then H = H ′ = V . The norm-dense subspaces required inthe lemma are provided by the space V ∞ of smooth vectors in V . A discussion (or indeed adefinition) of smooth vectors would be prohibitively lengthy at this point, but [3] providessome of the background theory.

Suffice to say that we may apply the lemma and we can write

[Af ](exp tP ) = At[f(exp tP )],

the operators At (t ∈ R) are bounded. Now, since π(g0)A = Aπ(g0), we get Atπt(g0) =πt(g0)At for each g0 ∈ G0, t ∈ R. By Schur’s lemma for πt, we have At = λ(t)IV0 .

But

Aπ(exp xP ) = π(exp xP )A

and π(exp xP ) : f(exp tP ) �→ f(exp (t+ x)P ),

so we must have λ(t) is constant. Thus by Schur’s lemma for π, π is irreducible.

dim g0 < dim g, so by the inductive assumption π0 is independent of the choice of maximalsubordinate subalgebra h (so long as it is contained in g0). Therefore π = Ind G

G0π0 doesn’t

depend on the choice of h either.

16

Case IIb: h � g0.

Our approach will be to find a subalgebra h′ ⊆ g0 of the same dimension as h, also subordinateto F with the property that π′ obtained by inducing ρF,H′ is equivalent to π. We then reduceto Case IIa.

Let h0 = h ∩ g0, which has dimension one less than h. Adding any element of g0 onto Pdoes not change the property of being a Kirillov basis, we may assume therefore that P ∈ hand so h = h0 ⊕ RP (for later convenience, we will also add a multiple of Z onto P so that〈F, P 〉 = 0). Now, Q /∈ h since otherwise Z ∈ [h, h] and we know 〈F, Z〉 �= 0, so defineh′ = h0 ⊕ RQ which has the same dimension as h.

Since Q is central in g0, h′ is a subalgebra of g0. h′ is subordinate to F , so it is clearlymaximal in this regard. Now define k = h0 ⊕ RP ⊕ RQ and k0 = h0 ∩ kerF .

Lemma 11. k is a subalgebra of g and k0 is an ideal in k.

Proof. k = h ⊕ RQ, we know already that h is a subalgebra of g. Note that Z ∈ h ⊆ k sinceZ is central and thus maximality of h would be contradicted otherwise. Q is central in g0

and [P,Q] = Z ∈ k, so k is indeed a subalgebra.

Let X ∈ k and Y ∈ k0. [X, Y ] ∈ k because it’s a subalgebra, if X ∈ k0 then 〈F, [X, Y ]〉 = 0because X, Y ∈ h which is subordinate to F , so [X, Y ] ∈ k0.

Next, [Q, Y ] = 0 ∈ k0 because Y ∈ g0, in which Q is central. Finally, 〈F, [P, Y ]〉 = 0 becauseh is subordinate to F , as above. Writing [P, Y ] = αP + W where W ∈ h0, we apply thesame argument as we used in lemma (5) to deduce that α = 0 and so [P, Y ] ∈ h0, hence k0.

So k0 is an ideal in k.

Now h and h′ are both subalgebras of k. They both have codimension 1, but 〈F, [P,Q]〉 =〈F, Z〉 �= 0 so they are both maximal subordinate to F . So we may induce the representationsρF,H and ρF,H′ up to K – call them τ and τ ′ respectively – then π ∼= Ind G

Kτ and defineπ′ ∼= Ind G

Kτ′.

We claim that τ |K0 = 1. For let k0 ∈ K0, then (using that K0 is a normal subgroup of K)

τ(k0)f(exp tQ) = f(exp tQk0 exp [−tQ] exp tQ)

= τ(exp tQk0 exp [−tQ])f(exp tQ) = f(exp tQ).

since F |k0 = 0. Exactly the same argument, with P in place of Q shows that τ ′|K0 = 1 too. Ifdim h0 ≥ 2 then k0 �= {0}. τ and τ ′ factor through to representations of the quotient K/K0.We apply the inductive assumption to see that τ ∼= τ ′ and hence π ∼= π′.

If dim h0 = 1 then h0 = RZ so k = RP ⊕ RQ ⊕ RZ is actually the Heisenberg algebraand K the Heisenberg group, h = RZ ⊕ RP and h′ = RZ ⊕ RQ. The final step is to writedown τ and τ ′ explicitly using the same kind of application of the Baker-Campbell-Hausdorffformula as we used in equation (3) and then demonstrate that they are equivalent.

17

Letting f(t) stand for f(exp tQ), we can regard τ as acting on L2(R), we have

τ(exp xP )f(t) = e−2πitxf(t),

τ(exp yQ)f(t) = f(t+ y),

τ(exp zZ)f(t) = e2πizf(t).

Similarly, if we let f(s) stand for f(exp sP ), then τ ′ also acts on L2(R) and

τ ′(exp xP )f(s) = f(s+ x),

τ ′(exp yQ)f(s) = e−2πityf(s),

τ ′(exp zZ)f(s) = e2πizf(s).

The Fourier transform F : L2(R) → L2(R)

[Ff ](t) =

∫ ∞

−∞e−2πitsf(s)ds

is known to be an isometry. We can easily show that F ◦ τ(exp xP ) = τ ′(exp xP ) ◦ F , etc.,therefore F is an intertwiner, τ and τ ′ are equivalent and so π and π′ are equivalent.

Lemma 12. Let K, K ′ be Hilbert spaces, let (S, μ) be a locally compact Hausdorff spacewith μ a σ-finite measure and let H = L2(S, μ,K), H ′ = L2(S, μ,K ′). Let A : H → H ′ bea bounded linear operator and let V , V ′ be norm-dense subspaces of H and H ′ respectively,satisfying

• If f ∈ V then f : S → K is continuous and similarly for V ′.

• For each s ∈ S, {f(s)|f ∈ V } is dense in K, likewise for V ′ and K ′.

• A(V ) ⊆ V ′.

• For each φ ∈ L∞(S, μ), A ◦Mφ = M ′φ ◦ A.

where Mφ is defined by [Mφf ](s) = φ(s)f(s), and similarly for M ′φ.

Then there are bounded linear operators As : K → K ′ (for each s ∈ S) such that

[Af ](s) = As(f(s)) for all f ∈ V .

Also for each s ∈ S, ‖As‖ ≤ ‖A‖.

Proof. We must first show that

|[Af ](s)| ≤ ‖A‖ |f(s)| (4)

18

for all f ∈ V , s ∈ S. This tells us that if f(s) = 0 then [Af ](s) = 0, which means that wecan define As on Vs = {f(s)|f ∈ V } ⊆ K by

As[f(s)] = [Af ](s)

and this is well-defined. Also we immediately have |Asf | ≤ ‖A‖ |f | for f ∈ Vs. As mustextend to a unique bounded linear operator mappingK toK ′. It clearly satisfies ‖As‖ ≤ ‖A‖.We now turn to proving (4). Fix s0 ∈ S, let c = |f(s0)| and C = |[Af ](s0)|. Let ε > 0 begiven and let U be an open neighbourhood of s0 such that |f(s)| < c+ ε and |Af(s)| > C− εfor all s ∈ U (choose U to have finite measure also).

Let χU be the indicator function of U , we have

∫U

|Af(s)|2dμ(s) =⟨M ′

χUAf,M ′

χUAf

⟩= 〈AMχU

f, AMχUf〉

= |AMχUf |2 ≤ ‖A‖2 |MχU

f |2≤ ‖A‖2 (c+ ε)2μ(U)

and

∫U

|Af(s)|2dμ(s) ≥ (C − ε)2μ(U)

Thus C − ε ≤ ‖A‖ (c + ε), letting ε → 0 shows that C ≤ ‖A‖ c. Since this holds for anys0 ∈ S we have (4) as required.

It is quite easy to prove that if A is unitary then all the As are unitary as well.

There is still one thing left to prove. We have already seen that all F in a given Ω give riseto the same representation, we would like to show that every orbit gives rise to a distinctrepresentation (so there are no two distinct orbits Ω1 and Ω2 giving rise to the same unirrepof G).

Theorem 13. Let F1, F2 ∈ g∗, with maximal subordinate subalgebras h1 and h2 and induced(irreducible, unitary) representations π1 and π2 of G. Suppose π1 and π2 are equivalent, thenF1 and F2 are conjugate by an element of G (i.e., F1 and F2 lie in the same coadjoint orbit).

Proof. The proof is by induction and follows the same basic pattern as the other two com-ponent theorems. For this reason we will be more brief in presenting the proof this time.The base case is R×

+ for which the result is known to hold. For the inductive step there aretwo cases.

Case I: Either π or π′ is trivial on a 1-dimensional subgroup C of Z(G).

Since π and π′ are equivalent, both are trivial on C if one is. So F |c ≡ F ′|c ≡ 0. We candefine the quotient representations π and π′ of G = G/C (which are equivalent) and define

19

F , F ′ ∈ g. We saw in the proof of theorem (9) that π is the unirrep of G associated to Fand π = π ◦ p (where p : G→ G is the canonical projection) and likewise π′.

By the inductive assumption, F and F ′ are conjugate under the coadjoint action of G, solet F ′ = K(g)F . Now let g ∈ G be some preimage of g under the canonical projection, then

〈K(g)F,X〉 =⟨F,Ad(g−1X)

⟩=

⟨F , p∗(Ad(g−1))X

⟩

=⟨F ,Ad(g−1)p∗(X)

⟩=

⟨K(g)F , p∗(X)

⟩=

⟨F ′, p∗(X)

⟩= 〈F ′, X〉 .

this holds for all X ∈ g, hence F ′ = K(g)F and we are done.

Case II: Z(G) is 1-dimensional, neither π nor π′ is trivial on Z(G).

In this case let P , Q and Z be as in lemma (4). We have the flexibility to choose 〈F, Z〉 = 1and 〈F, P 〉 = 〈F,Q〉 = 0. Since π(Z) and π′(Z) act as scalars, we can see easily that〈F ′, Z〉 = 1 too.

Also,

〈K(exp xP )F ′, Y 〉 = 〈F ′,Ad(exp[−xP ])Y 〉 = 〈F ′, exp[ad(−xP )]Y 〉= 〈F ′, Y − xZ〉 = 〈F ′, Y 〉 − x.

so we may replace F ′ by a conjugate if necessary to arrange that 〈F ′, Y 〉 = 0.

We know from lemma (6) that π and π′ are induced from unitary irreducible representationsσ0 and σ′

0 of G0. These are, in turn, ρF0,H0 and ρF ′0,H′

0induced from maximal subordinate

subalgebras h0 and h′0 respectively, where F0 = F |g0 and F ′

0 = F ′|g0.

Using the equivalence of π and π′, we would like to show that σ0 and σ′0 are equivalent.

Since π is induced from σ0, the restriction of π to G0 can be written as

π(g0)f(exp tP ) = σt(g0)f(exp tP )

where σt(g0) = σ0(exp tPg0 exp [−tP ]) (a similar calculation was done in theorem (10)).Using the special forms we assumed for P,Q and Z (and the Baker-Campbell-Hausdorffformula in the case of Q) we can explicitly compute

π(exp xP )f(t) = f(t+ x),

π(exp yQ)f(t) = e2πityf(t),

π(exp zZ)f(t) = e2πizf(t).

(and something similar for π′).

Let A be an intertwiner for the representations π and π′. As in theorem (10), we getA ◦Mφ = M ′

φ ◦ A for any φ ∈ L∞(R). By lemma (12) we now get a collection of bounded

20

linear operators At : Vσ0 → Vσ′0. At is easily seen to be an intertwiner between σt and σ′

t, inparticular, A0 is an intertwiner between σ0 and σ′

0.

We can finally apply the inductive assumption and deduce that F ′0 = K(g0)F0 for some

g0 ∈ G0, this means F ′ = K(g0)F when restricted to g0. Before proceeding, note that on g0,K(exp yQg0)F = K(exp yQ)K(g0)F = K(g0)F since Q is central in g0. Now

〈K(exp yQg0)F, P 〉 =⟨F,Ad(g−1

0 )Ad(exp [−yQ])P⟩

=⟨F,Ad(g−1

0 )[P + yZ]⟩

= 〈K(g0)F, P 〉 + y 〈F, Z〉= 〈K(g0)F, P 〉 + y.

By choosing an appropriate value for y, we are able to make K(exp yQg0)F agree with F ′ onRP and hence on all of g. So F and F ′ are in the same coadjoint orbit and we are done.

21

References

[1] L. J. Corwin, F. P. Greenleaf, Representations of nilpotent Lie groups and their appli-cations, Part I: Basic theory and examples, Cambridge University Press, Cambridge,1990

[2] Brian C. Hall, Lie Groups, Lie Algebras and Representations, Springer (India), NewDelhi, 2006

[3] A. A. Kirillov, Lectures on the Orbit Method, American Mathematical Society, RhodeIsland, 2004

[4] A. W. Knapp, Lie Groups Beyond a Introduction 2nd Ed., Birkhauser, Boston, 2002

[5] D. A. Vogan, Jr., The Method of Coadjoint Orbits for Real Reductive Groups, IAS/ParkCity Mathematics Series, Volume 8, 2000.

22