Math 307: Applied Linear Algebra Lecture Notesharsyram/AppliedLinearAlgebraWorkbook2018_newest.pdf\I...

$: Math 307: Applied Linear Algebra Lecture Notesharsyram/AppliedLinearAlgebraWorkbook2018_newest.pdf\I believe that linear algebra is the most important subject in college mathematics.$
Math 307: Applied Linear Algebra Lecture Notes

Created by: Dr. Amanda Harsy

February 12, 2018

1

.

2

Contents

1 Syllabus and Schedule 7

2 Solving Systems of Linear Equations 202.1 MA 307 –ICE 0 -Motivation for Linear Algebra . . . . . . . . . . . . . . . . 202.2 Linear Systems and Matrix Notation . . . . . . . . . . . . . . . . . . . . . . 222.3 Echelon Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.4 Gauss’s Method for Solving Systems of Linear Equations . . . . . . . . . . . 252.5 ICE 1: Solving Systems of Equations . . . . . . . . . . . . . . . . . . . . . . 30

3 March MATHness: Using Linear Algebra in Sports Ranking 343.1 Directed Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.2 The Colley Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363.3 Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383.4 The Massey System of Ranking . . . . . . . . . . . . . . . . . . . . . . . . . 403.5 Adding Weights to Colley and Massey Systems . . . . . . . . . . . . . . . . . 41

4 Radiography and Tomography Lab 1 444.1 Introduction to Radiography and Tomography . . . . . . . . . . . . . . . . 444.2 Lab 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5 Vector Spaces 525.1 Review of Lab 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 525.2 Vectors and Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

5.2.1 Common Vector Space Example 1: Rn: . . . . . . . . . . . . . . . . . 545.2.2 Common Vector Space Example 2: Matrices Mm×n: . . . . . . . . . . 565.2.3 Common Vector Space Example 3: Polynomial Spaces Pn: . . . . . . 565.2.4 Common Vector Space Example 4: Function Spaces F : . . . . . . . . 575.2.5 Common Vector Space Example 5: Space of Greyscale Images: . . . . 585.2.6 Common Vector Space Example 6: Homogeneous System of Equations: 595.2.7 Vector Space Example 7: Heat States in Diffusions and other models: 605.2.8 Vector Space Example 8: Galois Field 2: GF (2)n: . . . . . . . . . . . 60

5.3 Span and Subspace Motivation . . . . . . . . . . . . . . . . . . . . . . . . . 625.4 Span . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

5.4.1 ICE 2a: Span: Part 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . 665.5 Spanning Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

5.5.1 ICE 2b: Span: Part 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . 745.6 Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

5.6.1 ICE 2c: Span: Part 3 -Subspaces . . . . . . . . . . . . . . . . . . . . 825.7 Linear Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 845.8 Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

5.8.1 ICE 3: Basis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

3

6 Radiography and Tomography Lab 2: 966.1 Introduction to Lab 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

6.1.1 Goals from the Reading Assignment: . . . . . . . . . . . . . . . . . . 966.1.2 Questions from the Reading Assignment: . . . . . . . . . . . . . . . . 966.1.3 Summary of Terminology from the Reading Assignment: . . . . . . . 107

6.2 Radiography and Tomography Lab 2 . . . . . . . . . . . . . . . . . . . . . . 108

7 Linear Transformations 1187.1 Motivation for Linear Transformations . . . . . . . . . . . . . . . . . . . . . 1187.2 Introduction to Linear Transformations . . . . . . . . . . . . . . . . . . . . . 120

7.2.1 Motivation for Matrix Forms of Linear Transformations: . . . . . . . 1227.2.2 ICE 4 -Linear Transformations . . . . . . . . . . . . . . . . . . . . . 124

8 Coordinate Vectors 128

9 Matrix Forms of Linear Transformations: 1349.0.1 Radiographic Connection . . . . . . . . . . . . . . . . . . . . . . . . . 1399.0.2 ICE 5 -Matrix Linear Transformations . . . . . . . . . . . . . . . . . 140

10 Properties of Transformations 14210.1 Injective and Surjective Transformations . . . . . . . . . . . . . . . . . . . . 142

10.1.1 ICE 5 -Injective and Surjective Linear Transformations . . . . . . . . 14810.2 Bijections and Isomorphisms . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

11 Radiography and Tomography Lab 3 15411.1 Radiography and Tomography Intro to Lab 3 . . . . . . . . . . . . . . . . . 154

12 Transformation Spaces 16412.1 Nullspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

12.1.1 Nullspace of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . 16612.2 Range Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

12.2.1 Column/Range Space for Matrix . . . . . . . . . . . . . . . . . . . . 16812.3 Injectivity and Surjectivity Revisited . . . . . . . . . . . . . . . . . . . . . . 17012.4 The Rank-Nullity Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

12.4.1 ICE 7: Column Space, Row Space, Nullspace . . . . . . . . . . . . . . 176

13 Inverse Transformations 17813.1 Connection to Tomography . . . . . . . . . . . . . . . . . . . . . . . . . . . 17813.2 The Inverse Matrix, when m = n . . . . . . . . . . . . . . . . . . . . . . . . 178

14 The Determinant of a Matrix 18214.1 The Invertible Matrix Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 186

4

15 Eigenvalues and Eigenvectors 18815.1 Eigenvalue and Eigenvectors Motivation . . . . . . . . . . . . . . . . . . . . 188

15.1.1 ICE 8: Motivation for Eigenvalues and Vectors . . . . . . . . . . . . . 18915.2 Motivating Application for Eigenvalues and Eigenvectors . . . . . . . . . . . 192

15.2.1 Review of Heat States in a (Vector) State Space . . . . . . . . . . . . 19215.2.2 The Heat Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19415.2.3 The Discrete Heat Equation, in x . . . . . . . . . . . . . . . . . . . . 19415.2.4 The Discrete Heat Equation, in t . . . . . . . . . . . . . . . . . . . . 19515.2.5 Heat State Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

15.3 Eigenvalue and Eigenvectors Motivation Continued . . . . . . . . . . . . . . 20015.4 Eigenvalue and Eigenvectors Definition . . . . . . . . . . . . . . . . . . . . . 20115.5 ICE 9: Eigenvalue and Eigenvectors . . . . . . . . . . . . . . . . . . . . . . . 20615.6 Eigenbasis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208

16 Diagonalization 21216.1 ICE 10 -Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218

17 Markov Chains 22017.1 Introduction to Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . 22117.2 Steady State Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22417.3 Application to Google’s Page Rank Model . . . . . . . . . . . . . . . . . . . 22517.4 Application to Board Games . . . . . . . . . . . . . . . . . . . . . . . . . . . 22717.5 Ice 11 -Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230

18 Orthogonality 23218.1 Orthogonal Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23218.2 Orthogonal Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23318.3 Orthogonal Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23518.4 Projections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23818.5 The Gram-Schmidt Process for finding an Orthogonal Basis . . . . . . . . . 23918.6 Column Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24218.7 QR Factorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24318.8 Usefulness of QR Factorization . . . . . . . . . . . . . . . . . . . . . . . . . 245

19 Singular Value Decomposition 24619.1 Spectral Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24619.2 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . 24719.3 Applications of SVD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25219.4 Approximating Matrix Rank . . . . . . . . . . . . . . . . . . . . . . . . . . . 253

20 Radiography and Tomography Lab 4 256

21 Practice Problems and Review for Exams 266

5

22 Homework 288

6

1 Syllabus and Schedule

Thanks for taking Applied Linear Algebra with me! It is one of my favorite classes to teach.You may be asking yourself (or have asked yourself), “What is Linear Algebra, and why doI have to take this class?”

“I believe that linear algebra is the most important subject in college mathematics. IsaacNewton would not agree! But he isn’t taking mathematics in the 21st century.”

-Gilbert Strang

I, and many other mathematicians, definitely agree with Strang in that Linear Algebra hasemerged as a giant in mathematics -especially now that we have computers to do what usedto be extremely tedious work. It has such a vast set of applications and touches almost everymathematical subject. Linear algebra provides “essential preparation for advanced work inthe sciences, statistics, and computing. Linear algebra also introduces students to discretemathematics, algorithmic thinking, a modicum of abstraction, moderate sophistication in no-tation, and simple proofs. Linear algebra helps students develop facility with visualization,see connections among mathematical areas, and appreciate the power of abstract thinking.”1

Linear algebra is basically the study of multivariate linear systems and transformations and,in my opinion, is at its core trying to solve Two Fundamental Problems:

1. Solving Ax = b, and

2. Diagonalizing a matrix A (AKA Eigenvalue Problems).

The first problem relates to exploiting linear methods to solve complex and dynamical sys-tems and situations. Basically, it is using one of the best problem-solving techniques mathe-maticians use, what I like to call the “Wouldn’t it be nice if...” approach. That is, real life is amess and we often have to deal with really complex functions (if we are even lucky enough tohave a function at all!) which are difficult to manage. So instead we use linear functions (linesand planes) to approximate or model the complex, real-world situation, which is much easier.

The second problem relates to simplifying our system so that we can more easily solve orapproximate systems. The fact that some systems don’t have solutions leads directly intothe mathematical field of Numerical Analysis and we will dive into some basic numericalanalysis in this course. Because so many situations in life can be modeled linearly, LinearAlgebra shows up in many topics including (but not exhaustively) “Markov chains, graphtheory, correlation coefficients, cryptology, interpolation, long-term weather prediction, theFibonacci sequence, difference equations, systems of linear differential equations, networkanalysis, linear least squares, graph theory, Leslie population models, the power method

1Schumacher, etc. 2015 2015 CUPM Curriculum Guide to Majors in the Mathematical Sciences 37,39

7

of approximating the dominant eigenvalue, linear programming, computer graphics, cod-ing theory, spectral decomposition, principal component analysis, discrete and continuousdynamical systems, iterative solutions of linear systems, image processing, and traffic flow.”1

Another reason you are taking this course (which perhaps is why anyone is required to take amathematics class) is to learn how to think abstractly through problem solving. My husbandis a computer scientist (math minor) and he told me once that he learned specific skills inhis computer science courses, but that he learned how to think and problem solve in hismathematics classes. The latter is probably more important to him now that he is in theworkforce. In particular, he says that the abstract conceptual thinking he learned in hismathematics classes helps him tackle the real world problem solving he has to do in his jobtoday.

I hope you will enjoy this semester and learn a lot! Some students struggle at first withLinear Algebra because it is one of the first math courses students take which starts to reallyexploit abstract notation and thinking. Hang in there! It just takes time to digest. Pleasemake use of my office hours and plan to work hard in this class. My classes have a high workload (as all math classes usually do!), so make sure you stay on top of your assignmentsand get help early. Remember you can also email me questions if you can’t make my officehours or make an appointment outside of office hours for help. When I am at Lewis, I usuallykeep the door open and feel free to pop in at any time. If I have something especially press-ing, I may ask you to come back at a different time, but in general, I am usually available.The HW Assignments, and Practice Problems for Exams are at the end of this course packet.

I have worked hard to create this course packet for you, but it is still a work in progress.Please be understanding of the typos I have not caught, and politely bring them to myattention so I can fix them for the next time I teach this course. I look forward to meetingyou and guiding you through the wonderful course that is Applied Linear Algebra.

Cheers,Dr. H

Acknowledgments: No math teacher is who she is without a little help. I would like tothank my own undergraduate professors from Taylor University: Dr. Ken Constantine, Dr.Matt Delong, and Dr. Jeremy Case for their wonderful example and ideas for structuringexcellent learning environments. I also want to thank Dr. Heather Moon, Dr. Marie Snipes,Dr. Tom Asaki, Dr. Tim Chartier, Dr. Scott Kaschner, the members from both the IOLAand MathVote projects for sharing some of their resources and labs from their own courses.And finally, I would like to thank you and all the other student for making this job worthwhileand for all the suggestions and encouragement you have given me over the years to improve.

8

MATH-307-01 & MATH-307-02 APPLIED LINEAR ALGEBRASpring 2018 SYLLABUS

Instructor: Amanda Harsy Office: AS-120-AEmail: [email protected] Office phone: 815-836-5688Class Time Section 1: TR 12:30-1:45 Class Room Section 1: AAS-157-AClass Time Section 2: TR 2:00-3:15 Class Room Section 2: AS-157-AOffice Hours: M 2-3, T 10-11, W 1-2, R 11:30-12:30, F 12:30-1:30, or by appt

Office hours are subject to change, once I find out meeting times for my committees, but I will let you know if they change. Remember you

can ALWAYS make an appointment.

Course description

This is a 3 credit Linear Algebra course which focuses on the applications of Linear Algebra. The courseconsists of the study of the study of matrices and matrix algebra, systems of linear equations, determinants,and vector spaces with a focus on applications. Topics include LU-decomposition, inner products, orthog-onality, the Gram-Schmidt process, and eigenvalue problems. Applications include differential equations,Markov processes, and problems from computer science.

About the Course

Suggested Texts:

• Linear Algebra and Its Applications, any edition By David Lay. (ISBN: 9780321989925) MyMath-Lab is NOT needed.• When Life is Linear by Tim Chartier, *Note: The electronic text is cheaper http://www.maa.org/press/ebooks/when-life-is-linear-from-computer-graphics-to-bracketology Print ISBN:9780883856499, Electronic ISBN: 9780883859889• If you are a programmer who uses Python a lot, I highly recommend Coding the Matrix by Philip

N. Klein, ISBN-10: 0615880991

Free Bonus Texts: Linear Algebra by Jim Hefferon http://joshua.smcvt.edu/linearalgebra/ andA First Course in Linear Algebra by Robert Beezer http://linear.ups.edu/html/fcla.html

Prerequisites: Calculus II (13-201) or Applied Calculus (13-240).

Course Objectives:

To help each student

• understand and use concepts of Linear Algebra,• apply Linear Algebra concepts to various applications,• become better at solving mathematical problems, experience doing mathematics cooperatively,• effectively use technology, in particular, Octave and MatLab to study mathematics,• effectively communicate mathematics, and• enjoy his/her experience with mathematics.

1

9

2 MATH-307-01 & MATH-307-02 APPLIED LINEAR ALGEBRA SPRING 2018 SYLLABUS

Resources

Blackboard: Check the Blackboard website regularly (at least twice a week) during the semester forimportant information, announcements, and resources. It is also where you will find the course discussionboard. Also, check your Lewis email account every day. I will use email as my primary method of com-munication outside of office hours.

Help: Don’t wait to get help. Visit me during my office hours, use the discussion forum in Blackboard,go to the Math Study Tables, find a study partner, get a tutor!

Dr. Harsy’s web page: For information on undergraduate research opportunities, about the LewisMath Major, or about the process to get a Dr. Harsy letter of recommendation, please visit my website:http://www.cs.lewisu.edu/~harsyram.

Course Requirements And Grading Policy

Turn In Homework: Almost every week, I will collect a homework assignment. I will post thesehomework assignments on Blackboard. You may work with others on the homework, but it must be yourown work. If I catch you copying homework, you will get a 0. Please see the academic honesty sectionbelow. Sometimes the homework include a “warm-up” assignment to either supplement what was learnedin class or to introduce what will be covered in the following class.

WebWork HW: The weekly Webwork homework will count towards your homework grade. Mostassignments will due Tuesdays and Fridays.

Labs: One of the ways in which you will explore the utility of linear algebra is through labs. These willbe partner (possibly individual) projects which will allow students to go through a hand-on exploration ofRadiography, the Heat Equation, and other topics.

Discussion Board Participation: Mathematics is not learned in isolation. You must communicatewith others about its use how to solve problems. Work together. One way we will all work together in thisclass is by using a “Discussion Board in Blackboard. Whenever you have a question about a problem, or ifsomething in the textbook is not quite clear, you should post the question into an appropriate discussionforum. This allows everybody to get familiar with the other members of the class, and it allows me toanswer a single question once and for all. You will get a small amount of extra credit towards participation.You can get points several difference ways. One way is to ask a question from the homework or class andexplaining what steps you took towards solving the problem and where you got stuck. Another way is tohelp answer another student’s question or a question that I have posted. I may respond by attaching anecho pen pdf. You can also write your question and take a picture of it or scan it and post it on the forum.Blackboard has a nice editor you can use by clicking the down arrows on the right side of the text box.This will then show more options for writing in the forum. You can then click the fx symbol to pull upthe math editor.

Take-Home Exams: There will be Three (3) take-home exams given this semester one midterm andone due at the end of finals week. You will be able to use your notes, calculator, and Octave/MatLab.You may not use any person on this exam. If I catch you working with others or using the Internet inap-propriately, you will get a 0. Please see the academic honesty section below.

Mastery-Based Testing: In addition to three take-home exams, this course will use a testing methodcalled, “Mastery-Based Testing.” There will be three (3) paper-and-pencil, in-class Mastery Exams givenperiodically throughout the semester. In mastery-based testing, students receive credit only when theydisplay ”mastery”, but they receive multiple attempts to do so. The primary source of extra attemptscomes from the fact that test questions appear on every subsequent test. In this Applied Linear Algebra

10

MATH-307-01 & MATH-307-02 APPLIED LINEAR ALGEBRA Spring 2018 SYLLABUS 3

course, Test 1 will have 7 questions. Test 2 will have 12 questions -a remixed version of the questionsfrom Test 1 and new questions on the new concepts. Test 3 will have 16 questions- a remixed version ofthe questions from Test 2 and four new questions. We will also have three testing weeks. During theseweeks, students can use doodle to sign up to retest concepts during (extended) office hours and math studytables. Students are allowed to test any concept, but cannot retest that concept the rest of the week. Sofor example, a student can test concepts 2,3 and 5 on Monday and concept 6 on Tuesday, but would notbe able to test concept 5 again. These testing weeks are tentatively set for 3/19-3/23, 4/23-27, and duringFinals week. Students will be able to retest the new concepts from the last exam twice during the finaltesting week.

Note: The first Mastery concept: Gaussian Elimination will be given outside of class during office hoursor math study tables. No technology can be used for this concept except a scientific calculator (and hencewhy it will be separated from the rest of the concepts).

Grading of Mastery-Based Tests: The objectives of this course can be broken down into 18 mainconcepts/problems. For each sort of problem on the exam, I identify three levels of performance: masterlevel, journeyman level, and apprentice level. I will record how well the student does on each problem(an M for master level, a J for journeyman level, a 0 for apprentice level) on each exam. After the Finaltesting week, I will make a record of the highest level of performance the student has made on each sortof problem or project and use this record to determine the student’s total exam grade. Each of the first8 concept/questions students master will count 9% points towards their exam grade. After that, eachconcept/question will be worth 3.5% towards your exam grade. So for example, if you master 10 of the 15concepts your grade will be a 79%.

This particular way of arriving at the course grade is unusual. It has some advantages. Each of you willget several chances to display mastery of almost all the problems. Once you have displayed mastery of aproblem there is no need to do problems like it on later exams. So it can certainly happen that if you dowell on the midterms you might only have to do one or two problems on the Final. (A few students maynot even have to take the final.) On the other hand, because earlier weak performances are not averagedin, students who come into the Final on shaky ground can still manage to get a respectable grade for thecourse. This method of grading also stresses working out the problems in a completely correct way, sinceaccumulating a lot of journeyman level performances only results in a journeyman level performance. So itpays to do one problem carefully and correctly as opposed to trying get four problems partially correctly.Finally, this method of grading allows you to see easily which parts of the course you are doing well with,and which parts deserve more attention. The primary disadvantage of this grading scheme is that it iscomplicated. At any time, if you are uncertain about how you are doing in the class I would be more thanglad to clarify the matter. The tentative test dates are below.

Exam # Date1 Thursday, 3/12 Thursday, 4/123 Tuesday, 5/2

Grade scale90%-92% A- 93%-96% A > 97% A+80%-82% B- 83%-86% B 87%-89% B+70%-72% C- 73%-76% C 77%-79% C+60%-62% D- 63%-66% D 67%-69% D+

< 60% F

11


Category PercentageMastery Exams 50

Take-Home Exams 10Paper Homework 20

Online HW 10Labs 10

Course Policies and Procedures

Calculator/Technology/Phone Policy: No phones or other forms of technology can be used onin-class exams unless specifically noted. A basic scientific or graphing calculator is ok in Applied Linear atall times (except during the Mastery Concept 1 and Homework 1). Please do not have your phones out un-less it is a class activity. If I see a phone out during a quiz or exam, you will receive an F on that assessment.

Make-Ups: There will be no make-ups for any assignments. If you are late or miss class, your assign-ment will not be accepted and there will be no make-up offered, except in extenuating and unpredictablecircumstances. If you will miss class for a justifiable and unavoidable reason, you can contact me beforeyou miss class and it is possible you can have a make-up. If you do not contact me and explain yourabsence, you will not be allowed a make-up.

Class Attendance: Students are expected to attend all classes as part of the normal learning process.In addition, students must be especially consistent in attendance, both on-ground and online, during thefirst two weeks of the semester to confirm registration and to be listed on the official course roster. Studentswho fail to follow this procedure and who have not received prior approval from the instructor for absenceswill be withdrawn from the courses in question by certification of the instructor on the official class lists.Instructors may publish specific, additional reasonable standards of attendance for their classes in thecourse syllabus. Students may receive failing grades if they do not observe attendance requirements. TheIllinois Student Assistance Commission also requires attendance as a demonstration of academic progresstoward a degree as one criterion for retaining financial aid awards. (2015-2016 Undergraduate Catalog, p.34).

Attendance is critical for success in this course and is required of all students without exception. A studentabsent from class is responsible for all material covered that day. I usually post solutions to ICE sheets andpost notes. I will stop doing this if there is a trend of absences. In other words, I will not post lecture notesunless there is consistent attendance. Students are expected to attend all classes as part of the normallearning process.

Other Information

Course Relationship to Mission: Lewis University is a Catholic University in the Lasallian Tra-dition. Our Mission is integrated into all aspects of University life, including this course. This courseembraces the Mission of the University by fostering an environment in which each student is respected asan individual within a community of learners. In the spirit of the vision of Lewis University, the goals andobjectives of this course seek to prepare students to be successful, life-long learners who are intellectuallyengaged, ethically grounded, socially responsible, and globally aware.

Academic Honesty: Scholastic integrity lies at the heart of Lewis University. Plagiarism, collusionand other forms of cheating or scholastic dishonesty are incompatible with the principles of the University.Students engaging in such activities are subject to loss of credit and expulsion from the University. Casesinvolving academic dishonesty are initially considered and determined at the instructor level. If the studentis not satisfied with the instructors explanation, the student may appeal at the department/program level.Appeal of the department /program decision must be made to the Dean of the college/school. The Dean

12


reviews the appeal and makes the final decision in all cases except those in which suspension or expulsionis recommended, and in these cases the Provost makes the final decision.

Classroom Decorum: In order to maintain an environment conducive to learning and student develop-ment, it is expected that classroom discourse is respectful and non-disruptive. The primary responsibilityfor managing the classroom environment rests with the faculty. Students who engage in any prohibited orunlawful acts that result in disruption of a class may be directed by the faculty member to leave class forthe remainder of the class period. Students considered to be a disruption or who present a threat of poten-tial harm to self or others may be referred for action to the Dean of Student Services. 2015-2016 StudentHandbook, p. 14, Lewis University website https://www.lewisu.edu/sdl/pdf/studenthandbook.pdf.

Sanctified Zone: This learning space is an extension of Lewis Universitys Sanctified Zone, a placewhere people are committed to working to end racism, bias and prejudice by valuing diversity in a safeand nurturing environment. This active promotion of diversity and the opposition to all forms of prej-udice and bias are a powerful and healing expression of our desire to be Signum Fidei, Signs of Faith,in accordance with the Lewis Mission Statement. To learn more about the Sanctified Zone, please visit:http://www.lewisu.edu/sanctifiedzone.

Students Requiring Special Accommodations: Lewis University is committed to providing equalaccess and opportunity for participation in all programs, services and activities. If you are a student witha disability who would like to request a reasonable accommodation, please speak with the Learning AccessCoordinator at the Center for Academic Success and Enrichment. Please make an appointment by calling815-836-5593 or emailing [email protected]. For more information about academic support ser-vices, visit the website at: www.lewisu.edu/CASE. Since accommodations require early planning and arenot provided retroactively, it is recommended that you make your request prior to or during the first weekof class. It is not necessary to disclose the nature of your disability to your instructor.

Applied Linear Algebra Student Learning Outcomes:

Upon successful completion of this course, students should be able to

(1) Solve systems of linear equations.(2) Analyze vectors in Rn geometrically and algebraically.(3) Recognize the concepts of the terms span, linear independence, basis, and dimension, and apply

these concepts to various vector spaces and subspaces.(4) Use matrix algebra and the related matrices(5) Use and determine formulas and representations of linear transformations.(6) Compute and use determinants.(7) Compute and use eigenvectors and eigenvalues.(8) Apply Linear Algebra techniques like Least Squares, Markov Processes, and Cramer’s Rule to

complete Data-Driven Applications.(9) Construct and use diagonalization, QR factorization, and Singular Value Decomposition to solve

systems of equations and other application problems.(10) Determine and use orthogonality.(11) Use technological tools such as computer algebra systems and graphing calculators for visualization

and calculation of linear algebra concepts.

Additional topics: These are at the instructor’s discretion as time allows. Some suggested additionaltopics are using tools such as computer algebra systems or graphing calculators for visualization and calcu-lation of linear algebra concepts and incorporating linear algebra application based labs (e.g. radiography,computer graphics, differential equations).

13


Mathematics Program Learning Outcomes:

A Lewis University Mathematics Graduate will be able to:

(1) Model the essential components of a problem mathematically so that it can be solved throughmathematical means.

(2) Solve problems through the appropriate technology.(3) Write clear and concise formal proofs of mathematical theorems, and evaluate the validity of proofs

written by others.(4) Solve mathematics problems as a member of a group and in front of a group.(5) Describe and leverage the interdependency of different areas of mathematics, the connections be-

tween mathematics and other disciplines, and the historical context for the development of math-ematical ideas.

Of these, this course addresses program learning outcomes 1, 2, 4, and 5, all at a reinforced level.

Computer Science Program Learning Outcomes:

A Lewis University Computer Science Graduate will be able to:

(1) Develop efficient programs using languages of various programming paradigms and for a variety ofplatforms.

(2) Explain how computer hardware and software is organized, implemented, and securely intercon-nected.

(3) Work in a team to design and implement complex data processing systems collaboratively.(4) Make use of mathematical structures and formulations to express theoretical ideas in computer

science and solve problems.(5) Provide examples of security and privacy challenges related to developing computing systems.

Of these, this course addresses program learning outcomes 4 at an introductory level.

Lewis University Baccalaureate Characteristics:

A Lewis University undergraduate education is marked by the following characteristics:

(1) Essential Skills: The baccalaureate graduate of Lewis University will read, write, speak, calculate,and use technology at a demonstrated level of proficiency.

(2) Major Approaches to Knowledge: The baccalaureate graduate of Lewis University will under-stand the major approaches to knowledge.

(3) Faith, Religion and Spirituality: The baccalaureate graduate of Lewis University will under-stand the place of faith, religion, and spirituality in the search for truth and meaning.

(4) Moral and Ethical Decision-Making: The baccalaureate graduate of Lewis University willunderstand and prepare for moral and ethical decision-making.

(5) Responsible Citizenship: The baccalaureate graduate of Lewis University will become an in-formed, involved, and responsible citizen of a diverse yet interconnected national and global com-munity through a grounding in economic, political, social, and historical influences that are inherentin shaping, developing, and advancing nations and the world.

(6) Critical Thinking: The baccalaureate graduate of Lewis University will think critically andcreatively.

(7) Lifelong Learning: The baccalaureate graduate of Lewis University will possess the knowledge,skills, and dispositions to enter or advance a career, or to begin graduate study.

This course addresses Baccalaureate Characteristics 1, 6, and 7 all at a reinforced level. It providesstudents essential skills in mathematics, abstraction, and algorithmic thinking. It reinforces quantitativeand empirical approaches to knowledge. In asking students to reduce problems to their important compo-nents and charting a path for solving them, it fosters critical thinking. Finally, in casting mathematics asa dynamic and evolving influencer of our collective future, it encourages curiosity and lifelong learning.

14


Class Notation

• R represents all real numbers (so no imaginary numbers)• Z represents integer numbers• Q represents rational numbers [numbers that can be written as a fraction p

qwhere p,q are integers]

• { } represents a set. For example {x; such that x is an integer }=Z• ∈ means “contained in”. For example, 5 ∈ <• /∈ means “not contained in”. For example π /∈ Z.• s.t. means “such that”• “|” or “:” mean “such that” in a set setting. For example {x|x ∈ Z}= {x : x ∈ Z} ={x; such that x is an integer } = Z• ••• means “therefore”.• ∃ means “there exists”• ∀ means “for all”• ⇒ means “this implies”• ⇐⇒ or “iff” means “if and only if”• Pf means “Proof”

This, That, Tips, and Suggestions...

Here are some suggestions that may help you in the class.

Situation: I am a student that learn best from writing everything down and I wish the lecture notes weren’t sotyped out.Suggestion: Writing is the best way for me to remember things too! I want to encourage you to write if it helpsyou. Using lecture notes keep me somewhat organized and helps us get through all of the material (of which thereis a ton)! Also, my handwriting is horrible so having some words typed is good for you. My suggestion is torewrite your lecture notes into a notebook. Rewriting notes is a wonderful study tool and I encourage you to dothis if you don’t think the partially filled out notes allow you to learn best.

Situation: I can’t see the document camera.Suggestion: I am used to higher quality document cameras and am waiting to get a better one. I have horribleboard handwriting so I don’t like to write on the board for the majority of class. I do write on the board sometimes.Also I like the fact that the document camera allows me to face the class, so hopefully I can keep tabs on how theclass is doing. My suggestion for you is to move to a different seat if you can’t see the board. And also please letme know if you are having issues and I can try to refocus/adjust the document camera.

Practice, Practice, Practice: The most common suggestion from past students is to do ALL the homework.

Learning mathematics is like any activity and requires that you practice in order to become proficient.

15

16

Applied Linear Algebra Schedule Spring 2018 This schedule is subject to change, but you will be notified of any changes ahead of time.

Agenda Special Notes

Week 1: 1/16 Systems of Equations, Matrix Multiplication

Week 2: 1/23 March Madness Application,

R/T Lab 1

Can test and retest Concept 1

Week 3: 1/30 Vector spaces and subspaces


Week 4: 2/6 Span and Spanning Sets

Linear Independence


Week 5: 2/13 Basis

R/T Lab 2 reading, R/T Lab 2


Week 6: 2/20 Linear Transformations,

Coordinate Vectors


Week 7: 2/27 Matrix Rep of Transformations

Mastery Exam 1,

Mastery Exam 1 on Thursday

Spring Break: 3/6 No class this week!

Remember to fill out your brackets by 3/11 unless you are an NCAA athlete.

Week 8: 3/13 Properties of Transformations

RT Lab 3

Week 9: 3/20 Transformation Spaces

Rank Nullity theorem Retesting Week 1

Retesting Week 1

Week 10: 3/27 Inverse Matrix Theorem,

Determinants

Week 11: 4/3 Eigenvalues, Eigenvectors, Eigenbasis

Heat Equation Application

Week 12: 4/10 Diagonalization

Mastery Exam 2

Mastery Exam 2 on Thursday

Week 13: 4/17 Markov Chains Orthogonality

No class on Monday due to Easter Break

Week 14: 4/24 Singular Value Decomposition

Retesting week 2

Singular Value Decomposition

Retesting week 2

Week 15: 5/1 RT Lab 4 (if time) Mastery Exam 3

Mastery Exam 3 on Tuesday

Week 16: 5/8 Finals/Take-Home Exam 2

Final Testing week, Take-Home Exam due Friday 2pm.

17

Assessment and Mapping of Student Learning Objectives:

Course Student Learning Objectives Math

Program

Computer Science

Program

Baccalaureate

Characteristic

Demonstrated by

1. Solve systems of linear equations. 1,2,4, 5

Reinforced

4

Reinforced 1, 6

Reinforced

Homework, Exams, Labs

2. Analyze vectors in Rn geometrically

and algebraically.

1,2,4, 5

Reinforced

4

Reinforced

1, 6

Reinforced


3. Recognize the concepts of the terms

span, linear independence, basis,

and dimension, and apply these

concepts to various vector spaces

and subspaces.

1,2,4, 5

Reinforced

4

Reinforced

1, 6

Reinforced


4. Use matrix algebra and the related

matrices

1,2,4, 5

Reinforced

4

Reinforced 1, 6

Reinforced


5. Use and determine formulas and

representations of linear

transformations.

1,2,4, 5

Reinforced

4

Reinforced

1, 6, 7

Reinforced


6. Compute and use determinants. 1,2, 5

Reinforced

4

Reinforced 1, 6

Reinforced Homework, Exams,

7. Compute and use eigenvectors and

eigenvalues.

1,2, 5

Reinforced

4

Reinforced

1, 6, 7

Reinforced Homework, Exams

8. Apply Linear Algebra techniques

like Least Squares, Markov

Processes, and Cramer's Rule to

complete Data-Driven Applications.

1,2,4, 5

Reinforced

4

Reinforced 1, 6, 7

Reinforced Homework, Exams, Labs

9. Apply Linear Algebra techniques to

complete data-driven applications.

1,2,4, 5

Reinforced

4

Reinforced

1, 6, 7

Reinforced Homework, Labs

10. Determine and use orthogonality. 1,2, 5

Reinforced

4

Reinforced 1, 6

Reinforced Homework, Exams,

11. Use technological tools such as

computer algebra systems and

graphing calculators for visualization

and calculation of linear algebra

concepts.

1,2,4, 5

Reinforced

4

Reinforced

1, 6

Reinforced Homework, Labs

Notes the labs in this course are done in groups of 2-3 which reinforces the Math Program Outcome 4.

18

2 Solving Systems of Linear Equations

2.1 MA 307 –ICE 0 -Motivation for Linear Algebra

1. Think of a number between 1 and 20. Double the number and then add 10. Nextdivide by 2 and then subtract your original number. What number did you get?

2. What is linear algebra used for?

In algebra class, we learned how to solve systems of equations. Let’s review some of these!

4. A system of linear equations could not have exactly solutions.(a) 0(b) 1(c) 2(d) infinite(e) All of these are possible numbers of solutions to a system of linear equations.

5. What is the solution to the following system of equations?2x+ y = 33x− y = 7

(a) x = 4 and y = −5(b)x = 4 and y = 5(c) x = 2 and y = −1(d) There are an infinite number of solutions to this system.(e) There are no solutions to this system.

6. Create a set of linear equations (called a linear system) that has no solutions.

19

7. We have a system of three linear equations with two unknowns, as plotted in the graphbelow. How many solutions does this system have?

(a) 0(b) 1(c) 2(d) 3(e) Infinite

8. Set up and solve the the following problem: “There are three classes of grain, of which three

bundles of the first class, two of the second, and one of the third make 39 measures. Two of

the first, three of the second, and one of the third make 34 measures. And one of the first,

two of the second, and three of the third make 26 measures. How many measures of grain

are contained in one bundle of each class?”

-Jiuzhang Suanshu, a Chinese manuscript from about 200 BC. 1

1 When Life is Linear pg 3.

20

2.2 Linear Systems and Matrix Notation

During your Ice sheet, we reviewed solving simple systems of equations. This is very nicewhen we have only a few variables, but it quickly gets tedious when we have systems witheven more variables. Up until now, our method for solving these systems has been a littlead-hoc. In this section we will discuss an algorithm which will help us solve larger systemsof equations. Now, this algorithm may seem weird and tedious, and is a rather uninspiringway to start this course, but it has allowed us to program computer to do this work for us.And after HW 1, you will be able to use Octave, Matlab, Sage, or a calculator to do this foryou. Let’s get started...

Definition 2.1. A linear combination of variables (unknowns) x1, ..., xn is an expression inthe form:

where the a′is (a1, a2, ..., an) are real numbers which are called the combination’s

Examples:

Non-Examples:

Definition 2.2. We can turn a linear combination into a linear equation by:

Then we can create a system of linear equations:

a1,1x1 + a1,2x2 + a1,3x3 + ...+ a1,nxn = b1

a2,1x1 + a2,2x2 + a2,3x3 + ...+ a2,nxn = b2...

an,1x1 + an,2x2 + an,3x3 + ...+ an,nxn = bn

Definition 2.3. A solution to a linear system is an “n-tuple”which solves each equation in the system.

21

“Did you know that the first Matrix was designed to be a perfect human world? ” -Agent Smith, The Matrix

a1,1x1 + a1,2x2 + a1,3x3 + ...+ a1,nxn = b1

a2,1x1 + a2,2x2 + a2,3x3 + ...+ a2,nxn = b2...

an,1x1 + an,2x2 + an,3x3 + ...+ an,nxn = bn

We would like to be able to record a linear system compactly. We can do this by usingMatrices:

The Coefficient Matrix:

A = (aij) =

a1,1 a1,2 a1,3 . . . a1,n

a2,1 a2,2 a2,3 . . . a2,n...

......

......

am,1 am,2 am,3 . . . am,n

The Augmented Matrix:

(A b

)=

a1,1 a1,2 a1,3 . . . a1,n b1

a2,1 a2,2 a2,3 . . . a2,n b2...

......

......

...am,1 am,2 am,3 . . . am,n bn

Example 2.1. Write the following system as an augmented matrix:x1 − 3x2 + 5x3 − 2x4 = 0x2 − 3x3 = 29x3 − 4 = 0x4 = 1

Definition 2.4. An m × n matrix is a rectangular ordered arrangement (an array) ofnumbers with rows and columns.

Examples:

A =

[2 7 90 1 5

]=

(2 7 90 1 5

), B =

[1 54 1

], C =

12030

, D =

[2 4 6

].

22

Example 2.2. a) Convert the following augmented matrix back into a system of equations:

2 1 0 00 3 0 61 −1 1 0

b) When you solve this system, you get x = −1, y = 2, z = −1. Is this an equivalent form ofthe sytem of equations above?

c) Write this system in augmented form.

Wouldn’t it have been nice if we were given the latter system to solve rather than the initialsystem? It turns out we have a name for the “nicest form” of an augmented matrix.

That is we want an augmented matrix that has the maximum number of zeros, but doesn’tchange the original system’s solution. It turns out there is a an algorithm we can use toconvert a system into this “nice form.” But first, let’s describe what we mean by “nice” alittle more formally.

2.3 Echelon Forms

Definition 2.5. The leading term of a row is the leftmost nonzero term in that row. If arow has all zeros, it has no leading term.

Definition 2.6. A Matrix/system is in echelon form if

1. Every leading term is in a column to the left of the leading term of the row below it.

2. Any zero rows are at the bottom of the matrix.

Examples:

1 2 3 50 4 1 −20 0 3 1

,

1 2 0 50 5 1 00 0 0 00 0 0 0

,

0 1 50 0 10 0 0

,

1 0 00 1 00 0 1

,

0 1 5 00 0 1 00 0 0 10 0 0 0

Non-examples:

0 1 51 0 00 1 0

1 0 0 52 1 1 −20 0 0 0

,

1 0 50 0 00 1 0

,

2 0 40 1 00 0 0

23

Definition 2.7. In an echelon form system, a pivot position is an entry that correspondsto a leading term in the echelon form of A. Variables that correspond to a pivot position arecalled pivot variables.

Definition 2.8. In an echelon form system, the variables that are not leading are calledvariables.

Example 2.3. Which variable(s) are pivots? Which variable(s) are free for the followingmatrix representation with variables x1, x2, x3, x4?

2 4 0 4 00 9 18 3 00 0 1 1 0

.

Definition 2.9. A Matrix/system is in reduced row echelon form if

1. It is in echelon form.

2. All pivot positions contain a 1.

3. The only nonzero term in a pivot column is in pivot position.

Examples:[1 0 0 50 1 1 2

],

0 1 0 −2 0 00 0 1 −6 0 30 0 0 0 0 0

,

1 0 00 1 00 0 1

,

0 0 00 0 00 0 0

Example 2.4. Which of the following matrices are in Reduced Row Echelon form?

A =

[5 0 0 10 1 0 2

]B =

1 0 0 10 0 1 00 0 5 2

C =

1 2 0 10 0 1 00 0 0 0

2.4 Gauss’s Method for Solving Systems of Linear Equations

It would be nice if we were given a system whose matrix form was in echelon form or evenbetter, reduced row echelon form (RREF) (sometimes this form is called row reduced ech-elon form). In algebra courses, you often solve a system of 2 (maybe 3) equations usingsubstitution or elimination. Unfortunately, this can get tricky when we have large systemsthough. Luckily we have an algorithm to help us calledGaussian Elimination.

24

Gaussian Elimination/ Gauss’s Method: If a linear system is changed to another byone of the operations below, then the two systems have the same set of solutions (that is,they are equivalent systems).

1. Swapping: Swap one row (equation) with another.

2. Rescaling: Multiply one row (equation) by a nonzero constant.

3. Row Combination: Replace one row (equation) by the sum of itself and a multipleof another.

Definition 2.10. The 3 operations above are called elementary row operations. Note thatthey are reversible.

Basic Strategy:Replace one system of equations with an equivalent system that is easier to solve.

In the end, we want a matrix that is as close as it can be to looking like:

1 0 0 0 0 0 . . . 0 ∗0 1 0 0 0 0 . . . 0 ∗0 0 1 0 0 0 . . . 0 ∗...

......

......

......

......

0 0 0 0 0 0 . . . 1 ∗

Why is this form helpful?

Tie in with Echelon Forms:Notice there seems to be two “steps” in Gaussian Elimination:

Step 1: Get matrix in “Upper Triangular Form:

1 ∗ ∗ ∗ ∗ ∗ . . . ∗ ∗0 1 ∗ ∗ ∗ ∗ . . . ∗ ∗0 0 1 ∗ ∗ ∗ . . . ∗ ∗...

......

......

......

......

0 0 0 0 0 0 . . . 1 ∗

Note this can also be something like:

1 ∗ ∗ ∗ ∗ ∗ . . . ∗ ∗0 1 ∗ ∗ ∗ ∗ . . . ∗ ∗0 0 1 ∗ ∗ ∗ . . . ∗ ∗...

......

......

......

......

0 0 0 0 0 0 . . . 1 ∗0 0 0 0 0 0 . . . 0 ∗0 0 0 0 0 0 . . . 0 ∗0 0 0 0 0 0 . . . 0 ∗

Step 2: Back substitute to get the matrix in Row Reduced Reduced Echelon Form.

Note each matrix’s Row Reduced Echelon Form is unique!

25

Example 2.5. Solve the linear system represented by the augmented matrix using GaussianElimination:

6 3 −2 −40 2 −6 −81 0 2 3

What should you be asking yourself about linear systems?

1.

2.

26

We can use Guassian Elimination to identify how many solutions we will have in a system.

Example 2.6. Convert the following augmented matrices back into a system of equations:

a)

1 0 0 20 1 0 00 0 1 −3

⇒

c)

1 0 0 20 1 0 00 0 0 −3

⇒

b)

1 0 0 20 1 0 00 0 0 0

⇒

d)

1 0 1 20 1 0 00 0 0 0

⇒

Theorem 2.1. A system of equations can have solutions, exactlysolutions, or solutions.

Definition 2.11. A system is called consistent if it has or solutions.A system is called inconsistent if it has solutions.

Example 2.7. Solve the system of equations using Gaussian Elimination.

x1 + x2 + x3 = 4x1 + x2 + x3 = 0

27

Example 2.8. Solve the system of equations using Gaussian Elimination.

2x + 2y + 2z = 4−x + y − 2z = −2x + y + z = 2

28

2.5 ICE 1: Solving Systems of Equations

1. What is the solution to the following system of equations? (Note this is called a homogeneous

system since the right-hand-sides of each equation is 0.)x+ 2y + z = 0x+ 3y − 2z = 0

Archer has reduced this system into Row Reduced Echelon Form (RREF):

(1 0 7 00 1 −3 0

)

(a)

xyz

=

−731

s

(b)

xyz

=

7−31

s

(c)

xyz

=

−730

s

(d) None of the above.(e) More than one of the above.

Bonus question: How many vectors are in this solution set?

2. What is the solution to the following system of equations? (Note this system is called a

non-homogeneous system.)

x+ 2y + z = 3x+ 3y − 2z = 4

Eva has reduced this system into RREF:

(1 0 7 10 1 −3 1

)

(a)

xyz

=

730

+

111

s

(b)

xyz

=

−1−10

+

−731

s

(c)

xyz

=

110

+

−731

s


29

Bonus question: What is the relationship between the previous two solution sets?

3. What is the solution to the following system of equations (called a non-homogeneoussystem)?x+ 2y + z =−2x+ 3y − 2z = 1

Archer has reduced this system into RREF:

(1 0 7 −80 1 −3 3

)

(a)

xyz

=

−830

+

−731

s

(b)

xyz

=

8−30

+

−731

s

(c)

xyz

=

−830

+

7−3−1

s


4. Let matrix R be the reduced row echelon form of matrix A. True or False The solutionsto Rx = 0 are the same as the solutions to Ax = 0.

(a) True, and I am very confident(b) True, but I am not very confident(c) False, but I am not very confident(d) False, and I am very confident

5. Let matrix R be the reduced row echelon form of matrix A. True or False: The solu-tions to Rx = b are the same as the solutions to Ax = b.


30

6. What is the value of a so that the linear system represented by the following matrixwould have infinitely many solutions?(

2 6 81 a 4

)

(a) a = 0(b) a = 2(c) a = 3(d) This is not possible.(e) More than one of the above

7. Solve the system represented by the augmented matrix

1 7 3 −40 1 −1 30 0 0 10 0 1 −2

. [Hint: Eva

says that you can answer this without doing any work!]

8. Rewrite the linear system represented by the augmented matrix below in Row Reduced

Echelon Form to find the solution set:

1 −1 0 0 −40 1 −3 0 −70 0 1 −3 −10 0 0 1 −3

.

31

9. Solve the system from Jiuzhang Suanshu.

3x+ 2y + z = 392x+ 3y + z = 34x+ 2y + 3z = 26

32

3 March MATHness: Using Linear Algebra in Sports

Ranking

Linear Algebra can be used in sports analytics. In this section, we will discuss how we canuse linear algebra to develop sports ranking systems.

The most basic way to ranks teams is through win percentage.Recall Win Percentage=

Why would basing a team’s ranking only by win percentage not necessarily be the bestmethod for sports ranking?

On reason this may not work well is that there is no way to distinguish strength of schedule.Team A may only play and win 1 game and thus has a win percentage of whileTeam B may play 100 games and win 99 of them which would give them a win percentageof . If we based teams only by win percentage, we would rank team A over teamB. Is this fair? Similarly if team C plays and wins all 100 games, they have the same winpercentage as team A. Is it fair to give them the same ranking? How can we distinguishbetween a team that is undefeated with a difficult schedule verses an undefeated team in aweaker conference? It turns out linear algebra can help.

3.1 Directed Graphs

Before we discuss two different ranking methods, first we will talk about a way to organizesports data. Suppose we have 4 teams, Team A, B, C, & D. Team A beats B and D, Cbeats D and B, and Team D beats team B. You can represent this situation using a directedgraph with the teams making up the vertices of the graph. This graph is sometimes calledthe “The Poor Sportsmanship Graph”. Why do you suppose that is? How would you rankthe teams in this system?

A

B D

C

33

Example 3.1. How would you rank this system of teams?

A

B D

C

F

E

G

H

The two examples above are easy to rank and are called “perfect seasons.” But often wedon’t have such nice systems for example, suppose we tweak the systems so we have:

A

B D

Cor

A

B D

C

F

E

G

H

What other information would you like to know about these teams and systems?

Our goal, as sports analysts, is to determine not only who is number one, but who is numbertwo, three, four, etc. Sports ranking is similar to Google’s task of ranking pages, but is muchtougher because the outcome of games require both skill and luck.

34

3.2 The Colley Method

As part of his mathematics PhD dissertation, Dr. Wesley Colley developed a method to rank teams

in a way that incorporates strength of schedule which can make it a little more accurate than win

percentage. Here’s why. At the start of the season, a team’s winning percentage would 0/0 which

is undefined! The Colley Method says an untried team should have a rating of 50%, 50-50 chance

the team will win or lose. Colley also wants to have the average of all rating be roughly 0.5 as well.

So we adjust how we calculate our win percentage.

Regular Winning Percentage:

Colley Winning Percentage:

Since, after each game, the denominator increases by 1, the losers rating will decreasing andthe winners rating will increase after each game.

35

Example 3.2. Use the Colley method to come u with a rating for the following system.

A

D B

C

36

Example 3.3. Suppose we have the teams, A, B, and C. Team C beats A by 17 points duringthe first part of the season. During the middle of the season, Team B beats C by 10 pointand Team A beats B by 10 points. And suppose near the end of the season, Team A beats Bby 5 points. Create and find a rating using the Colley Method.

3.3 Least Squares

Now that we have explored the Colley System for Rating Teams. What were some of thestrengths and weaknesses of this method?

Suppose we want to use a system of rating that incorporates the scores of the games. TheMassey System does this. Suppose I have 3 teams, A, B, and C. Team A beats B by 10points. Team B beats C by 1 point. Team C beats A by 17 points. And suppose Team Abeats B by 5 points. Notice that this system lacks “transitivity.” We can set up a system ofequations to represent this “season.”

37

Problem: This system is inconsistent. That is, there are solutions. What canwe do to “approximate” a solution?

Idea of the Least Squares Solutions: We want to find the best approximation forsolving Ax = b.

First we need to introduce the notion of distance and transpose.

Definition 3.1. Given a matrix A =

[a b cd e f

], the transpose of A, AT =

That is, the rows of A become the columns of AT .So if A is an n×m matrix, AT is a .

Octave Code: In octave AT can be written as A′

Note: Our Colley Matrix is a Symmetric Matrix . Note AT = A.

Definition 3.2. The inner product or dot product between two vectors u,v ∈ Rn,denoted u · v, is defined as follows:

Let u =

u1

u2

u3

,v =

v1

v2

v3

, u · v = uT · v =

Example 3.4. Determine u · v, v · u, and v · v for u =

10−2

and v =

3−11

Definition 3.3. The length or norm of a vector v∈ Rn is denoted ||v|| =

Note: There are many different ways to define distances/norms. Hopefully we will havetime to discuss them later in the semester!

38

Least Squares continued: So if we want to the best approximation for solving Ax = b,we want to minimize the distance between Ax and b.Goal: minimize:

Definition 3.4. Given A ∈ Mn×m and b∈ Rn, a least-squares solution of Ax = b is ax ∈ Rm such that ||b− Ax|| ≤ ||b− Ax|| for all x ∈ Rm.

Theorem 3.1. The set of least-square solutions of Ax = b coincides with the nonempty setof solutions to the normal equation: ATAx = ATb.

Proof. Omitted (We don’t have the theory yet!).

Example 3.5. Find a least-squares solution for Ax = b where A =

2 1−2 02 3

and b =

1−10

.

3.4 The Massey System of Ranking

Recall at the beginning of these notes we were trying to solve

1 −1 00 1 −1−1 0 11 −1 0

r=

101175

.

We now can use least squares to help solve this.

39

New problem: This system has infinitely many solutions. To fix this, we replace the last rowof our new matrix with all 1’s and the last entry in the vector with 0. We won’t go into thetechnical details for why we do this, but this adjustment forces the ratings to sum to 0.We now solve the new system:

Easy Adjustment for blowouts: Suppose you want to balance out any blow-outs fromthe games. This is quite easy with the Massey Method. Suppose you want to consider anygame in which one team beats the other team by 15 points or more a blow out and countthem all the same. We would just adjust our Massey System by capping the size of the win.

1 −1 00 1 −1−1 0 11 −1 0

r =

101155

3.5 Adding Weights to Colley and Massey Systems

The Colley and Massey Rating Methods are good, but there are a lot of other things we maywant to consider like...

We can weight both of these rating system to give us a more personalized rating. The keynow is that we count total weighted games and weighted scores.

40

Example 3.6. Suppose we have the same teams, A, B, and C where Team C beats A by 17 points

during the first part of the season. During the middle of the season, Team B beats C by 10 points

and Team A beats B by 10 points. And suppose near the end of the season, Team A beats B by 5

points. Create and find the rating using a weighted Massey Method with a 10 point cap.

41

Example 3.7. Suppose we have the same teams, A, B, and C where Team C beats A by 17 points

during the first part of the season. During the middle of the season, Team B beats C by 10 points

and Team A beats B by 10 points. And suppose near the end of the season, Team A beats B by 5

points. Create and determine the weighted Colley rating by weighting the time of the game. Use

the weighting where the first part of the season is weighted by 0, the middle by 1, and the end of

the season by 1.5. Recall we already did this ranking without weights in Example 3.3.

42

4 Radiography and Tomography Lab 1

4.1 Introduction to Radiography and Tomography

Radiography is an imaging technique used to view internal structures of the body thatuses electromagnetic radiation (x-rays) by exposure of film specially sensitized to x-rays orgamma rays. Tomography is any method that produces images of single tissue planes bycomputer processing -think “CAT scan” (computerized axial tomography). This scan is di-vided into a square matrix of pixels.Why is it good: noninvasive and has high contrast resolutionSource:http://medical-dictionary.thefreedictionary.com/tomography

Go to the course’s blackboard site. Click the Labs link and download the “Source Code forLabs.” Save it to a folder you will be able to find later. Open Matlab and make sure you arein the correct folder.

Type: “radex(3)”This shows a sequence of horizontal slices through a brain. The first frame is somewherenear the level of the nose and each subsequent frame (181 in all) is higher than the previousone. At the end of the “movie, you will see a rectangular object which is a stabilizer forkeeping the subjects head from moving during the procedure.These slices are examples of desired images of the brain one might want to use to identifyor evaluate abnormalities. Of course, we cannot physically slice a persons head to get suchinformation. What we can do is take images such as what happens in a CAT scan and usethese to produce the slices.

Type “radex(1)”This shows a sequence of the CAT scan images that are actually taken in such a radiographicexam. Each view is taken from a different angle around the head. Each view is perpendicularto the horizontal slices that we just viewed.

Type “radex(2)”This shows the 30 views that were taken in the CAT scan movie. In an actual CAT scanthere are typically hundreds if not over a thousand scans taken in order to produce the brainimages that we saw originally.

Question: Why might we want to take fewer than hundreds of views?

43

http://medical-dictionary.thefreedictionary.com/tomography

Type “radex(4)”This shows a sequence of horizontal slices created by reconstructing from only the 30 radio-graphs. This reconstruction used Linear Algebra techniques and was really only a first passat getting a reconstruction. With more Linear Algebra work, one can even do better.

Question: What are the goals of a researcher in this mathematical field?

Type “radex(5)”This shows a comparison of four slices of a head. In the top row are the reconstructions usinghundreds of radiographs and the bottom row has the reconstructions using only 30 radio-graphs. Again, this is done using Linear Algebra techniques that we will learn in this class.We will also play with these techniques to get an even better reconstruction with so few scans.

44

4.2 Lab 1 Introduction

Definition 4.1. An image is a finite ordered list of real values with an associated geometricarray description.

Examples:

1

2

36

7

5

41 8

4

5

2

6

3

7

1

2

3 8

9

10

54 6 7

. . .

. . . . . .

Figure 1: Examples of image arrays. Numbers indicate example pixel ordering.

Addition:

Definition 4.2. The sum of two images is the image that results by adding correspondingvalues of the ordered list.

Definition 4.3. The sum of two images is the image that results by pixel-wise addition.

Definition 4.4. Given two images x and y with (ordered) intensity values (x1, x2, ..., xn)and (y1, y2, ..., yn), respectively, the sum, written z = x+ y is the image with intensity valueszi = for all i ∈ {1, 2, ..., n}.Scalar Multiplication:

Definition 4.5. A scalar times an image is the image which results from multiplicationof each of the intensity values (or values in the ordered list) by that scalar.

Definition 4.6. A scalar times an image is the image that results by pixel-wise scalar mul-tiplication.

Definition 4.7. Given scalar a and image x with (ordered) intensity values (x1; , x2, ..., xn),the scalar product, written z = ax is the image with intensity values zi = for alli ∈ {1, 2, ..., n}.

45

46

DRAFT: Radiography/Tomography c©2016 T. Asaki, C. Camfield, H. Moon, M. Snipes

Radiography and Tomography: Lab 1

Grayscale Images

A grayscale image can be associated with a set of numbers representing the brightness ofeach pixel. For example, the 2×2 pixel image below can be associated with the correspondingarray of numbers, where the black squares represent pixels with intensity 0 and the whitesquares represent pixels with intensity 16:

0

~

16 8

16

Note that this interpretation gives us a way to think about multiplying an image by aconstant and adding two images. For example, suppose we start with the three images (A,B, and C) below.

Image A Image B Image C

Then multiplying Image A by 0.5 results in Image 1 below. Note that the maximumintensity is now half what it previously was, so the all pixels have become darker gray(representing their lower intensity). Adding Image 1 to Image C results in Image 2 below;so Image 2 is created by doing arithmetic on Images A and C.

Image 1 Image 2

1

47


Image 3 Image 4

Exercise 1: Write Image 3 and Image 4 using arithmetic operations of Images A, B, andC.

Images in Matlab

We’ll now see how to input matrices, convert them into grayscale images, and displaythem in Matlab (or Octave). If you do not have access to Matlab or Octave through academicor personal computing resources then you may complete this Lab using the online serviceoctave-online.net (instructions at the end of this writeup).

Exercise 2 Input these lines into the command window of Matlab. Note that ending a linewith a semicolon suppresses the output. If you want to show the result of a computation,delete the semicolon at the end of its line. Briefly describe what the output of each of theselines of code gives you. Note: In order to display an image, we have to know the minimumand maximum pixel intensities. The imshow function allows the user to specify these values.

M_A = [0 0 8 8; 0 0 8 8; 8 8 0 0; 8 8 0 0];

M_B = [0 8 0 8; 8 0 8 0; 0 8 0 8; 8 0 8 0];

M_C = [8 0 0 8; 0 8 8 0; 0 8 8 0; 8 0 0 8];

figure;

subplot(1,3,1), imshow(M_A, [0, 16]),title(’Image A’);

subplot(1,3,2), imshow(M_B, [0, 16]),title(’Image B’);

subplot(1,3,3), imshow(M_C, [0, 16]),title(’Image C’);

Exercise 3: Enter the following lines of code one at a time and state what each one does.

M_A

M_1 = .5*M_A

M_2 = M_1 + M_C

figure;

subplot(1,2,1), imshow(M_1, [0, 16]), title(’Image 1’);

subplot(1,2,2), imshow(M_2, [0, 16]), title(’Image 2’);

2

48


Exercise 4 Now write your own lines of code to check your conjectures to get Images 3and 4. How close are these to Images 3 and 4? Be sure to print out your code and submitit with this lab.

Food For Thought Be prepared to discuss these in class:

1. What happens when pixel intensities in an image exceed the display range as inputinto the imshow function?

2. How should we interpret pixel intensities that lie outside our specified range?

3. What algebraic properties does the set of all images have in common with the set ofreal numbers?

Instructions for using octave-online.net

1. Open octave-online.net in a web browser. Close (or read and close) the introductionsplash window.

2. Optional: Click the three bars in the upper right of the screen and sign in. This willbe necessary if you complete the next Lab using this service.

3. The rightmost window is the command prompt terminal. You will type your commandshere. You also have the option of creating your own scripts and functions to save andrun.

4. Now you are ready to complete the assignment!

3

49

50

5 Vector Spaces

Today we are going to introduce the “Playground” that Linear Algebra exists in along witha set of usual suspects that play in this arena. If we were doing a more theoretical linearalgebra, we would do lots of proofs in this section. For our purposes, treat this mainly asa “Who’s Who” and learning about what two operations we can use in our “playground.”Seriously though, there is a lot of nice theory here which provides the underpinning for allwe do later this semester.

5.1 Review of Lab 1

In Lab 1, we noticed that if you add two images, you get a new image. We also saw thatif you multiply an image by a scalar, you get a new image. We also saw in Lab 1 that(rectangular pixelated) images can be represented as a rectangular array of values or equiva-lently as a rectangular array of grayscale patches. This is a very natural idea especially sincethe advent of digital photography. It is tempting to consider an image (or image data) as amatrix – after all, it certainly looks like one. Recall, we defined an image in the following way:

Definition 5.1. An image is a finite ordered list of real values with an associated geometricarray description.

We also defined pixel-wise addition and scalar multiplication of images.Three examples of arrays along with an index system specifying the order of patches canbe seen in Figure below. Each patch would also have a numerical value indicating thebrightness of the patch (not shown). The first is a regular pixel array commonly used fordigital photography. The second is a hexagon pattern which also nicely tiles a plane. Thethird is a square pixel set with enhanced resolution toward the center of the field of interest.The key point here is that only the first example can be written as a matrix, but all satisfythe definition of image. We found that these operations had a place in a real-life exampleand when applying these operations it is very important that our result is still an image inthe same configuration. It turns out that these operations are important enough that wegive a name to sets that have these operations with some pretty standard properties.

51

5.2 Vectors and Vector Spaces

The word vector may be familiar for many students who have taken Vector Calculus and/orPhysics. In these courses, there is a very specific type of vector used, vectors in Rm. That is,the word vector may bring to mind something that looks like 〈a, b〉, 〈a, b, c〉, or 〈a1, a2, . . . , an〉.Maybe you’ve even seen things like any of the following

(a, b), (a, b, c), (a1, a2, . . . , an),

abc

,

abc

,

a1

a2...an

,

a1

a2...an

called vectors. In this section, we first define the type of set we want to consider when talkingabout linear algebra concepts. Then, we discuss the elements of these sets called vectors.

Definition 5.2. A set V with a set of scalars and operations + and scalar multiplication· is called a vector space if the following ten properties hold. Elements of V are calledvectors. Let u, v, w ∈ V be vectors and α, β be scalars (constants).

1. V is closed under addition +: u+v ∈ V (Adding two vectors gives a vector in the set.)

2. V is closed under scalar multiplication α · u ∈ V . (Multiplying a vector by a scalargives a vector in the set.)Question: Does the set of odd real numbers satisfy the properties above? How aboutthe set of odd real numbers?

3. Addition is commutative: u+ v = v + u.

4. Addition is associative: (u+ v) + w = u+ (v + w).

5. Scalar multiplication is associative: α · (β · v) = (α · β)v.

6. Scalar multiplication distributes over vector addition: α · (u+ v) = α · u+ α · v.

7. Scalar multiplication distributes over scalar addition: (α + β) · v = α · v + β · v.

8. V contains the 0 vector, where 0 + v = v + 0 = v.Note: 0 is the additive identity.

9. V has additive inverses −v: v +−v = 0.

10. The scalar set has an identity element 1 for scalar multiplication: 1 · v = v for allv ∈ V .

52

Careful: It is important to note that the identity element for scalar multiplication need notbe the number 1 and the zero vector need not be (and in general is not) the number 0.

Notice also that elements of a vector space are called vectors. These need not look like thevectors presented above.We now present some examples of vector space and the corresponding vectors arguments.

5.2.1 Common Vector Space Example 1: Rn:

Rn: the set of points or vectors in n-dimensional space is a vector space with scalars takenfrom the set of real numbers. Here, we recognize that addition of two vectors in Rn iscomponent-wise. Here the vectors are just like the vectors we knew before Linear Algebra.

Examples:

(10

),

(12

),

(π√2

), and many more.

Example 5.1. R, the set of real numbers is a vector space with scalars taken from the setof real numbers. We define addition and multiplication as usual.

Example 5.2. We will show that R2 is a vector space and recognize how the same proofs

generalize. Let

(u1

u2

),

(v1

v2

),

(w1

w2

)∈ R2 be vectors and α, β be scalars.

• V is closed under addition +:

(u1

u2

)+

(v1

v2

)=

(u1 + v1

u2 + v2

)∈ R2.

• V is closed under scalar multiplication ·: α ·(v1

v2

)=

(αv1

αv2

)∈ R2.

• Addition is commutative:(u1

u2

)+

(v1

v2

)=

(u1 + v1

u2 + v2

)=

(v1 + u1

v2 + u2

)=

(v1

v2

)+

(u1

u2

).

• Addition is associative:((u1

u2

)+

(v1

v2

))+

(w1

w2

)=

(u1 + v1

u2 + v2

)+

(w1

w2

)=

((u1 + v1) + w1

(u2 + v2) + w2

)=

(u1 + (v1 + w1)u2 + (v2 + w2)

)=

(u1

u2

)+

(v1 + w1

v2 + w2

)=

(u1

u2

)+

((v1

v2

)+

(w1

w2

)).

• Scalar multiplication is associative:

α ·(β ·(v1

v2

))= α

(βv1

βv2

)=

(α(βv1)α(βv2)

)=

((αβ)v1

(αβ)v2

)= (αβ) ·

(v1

v2

).

53

• Scalar multiplication distributes over vector addition:

α ·((

u1

u2

)+

(v1

v2

))= α

(u1 + v1

u2 + v2

)=

(α(u1 + v1)α(u2 + v2)

)=

(αu1 + αv1

αu2 + αv2

)=

(αu1

αu2

)+

(αv1

αv2

)= α ·

(u1

u2

)+ α ·

(v1

v2

).

• Scalar multiplication distributes over scalar addition: (α+β)·(v1

v2

)=

((α + β)v1

(α + β)v2

)=

(αv1 + βv1

αv2 + βv2

)=

(αv1

αv2

)+

(βv1

βv2

)= α ·

(v1

v2

)+ β ·

(v1

v2

).

• V contains the 0 vector, where(00

)+

(v1

v2

)=

(0 + v1

0 + v2

)=

(v1 + 0v2 + 0

)=

(v1

v2

)+

(00

)=

(v1

v2

).

• V has additive inverses −v:

(v1

v2

)+

(−v1

−v2

)=

(v1 + (−v1)v2 + (−v2)

)=

(00

).

• The scalar set has an identity element 1 for scalar multiplication:

1 ·(v1

v2

)=

(1(v1)1(v2)

)=

(v1

v2

).

Key: If I want to show a set is not a vector space, I just need to show that ofthe 10 properties fails.

Example 5.3. Which property of vector spaces is not true for the following subset of R2?

V = {[−2−2

],

[−1−1

],

[00

],

[11

],

[22

]}

(a) Closure under vector addition(b) Existence of an additive identity(c) Existence of an additive inverse for each vector(d) None of the above

Question: What would I have to add to the set to make it a vector space?

54

5.2.2 Common Vector Space Example 2: Matrices Mm×n:

Example 5.4. M2×3 =

{(a b cd e f

)| a, b, c, d, e, f ∈ R

}is a vector space when addition

and scalar multiplication are defined as usual with Matrix operations and scalars are takenfrom R.(a1 b1 c1

d1 e1 f1

)+

(a2 b2 c2

d2 e2 f2

)=

α ·(a1 b1 c1

d1 e1 f1

)=

5.2.3 Common Vector Space Example 3: Polynomial Spaces Pn:Example 5.5. P2 = {ax2 + bx + c| a, b, c ∈ R} is a vector space with scalars taken from Rand addition and scalar multiplication defined in the standard way for polynomials.

55

5.2.4 Common Vector Space Example 4: Function Spaces F :

F = {f : R → R}: the set of all functions whose domain is R and whose range is a subsetof R. F is a vector space with scalars taken from R. We can define addition and scalarmultiplication in the standard way. Note: Here, the vectors are functions.

Examples vectors in this Vector Space:

How we define our two operations: Addition: f + g = (f + g)(x) = f(x) + g(x)Scalar Multiplication: αf = (αf)(x) = α · f(x)Check it is a Vector Space: Let f, g, h ∈ F and α, β ∈ R then

• f : R→ R and g : R→ R. Based on the definition of addition, f + g : R→ R. So Fis closed over addition.

• Similarly, F is closed under scalar multiplication.

• Addition is commutative: (f + g)(x) = f(x) + g(x) = g(x) + f(x) = (g + f)(x). So,f + g = g + f .

• Addition is associative: ((f + g) + h)(x) = (f + g)(x) + h(x) = (f(x) + g(x)) + h(x) =f(x)+(g(x)+h(x)) = f(x)+(g+h)(x) = (f+(g+h))(x). So (f+g)+h = f+(g+h).

• Scalar multiplication is associative: (α · (β · f))(x) = (α · (βf(x))) = (αβ)f(x) =((αβ) · f)(x). So α · (β · f) = (αβ) · f .

• Scalar multiplication distributes over vector addition: (α ·(f+g))(x) = α ·(f+g)(x) =α · (f(x) + g(x)) = α · f(x) + α · g(x) = (α · f + α · g)(x). So α(f + g) = αf + αg.

• Scalar multiplication distributes over scalar addition: ((α+β) ·f)(x) = (α+β) ·f(x) =α · f(x) + β · f(x) = (α · f + β · f)(x). So, (α + β) · f = α · f + β · f .

• F contains the constant function defined by z(x) = 0 for every x ∈ R. And, (z+f)(x) =z(x) + f(x) = 0 + f(x) = f(x) = f(x) + 0 = f(x) + z(x) = (f + z)(x). That is,z + f = f + z = f . So, the 0 vector is in F .

• F has additive inverses −f defined to as (−f)(x) = −f(x) and (f + (−f))(x) =f(x) + (−f(x)) = 0 = z(x), where z is defined in part 5.2.4. So, f + (−f) = z.

• The real number 1 satisfies: (1 · f)(x) = 1 · f(x) = f(x). So, 1 · f = f .

56

5.2.5 Common Vector Space Example 5: Space of Greyscale Images:

Notice that the set of images of a with a specified geometric arrangement is a vector spacewith scalars taken from R. That is, if we consider the set

V = {Ia| I is of the form below and a1, a2, . . . a14 ∈ R} .

Ia =a2

a1

a3

a4

a5

a6

a7

a8

a9

a10

a11

a12

a13

a14

We know this is a vector space since, by definition of addition and scalar multiplication onimages, we see that both closure properties hold. Notice that there is a corresponding realnumber to each pixel (or voxel). Because addition and scalar multiplication are taken pixel-wise (or voxel-wise), we can see that these 10 properties hold within each pixel (or voxel).So, we know all 10 properties hold (just like they did in the last example5.2).

Note: It turns out that given any geometric configuration our definitions of image andoperations on images guarantee that the space of images with the chosen configuration is avector space. The vectors are then images.

Example 5.6. Is the following space V = {

000

} a Vector Space?

(a) Yes, and I am very confident(b) Yes, but I am not very confident(c) No, but I am not very confident(d) No, and I am very confident

57

5.2.6 Common Vector Space Example 6: Homogeneous System of Equations:

Example 5.7. Determine whether or not V is a vector space. Verify your answer.

V =

xyz

∈ R3

∣∣∣∣∣∣x+ y + z = 1, 2x+ 2y + 2z = 2, and − x− y − z = −1

.

Notice when we change the system to a homogeneous system we get a vector space.

Example 5.8. Determine whether or not V is a vector space.

V =

xyz

∈ R3

∣∣∣∣∣∣x+ y + z = 0, 2x+ 2y + 2z = 0, and − x− y − z = 0

.

58

5.2.7 Vector Space Example 7: Heat States in Diffusions and other models:

A manufacturing company uses a process called diffusion welding to adjoin several smallerrods into a single longer rod. The diffusion welding process leaves the final rod heated tovarious temperatures along the rod with the ends of the rod having the same temperature.Every acm along the rod, a machine records the temperature difference from the temperatureat the ends to get an array of temperatures called a heat state.

1. Plot the heat state given below (let the horizontal axis represent distance from the left end

of the rod and the vertical axis represent the temperature difference from the ends).

u = (0, 1, 13, 14, 12, 5,−2,−11,−3, 1, 10, 11, 9, 7, 0)

2. How long is the rod represented by u, the above heat state, if a = 1cm?

3. Give another example of a heat state for the same rod, sampled in the same locations.

4. Notice that the set of all heat states, for this rod, is a vector space.

5. What do the vectors in this vector space look like? That is, what is the key property(or what are the key properties) that make these vectors stand out?

5.2.8 Vector Space Example 8: Galois Field 2: GF (2)n:

See HW 4.

59

60

5.3 Span and Subspace Motivation

Span: It is now natural to ask the question: What images can be created as linear com-binations of a given set of images? We say that image z is in the span of a set of imagesI = {x, y, ...} if z can be written as a linear combination of images in I. We write z ∈Span(I).We also say that Span(I) is the set of all linear combinations of the images in I.

Example 5.9. Image 3 from Lab 1 was in the span of Images A, B and C sinceImage 3= a*Image A+b*Image B+c*Image C. But Image 4 is not in Span(Image A, ImageB, Image C) since we could not write it as a linear combination of Images A, B and C.

Image A Image B Image C

Image 3 Image 4

Is Image 3∈ span {Image A, Image B, Image C}?

Subspace: Because Span(I) is determined by the same operations as closure of vector spaces,we might ask if Span(I) is a vector space of images. Indeed it is, and it is also a subset of theset of all images (of the same geometry). We say that W is a subspace of V if it is nonemptyand is closed under image addition and scalar multiplication.

Notation: x ∈ V means x is an element of V.

Example: If V = R, then 3 ∈ R.

U ⊆ V or U ⊂ V means U is a subset of V.

Example: If U = { even numbers }, then U ⊆ R.

61

5.4 Span

In this class, we will use the word Span as both a and . Let us beginwith the definition of the noun span.

Definition 5.3. (n.) Let V be a vector space and let X = {v1, v2, . . . , vn} ⊂ V . Then thespan of the set X is the set of all linear combinations of the elements of X. That is,

span X = span {v1, v2, . . . , vn} = {α1v1 + α2v2 + . . .+ αnvn| α1, α2, . . . , αn are scalars}.

Why do we care about “Span”?

• Allows us to encode a Vector Space compactly into its building blocks (uses bases-coming soon!)

• Helps us to represent the set of possible outputs of a Vector Space.

• Allows us to determine if subsets of vectors produce the same set of outputs.

• Helps us determine if a vector is in a Vector Space.

Example 5.10. Let v1 =

−1

01

, v2 =

100

, and v3 =

001

∈ R3. Can we write

Span({v1, v2, v3}) as a span of 2 vectors? 1 vector?

62

Example 5.11. Find the span of the two polynomials x and 1 in P1, denoted span {x, 1}.Recall P1 = {ax+ b| a, b ∈ R}. Also determine span {x, 2}. Is span {x, 2} = span {x, 1}?

Note: This example is interesting because it shows two different ways to write the same setas a span.

Method: How to show a vector ~u is in the span {v1, v2, ...vn}:Try to find scalars/coefficients αi such that u = α1v1 + α2v2 + α3v3 + · · ·+ αnvn.

Example 5.12. Let v1 = 3x+4, v2 = 2x+1, v3 = x2+2, and v4 = x2. Is v1 ∈ span {v2, v3, v4}?

63

64

5.4.1 ICE 2a: Span: Part 1

Eva and Archer are going on a trip for the first time. Dr. Harsy wants to help them on theirjourney so she gives them two gifts. Specifically, she gives them two forms of transportation:a hover board and a magic carpet. Dr. Harsy informs them that both the hover board andthe magic carpet have restrictions in how they operate:

We denote the restriction on the hover board’s movement by the vector

[31

].

By this we mean that if the hover board traveled ”forward” for one hour, it wouldmove along a ”diagonal” path that would result in a displacement of 3 miles Eastand 1 mile North of its starting location.

We denote the restriction on the magic carpet’s movement by the vector

[12

].

By this we mean that if the magic carpet traveled ”forward” for one hour, itwould move along a ”diagonal” path that would result in a displacement of 1mile East and 2 miles North of its starting location.

Scenario One: The Maiden VoyageEva and Archer’s first adventure is to go visit their cousin Mako who lives in a cabin that is107 miles East and 64 miles North of their home.

Task 1a:Investigate whether or not Eva and Archer can use the hover board and the magic carpet toget to Mako’s cabin. If so, how? If it is not possible to get to the cabin with these modes oftransportation, why is that the case?

65

Task 1b: What is another “linear algebra” way to ask the question in Task 1?

Task 2b: Suppose the magic carpet is acting up and now Eva and Archer need to borrowtheir friend’s Habanero (Dr. Haven’s cat) to get to Mako’s cabin. She has many different

carpets which movement restrictions given by c1 =

[31

], c2 =

[64

], c3 =

[13

], and c4 =

[−3−1

].

Archer doesn’t want to do much work calculating this answer. Eva says that Archer couldprobably answer this in his head. Which carpet should he borrow? Check your answer ifyou have time.

66

5.5 Spanning Sets

Definition 5.4. (v.) We say that the set of vectors {v1, v2, . . . , vn} spans a set X ifX = span {v1, v2, . . . , vn}. In this case, we call the set {v1, v2, . . . , vn} a spanning set of X.

In Example 5.11, we found that both {x, 1} span P1.

67

Example 5.13. True or False: Spanning Sets are unique.


Method: How to show a subset of vectors {v1, v2, ...vn} spans (is a spanningset for) a Vector Space W:

This means we want to show we can produce any vector w in W using a linear combinationof vectors in the set{v1, v2, ...vn} ⊆ W .The problem is that W often has infinitely many vectors in it. How can we check them all?

Solution: Pick an arbitrary vector in W and see if we can write the arbitrary vector as alinear combination of v1, v2, ...vn Try to find scalars/coefficients (αi’s) such thatu = α1v1 + α2v2 + α3v3 + · · ·+ αnvn.

What is an arbitrary vector representative for R3?

What is an arbitrary vector representative for P3?

Example 5.14. Which of the following sets are spanning sets for P1 = {ax+ b|a, b ∈ R}:

A = {x, x+ 1} B = {x}

68

C = {200, 1}

E = {x, 1, 3x+ 4}

D = {x, 1, x2}

F = {x+ 1, 2x+ 2}

69

Example 5.15. Show that R2 is spanned by both

{(10

),

(01

)}and

{(11

),

(12

),

(−13

)}.

70

Notice that in each of these examples, we found that the span of a set turned out to be avector space. It turns out that this is always true.

Because of this, we now have a new way to determine if a set is a vectors space -if it can bewritten as a span.

Method: How to write a set of vectors {v1, v2, ...vm} as a span (if you can):

Ultimate Goal: Be able to write the set S = {v1, v2, ...vm} in the form{α1v1 + α2v2 + . . .+ αnvn| α1, α2, . . . , αn ∈ R} = span {v1, v2, ..., vn}*Note there are NO restrictions on the constants!For example, P1 = {ax+ b|a, b ∈ R} = span {1, x}.

Example 5.16. Are any of these sets spanning sets of {v1, v2}, {v1}, or {v2}?S = {v1 + av2|a ∈ R}

T = {5 + av1|a ∈ R}

V = {a(v1 + v2)|a ∈ R}

Example 5.17. Show that {a0 + a1x|a0 + a1 = 0} is a vector space by writing it as a spanof vectors.

71

Here, we summarize the terminology:

We say {v1, v2, . . . , vn} spans a set V if V is the span of {v1, v2, . . . , vn}.

We say {v1, v2, . . . , vn} is a spanning set for V if V is the span of {v1, v2, . . . , vn}.

We say V is spanned by {v1, v2, . . . , vn} if V is the span of {v1, v2, . . . , vn}.

Finally, all of these mean that if v ∈ V then there are scalars α1, α2, . . . , αn so thatv = α1v1 + α2v2 + . . . αnvn.

Example 5.18. How do you describe the span of the vectors (1, 0, 0), (0, 1, 0), (0, 0, 1)?(a) A point(b) A line segment(c) A line(d) R2

(e) R3

Example 5.19. How do you describe the set of all linear combinations of the vectors (1, 0, 0),(0, 1, 0)?(a) A point(b) A line segment(c) A line(d) R2

(e) R3

Example 5.20. How do you describe the set of all linear combinations of the vector (1, 0, 0)?(a) A point(b) A line segment(c) A line(d) R2

(e) R3

72

5.5.1 ICE 2b: Span: Part 2

Scenario two: Getting Back HomeSuppose Eva and Archer are now in a three-dimensional world for the carpet ride problem,

and they have three models of transportation: v1 =

111

, v2 =

638

, v3 =

416

.

Task 1:2 Eva and Archer are only allowed to use each mode of transportation once (in theforward or backward direction) for a fixed amount of time (c1 on v1, c2 on v2, c3 on v3). Findthe amounts of time on each mode of transportation (c1, c2, and c3, respectively) neededto go on a journey that starts and ends at home OR explain why it is not possible to do so.

Hint 1: We don’t know where Eva and Archer’s house is (and Dr. Harsy is not sure shewants to share this with her class), so what could we use to represent the location of theirhouse?

Hint 2: Set up an equation you want to solve with these vectors. And see if you can findvalues for c1, c2, and c3.

22 sided!

73

Bonus Task 1: What equation do we really solve in Task 1?

Task 2: Is there more than one way to make a journey that meets the requirements describedabove? (In other words, are there different combinations of times you can spend on the modesof transportation so that you can get back home?) If so, how? (use the same vectors fromthe first page.3)

3v1 =

111

, v2 =

638

, v3 =

416

.

74

5.6 Subspaces

Many times, we work in subsets of vector spaces.Definition: Let V be a vector space. If W ⊆ V is a subspace of V if it satisfies thefollowing three properties.

1. ∈ W(We don’t allow W to be empty and so we require that it contains the 0 vector.)

2. Let u ∈ W and α be a scalar. Then ∈ W .The Closure Property under scalar multiplication.

3. Let u, v ∈ W Then ∈ W .The Closure Property under vector addition.

Note: You can combine 2 & 3 by checking whether the space is closed under linear combi-nations:Let u, v ∈ W and α, β be scalars. Then αu+ βv ∈ W .

1. ∈ W(We don’t allow W to be empty and so we require that it contains the 0 vector.)

2.

Heuristic Verification: Why Only These 2 Properties Are Necessary To Check:Notice that if W is a subspace of V , then W is a vector space as well. The Vector Propertiesare inherited from V since V is like a parent set to W . The scalar 1 still exists in the scalarset also. The only issue that may happen comes with the Closure Properties. This meansthat we need only show closure under addition and scalar multiplication and that 0 ∈ W .

Theorem: Let V be a vector space and let v1, v2, . . . , vn ∈ V . Then span {v1, v2, . . . , vn} isa subspace of V .

Method: How to show a subset of vectors, V, of a Vector Space W is asubspace: There are 2 methods:

First Method:Show that both1) 0 ∈ V and2) Pick arbitrary vector u ∈ V & scalar αand show that αu ∈ V .3) Pick arbitrary vectors v, u ∈ Vand show that u+ v ∈ V .

Second Method:Write the set as a span.That is, show thatV = {a1v1 + a2v2 + ...+ anvn|ai ∈ R}⇒ V = span {v1, v2, ..., vn}

*Note that in both methods, the subset must follow the normal rules of the bigger space.

75

Example 5.21. Let W = {(x, y) ∈ R2| − 2x + 7y = 9, x − 4y = −2}. Is W a subspace ofR2?

Note: A line in R2 that does NOT go through the origin is not a subspace of R2. Similarlya hyper-plane in Rn that doesn’t contain the zero vector is not a subspace.

Example 5.22. Is R2 a subspace of R3?

Example 5.23. Is W = {a[21

]|a ∈ R} a subspace of R2?

76

Example 5.24. Let V = {(x, y, z) ∈ R3| x + y + z = 0, 3x + 3y + 3z = 0}. Write V as aspan to show that V is a subspace of R3.

77

Reading Example: Please read on your own time.

Example 5.25. Consider the set of images V = {I| I is of the form below and a, b, c ∈ R}.

+2b

a

2a

a+b

I =

a−c

b

−ba

a

c−a

b

c+b

b+2c

2a+2b+2c

c

a

We can show that V is a subspace of images with the same geometric configuration. We showedabove that the set of these images is a vector space, so we need only show the two subspace properties.First, notice that the 0 image is the image with a = b = c = 0 and this image is in V . Now, we needto show that linear combinations are still in the set V . Let α, β ∈ R, be scalars and let I1, I2 ∈ V ,then there are real numbers a1, b1, c1, a2, b2, and c2 so that

a1

I1 =

a1

2a1

b1

c1

b1

a1

+b1

a1

−b1a1

−a1

+b1

+2c1

b1

2a1+2b1+2c1

+2b1

−c1a1

c1

c1

+2c2

a2

2a2

I2 = a2+b2

a2−c2

b2

−b2a2

a2+2b2

c2

b2

a2

c2−a2

c2+b2

b2+2c2

2a2+2b2

Notice that αI1 + βI2 is also in V since... (see image on next page)

78

αc1 + βc2

αI1 + βI2 =

2αa1 + 2βa2+β(c2 − a2)

α(c1 − a1)

αa1 + βa2αa1 + βa2

+β(a2 − b2)

α(a1 − b1)+β(c2 + b2)

α(c1 + b1)

αb1 + βb2+β(a2 + b2)

α(a1 + b1)

+β(a2 − c2)

α(a1 − c1)

αb1 + βb2 +β(b2 + 2c2)

α(b1 + 2c1)

+2b2 + 2c2)+β(2a2+2b1 + 2c1)α(2a1

+β(a2 + 2b2)

α(a1 + 2b1)

Notice that performing the operations inside each pixel shows that we can write αI1 + βI2 in the

same form as I above. That is, αI1 + βI2 ∈ V . Thus V is a subspace of images that are laid out

in the geometric form above. Notice, this means that V is in itself a vector space.

79

80

5.6.1 ICE 2c: Span: Part 3 -Subspaces

Which of these subsets are subspaces of M2×2? For each one that is a subspace, write theset as a span. For each that is not, show the condition that fails.4!

A =

{(a 00 b

)∣∣∣∣ a, b ∈ R}

B =

{(a 0c b

)∣∣∣∣ a− 2b = 0, c ∈ R}

4Two Sided

81

C =

{(a 0c 0

)∣∣∣∣ a+ c = 1

}

D =

{(a 10 b

)∣∣∣∣ a, b ∈ R}

82

5.7 Linear Independence

In the Radiography and Tomography Lab #1, you were asked to write one image usingarithmetic operations on three others. You found that this was possible sometimes and notothers. It becomes a useful tool to know whether or not this is possible.

Definition: Let V be a vector space. We say that the set {v1, v2, . . . , vn} ⊂ V , is linearlyindependent if no element is in the span of the others. This means, we can’t write thisvector as a linear combination of the other vectors in the set.

Check for Linear Independence: α1v1 + α2v2 + . . . + αnvn = 0 is true only when α1 =α2 = . . . = αn = 0., then the vectors v1, v2, . . . , vn are linearly independent.This check is called the linear dependence relation.

Definition 5.5. If {v1, v2, . . . , vn} is not linearly independent, then we say that it is linearlydependent. Note this means that we can write one element as a linear combination of theothers.

Here is the process to check whether or not {v1, v2, ...vn} is linear independent:Step 1: Set an arbitrary linear combination of vi’s equal to 0:α1v1 + α2v2 + ...+ αnvn = 0Step 2: See if you can find values for the αi’s that are not all 0. If you can, the set isdependent. If α1 = α2 = ... = αn = 0 is the only possible values for the α’s, then the set islinear independent.

Careful: Of course one solution for the linear dependence relation will always be α1 =α2 = . . . = αn = 0, but this tells us nothing about the linear dependence of the set. That is,just because α1 = α2 = . . . = αn = 0 works, it doesn’t mean the set is linear independent.You must show that it is the solution!

Demonstration: If the set {v1, v2} is linearly dependent, then we can find α1 and α2,BOTH not zero, such that α1v1 + α2v2 = 0.

83

Example 5.26. The set {0, v1, v2, . . . , vn} is always linearly dependent.

Thus, it if we are considering the linear dependence a two element set, we need only checkwhether one can be written as a scalar multiple of the other.

Example 5.27. Determine whether {x2 + x, x2, 1} is linearly dependent or independent.

84

Notice that whenever we are determining whether a set is linearly independent or dependent, we

always start with the linear dependence relation and determine whether or not there is only one

set of scalars that (when they are all zero) to make the linear dependence relation true.

Example 5.28. Determine the linear dependence of the set

{(1 31 1

),

(1 11 −1

),

(1 21 0

)}.

(To See Row Reduction)5

5

1 1 1 03 1 2 01 1 1 01 −1 0 0

R2=−3r1+r2−→R3=−r1+r3,R4=−r1+r4

1 1 1 00 −2 −1 00 0 0 00 −2 −1 0

R2=1

−2 r2−→R4=−r2+r4

1 1 1 00 1 1

2 00 0 0 00 0 0 0

R1=−r2+r1−→

1 0 12 0

0 1 12 0

0 0 0 00 0 0 0

85

5.8 Basis

Idea: Describe a vector space in terms a much smaller subset. This set can be consideredbuilding blocks which we can use to build any vector from the vector space.Recall, a vector space can be written as a span of vectors. Sometimes when we compute thespan of a set of vectors, we see that it can be written as a span of a smaller set of vectors.This means that the vector space can be described with the smaller set of vectors. Thishappens when the larger set of vectors is linearly dependent.

Main Goal: Find a spanning set that is big enough to describe all of the elements of ourvector space, but not so big that there’s repetition.

Definition 5.6. Given a vector space V , we call B = {v1, v2, . . . , vn} the basis of V if andonly if B satisfies the following conditions:

1. span (B) = and

2. B is linearly independent.

Think of it this way, we are like Goldie Locks, we want our set to be1) Big Enough so2) Not to Big that we have

Example 5.29. The standard basis for R3 is B =

100

,

010

,

001

.

86

Notation: Because the standard basis is used often, we introduce notation for each of thevectors. We let e1, e2, and e3 denote the three vectors in S, where

e1 =

100

, e2 =

010

, and e3 =

001

.

In general, for Rn, the standard basis is {e1, e2, . . . , en}, where ei is the n × 1 vector arraywith zeros in every entry except the ith entry which contains a one.

Example 5.30. Show that another basis for R3 is B =

201

,

110

,

001

.

87

Example 5.31. The standard basis for P2 is S = {x2, x, 1}. Show S is a basis for P2.Recall, P2 = {ax2 + bx+ c · 1|a, b, c ∈ R} = span {x2, x, 1}.

Example 5.32. B = {1, x+ x2, x2} is also a basis for P2. In Exercise5.27, we showed thatB is linearly independent. So, we need only show that span B = P2. again, we need onlyshow that P2 ⊆ span B since it is clear that span B ⊆ P2.

88

Notice in the last four examples, we see two examples where the bases of a vector space hadthe same number of elements. That is, both bases of R3 had the same number of elementsand both bases for P2 had the same number of elements. One should wonder if this is alwaystrue. The answer is given in the next theorem.

Theorem 5.1. Let V be a vector space with bases B1 = {v1, v2, . . . , vn} and B2 = {u1, u2, . . . , um}.Then the number of elements, n in B1 is the same as the number of elements, m in B2.

That is, n = .

Proof. Suppose both B1 and B2 are bases for V . We show that this is true by assuming itis not true and showing that this is an impossible scenario. That is, we will assume thatm 6= n and find a reason that this cannot be true. Suppose m > n (a very similar argumentcan be made if we assumed n > m). Since both B2is a subset of V , we know that there existαi,j for 1 ≤ i ≤ m and 1 ≤ j ≤ n so that

u1 =α1,1v1 + α1,2v2 + . . .+ α1,nvn

u2 =α2,1v1 + α2,2v2 + . . .+ α2,nvn...

um =αm,1v1 + αm,2v2 + . . .+ αm,nvn.

We want to show that B cannot be linearly independent (which would be impossible if it isa basis). Let

β1u1 + β2u2 + . . .+ βmum = 0.

We will then find β1, β2, . . . , βm. Notice that if we replace u1, u2, . . . , un with the linearcombinations above, we can rearrange to get

(β1α1,1 + β2α2, 1 + . . .+ βmαm,1)v1

+(β1α1,2 + β2α2, 2 + . . .+ βmαm,2)v2

...

+(β1α1,n + β2α2, n+ . . .+ βmαm,n)vn = 0.

Since B1 is a basis, we get that the coefficients of v1, v2, . . . , vn are all zero. That is

β1α1,1 + β2α2, 1 + . . .+ βmαm,1) = 0 (1)

β1α1,2 + β2α2, 2 + . . .+ βmαm,2 = 0 (2)

... (3)

β1α1,n + β2α2, n+ . . .+ βmαm,n = 0. (4)

We know that this system has a solution because it is homogeneous. But, because there are

89

more scalars β1, β2, . . . , βm (that we are solving for) than there are equations, this systemmust have infinitely many solutions. This means that B2 cannot be linearly independent andso it cannot be a basis. Thus, the only way both B1 and B2 can be bases of the same vectorspace is if they have the same number of elements.

Because the number of elements in a basis is unique to the vector space, we can give it aname.

Definition 5.7. Given a vector space V whose basis B has n elements, we say that thedimension of V is n, the number of elements in B (dimV = n.)Sometimes we say that V is n-dimensional.

Example 5.33. We can see that the last 4 examples show that both R3 and P2 are

a) 1 dimensionalb) 2 dimensionalc) 3 dimensionald) 4 dimensionale) 5 dimensional

Note: What we see is that, in order to find the dimension of a vector space, we need to finda basis and count the elements in the basis.

This can also help us immediately check whether or not a set of vectors is a basis.

Since P4 = {ax4 + bx3 + cx2 + dx+ e|a, b, c, d, e ∈ R} = span {x4, x3, x2, x, 1}, it’s dimensionis so we know any set that doesn’t have exactly vectors will not be abasis.

Bases are not unique, but often we want to the easiest basis (set of building blocks.) What

would be the easiest (standard) basis for M2×2 = {[a bc d

]|a, b, c, d ∈ R}?

90

Example 5.34. Let V =

xyz

∣∣∣∣∣∣x+ y + z = 0, 2x+ y − 4z = 0, 3x+ 2y − 3z = 0

.

Find the dimension of V. First, we need to rewrite V as a span.

x + y + z = 02x + y − 4z = 03x + 2y − 3z = 0

use matrix−→

1 1 1 02 1 −4 03 2 −3 0

R2=−2r1+r2−→

R3=−3r1+r3

1 1 1 00 −1 −6 00 −1 −6 0

R1=r2+r1−→R3=−r2+r3,R2=−r2

1 0 −5 00 1 6 00 0 0 0

−→

91

92

5.8.1 ICE 3: Basis

2 sides!

1. True or False: Each vector space has a unique basis.(a) True, and I am super confident!(b) True, but I am not very confident.(c) False, and I am super confident! (d) False, but I am not very confident.

2. Which of the following describes a basis for a subspace V ?(a) A basis is a linearly independent spanning set for V .(b) A basis is a minimal spanning set for V .(c) A basis is a largest possible set of linearly independent vectors in V .(d) All of the above(e) Some of the above

3. Which of the following sets of vectors forms a basis for R3?

i.

−213

,

35−1

ii.

−204

,

01−1

,

120

iii.

−200

,

120

,

30−2

,

62−2

iv.

462

,

321

,

8124

,

642

(a) i, ii, iii, and iv(b) ii, iii, and iv only(c) ii and iii only(d) iii and iv only(e) ii only

93

4. Which of the following sets of vectors spans R3?

i.

−213

,

35−1

ii.

−204

,

01−1

,

120

iii.

−200

,

120

,

30−2

,

62−2

iv.

462

,

321

,

8124

,

642

(a) i, ii, iii, and iv(b) ii, iii, and iv only(c) ii and iii only(d) iii and iv only(e) ii only

5. Which of the following describes the subspace of R3 spanned by the vectors from :

{

462

,

321

,

8124

,

642

}? (note this is set iv from the previous problem.)

(a) A line(b) A plane(c) R2

(d) All of R3

(e) Both (b) and (c)

94

6 Radiography and Tomography Lab 2:

6.1 Introduction to Lab 2

Read the following pages entitled “Transmission Radiography and Tomography: A Simplified

Overview” as a prerequisite to Lab 2. You may not understand everything from this handout,

but I would like you to look for the linear algebra in the methods and try to get a general overview

of the Radiography Process. You do not need to fully understand anything else besides

the linear algebra concepts that come up. I have included the following question for you to

answer to help guide you in your reading (below) along with the goals and summary of terminology

of what you should get out of the reading after the reading section.

6.1.1 Goals from the Reading Assignment:

• To understand a unified and simplified physical model of taking an x-ray radiographof an object.

• To recognize and appreciate that, sometimes, complex phenomena can be reasonablymodeled with straightforward mathematics.

• To recognize and appreciate that mathematical assumptions should be backed by phys-ical reality in order to preserve model integrity. This is justification!

• To be exposed to the notation used in the Lab 2.

• To understand the matrix equation representation of the radiographic process.

6.1.2 Questions from the Reading Assignment:

1. Given Tkj = 0.42, what does this value mean?

2. What is accomplished in the matrix multiply when multiplying an object vector by the kth

row of T?6

6More questions on back!

95

3. Explain how you change the radiographic operator when changing from one view to many.

4. Why do we use pixel values βk where βk is defined as βk ≡ −α ln(pkp0 ) instead of the expected

photon count pk?

96


Transmission Radiography and Tomography

A Simplified Overview

This material provides a brief overview of radiographic principles for the model in Lab #2 ofthe Radiography and Tomography Linear Algebra Modules. The goal is to develop the basicdiscrete radiographic operator for axial tomography of the human body. To accomplish thisgoal, it is not necessary to completely understand the details of the physics and engineering.We wish to arrive at a mathematical formulation descriptive of the radiographic process andestablish a standard scenario description with notation.

What is Radiography?

Transmission radiography and tomography are familiar and common processes in today’sworld, especially in medicine and non-destructive testing in industry. Some examples include

• Single-view X-ray radiography is used routinely to view inside the human body; forexample, bone fracture assessment, mammography, and angiographic procedures.

• Multiple-view X-ray radiography is realized in computerized axial tomography (CAT)scans used to provide 3D images of body tissues.

• Neutron and X-ray imaging is used in industry to quantify manufactured part assem-blies or defects which cannot be visually inspected.

Transmission Radiography is the process of measuring and recording changes in a high-energy particle beam (X-rays, protons, neutrons, etc.) resulting from passage through anobject of interest.

Tomography is the process of inferring properties of an unknown object by interpretingradiographs of the object.

X-rays, just like visible light, are photons or electromagnetic radiation, but at much higherenergies and outside of the range of our vision. Because of the wavelength of typical X-rays (on the order of a nanometer), they readily interact with objects of similar size such asindividual molecules or atoms. This property makes them particularly useful in transmissionimaging. Figure 1 is a cartoon of a typical x-ray radiographic experiment or procedure. Anx-ray beam is produced with known energy and geometric characteristics. The beam isaimed at a region of interest. The photons interact with matter in the region of interest,changing the intensity, energy and geometry of the beam. A detector measures the pattern

1

97


Source

x−ray

x−ray beam

Incident Region of InterestDetector

x−ray beam

Transmitted

E(0, y, z) E(Dx, y, z)

ρ(x, y, z)

Figure 1: Typical radiographic experiment.

(and possibly the distribution) of incident energy. The detection data, when compared tothe incident beam characteristics, contains the known signature of the region of interest.We consider the mathematics and some of the physics involved in each step with the goalof modeling a radiographic transformation appropriate for mixed soft and hard tissue axialtomography of the human body.

The Incident X-ray Beam

We begin with an x-ray beam in which the x-ray photons all travel parallel to each other inthe beam direction, which we take to be the positive x-direction. Additionally we assumethat the beam is of short time duration, the photons being clustered in a short pulse insteadof being continuously produced. A beam with these geometric characteristics is usuallynot directly achievable through typical x-ray sources (see supplementary material for somediscussion on x-ray sources). However, such a beam can be approximated readily throughso-called collimation techniques which physically limit the incident x-rays to a subset thatcompose a planar (neither convergent nor divergent) beam.

While not entirely necessary for the present formulation, we consider a monochromaticx-ray source. This means that every x-ray photon produced by the source has exactly thesame “color.” The term “monochromatic” comes from the visible light analog in which,for example, a laser pointer may produce photons of only one color, red. The energy of asingle photon is proportional to its frequency ν, or inversely proportional to its wavelengthλ. The frequency, or wavelength, determine the color, in exact analogy with visible light.In particular, the energy of a single photon is hν where the constant of proportionality h isknown as Planck’s constant.

The intensity (or brightness), E(x, y, z), of a beam is proportional to the photon density.The intensity of the beam just as it enters the region of interest at x = 0 is assumed to be

2

98


xρ(x)

dE(x) = E(x+ dx)− E(x) = µρ(x)E(x)dx

x+dx

E(x+dx)

0 D x

E(Dx )

E(0)

E(x)

Figure 2: X-ray beam attenuation computation for a beam path of fixed y and z.

the same as the itensity at the source. We write both as E(0, y, z). It is assumed that thisquantity is well known or independently measureable.

X-Ray Beam Attenuation

As the beam traverses the region of interest, from x = 0 to x = Dx (see Figure 2), theintensity changes as a result of interactions with matter. In particular, a transmitted (resul-tant) intensity E(Dx, y, z) exits the far side of the region of interest. We will see that underreasonable assumptions this transmitted intensity is also planar but is reduced in magni-tude. This process of intensity reduction is called attenuation. It is our goal in this sectionto model the attenuation process.

X-ray attenuation is a complex process that is a combination of several physical mecha-nisms (outlined in the supplementary material) describing both scattering and absorption ofx-rays. We will consider only Compton scattering which is the dominant process for typicalmedical x-ray radiography. In this case, attenuation is almost entirely a function of photonenergy and material mass density. As we are considering monochromatic (monoenergetic)photons, attenuation is modeled as a function of mass density only.

Consider the beam of initial intensity E(0, y, z) passing through the region of interest, atfixed y and z. The relative intensity change in an infinitesimal distance from x to x+ dx isproportional to the mass density ρ(x, y, z) and is given by

dE(x, y, z) = E(x+ dx, y, z)− E(x, y, z) = −µρ(x, y, z)E(x, y, z)dx

where µ is a factor that is nearly constant for many materials of interest. We also assumethat any scattered photons exit the region of interest without further interaction. This isthe so-called single-scatter approximation which dictates that the intensity remains planarfor all x.

Integrating over the path from x = 0 where the initial beam intensity is E(0, y, z) tox = Dx where the beam intensity is E(Dx, y, z) yields

dE(x, y, z)

E(x, y, z)= −µρ(x, y, z)dx

3

99


∫ E(Dx,y,z)

E(0,y,z)

dE(x, y, z)

E(x, y, z)= −µ

∫ Dx

0

ρ(x, y, z)dx

lnE(Dx, y, z)− lnE(0, y, z) = −µ∫ Dx

0

ρ(x, y, z)dx

E(Dx, y, z) = E(0, y, z)e−µ∫Dx0 ρ(x,y,z)dx.

This expression shows us how the initial intensity is reduced, because photons havebeen scattered out of the beam. The relative reduction depends on the density (or mass)distribution in the region of interest.

Radiographic Energy Detection

The transmitted intensity E(Dx, y, z) continues to travel on to a detector (e.g. film) whichrecords the total detected energy in each of m detector bins. The detected energy in any binis the intensity integrated over the bin cross-sectional area. Let pk be the number of x-rayphotons collected at detector bin k. pk is then the collected intensity integrated over the binarea and divided by the photon energy.

pk =1

hν

∫∫

(bin k)

E(0, y, z)(e−µ

∫Dx0 ρ(x,y,z)dx

)dydz.

Let the bin cross sectional area, σ, be small enough so that both the contributions of thedensity and intensity to the bin area integration are approximately a function of x only.Then

pk =σE(0, yk, zk)

hνe−µ

∫Dx0 ρ(x,yk,zk)dx,

where yk and zk locate the center of bin k. Let p0k be the number of x-ray photons initiallyaimed at bin k, p0k = σE(0, x, y)/hν. Due to attenuation, pk ≤ p0k for each bin.

pk = p0ke−µ

∫Dx0 ρ(x,yk,zk)dx.

Equivalently, we can write (multiply the exponent argument by σ/σ):

pk = p0ke−µσ

∫Dx0 σρ(x,yk,zk)dx.

The remaining integral is the total mass in the region of interest that the x-ray beam passesthrough to get to bin k. We will call this mass sk. Now we have

pk = p0ke−sk/α,

where α = σ/µ. This expression tells us that the number of photons in the part of the beamdirected at bin k is reduced by a factor that is exponential in the total mass encountered bythe photons.

Finally, we note that the detector bins correspond precisely to pixels in a radiographicimage.

4

100


total mass

along

total mass

Pixel

detector

region of interest

Voxel j

beam path k

Sk =∑Nj=1 Tkjxj

fraction of voxel jin beam path k in voxel j

kbeam path k

Figure 3: Object space and radiograph space discretization.

The Radiographic Transformation Operator

We consider a region of interest subdivided into N cubic voxels (three-dimensional pixels).Let xj be the mass in object voxel j and Tkj the fraction of voxel j in beam path k (seeFigure 3). (Note that this is a different context, in particular, xj is *not* related to thedirection that the x-ray beams are traveling!) Then the mass along beam path k is

sk =N∑

j=1

Tkjxj,

and the expected photon count at radiograph pixel k, pk, is given by

pk = p0ke− 1α

∑Nj=1 Tkjxj ,

or equivalently,

bk ≡(−α ln

pkp0k

)=

N∑

j=1

Tkjxj.

The new quantities bk represent a variable change that allows us to formulate the matrixexpression for the radiographic transformation

b = Tx.

5

101


Figure 4: Example axial radiography scenario with six equally spaced views. Horizontalslices of the object project to horizontal rows in the radiographs.

This expression tells us that given a voxelized object mass distribution image x ∈ RN , theexpected radiographic data (mass projection) is image b ∈ Rm, with the two connectedthrough radiographic transformation T ∈ Mm×N(R). The mass projection b and actualphoton counts p and p0 are related as given above. It is important to note that bk is definedonly for pk > 0. Thus, this formulation is only valid for radiographic scenarios in whichevery radiograph detector pixel records at least one hit. This is always the case for medicalapplications which require high constrast and high signal-to-noise ratio data.

Multiple Views and Axial Tomography

Thus far, we have a model that can be used to compute a single radiograph of the regionof interest. In many applications it is beneficial to obtain several or many different views ofthis region. In some industrial applications, the region of interest can be rotated within theradiographic apparatus, with radiographs obtained at various rotation angles. In medicalapplications the radiographic apparatus is rotated about the region of interest (includingthe subject!). In this latter case, the voxelization remains fixed and the coordinate systemrotates. For each of a view angles the new m pixel locations require calculation of new massprojections Tkj. The full multiple-view operator contains mass projections onto all M = a·mpixel locations. Thus, for multiple-view radiography: x is still a vector in RN , but b is avector in RM and T is a matrix operator in MM×N(R).

Finally, we make the distinction between the general scenario and axial tomography(CAT scans). In principle, we could obtain radiographs of the region of interest from anydirection (above, below, left, right, front, back, etc.). However, in axial tomography thephysical limitations of the apparatus and subject placement dictate that views from somedirections are not practical. The simplest scenario is to obtain multiple views by rotatingthe apparatus about a fixed direction perpendicular to the beam direction. This is why CATmachines have a donut or tube shaped appearance within which the apparatus is rotated.The central table allows the subject to rest along the rotation axis and the beam can passthrough the subject along trajectories.

This axial setup also simplifies the projection operator. If we consider the `th slice ofthe region of interest, described by an n × n arrray of N voxels, the mass projections ofthis slice will only occur in the `th row of pixels in each radiographic view see Figure 4.

6

102


As a result, 3D reconsructions can be obtained by a series of independent 2D reconstructedslices. For example, the brown slice of the spherical object (represented in RN) is related tothe collection of brown rows of the radiographs (represented in RM) through the projectionoperator T ∈ MM×N(R). The black slice and black rows are related through the sameprojection operator.

Model Summary

The list below gathers the various mathematical quantities of interest.

• N is the number of object voxels.

• M is the number of radiograph pixels.

• x ∈ RN is the material mass in each object voxel.

• b ∈ RM is the mass projection onto each radiograph pixel.

• p ∈ RM is the photon count recorded at each radiograph pixel.

• p0 ∈ RM is the incident photon count possible for each radiograph pixel.

• T ∈ MN×M(R) is voxel volume projection operator. Tij is the fractional volume ofvoxel j which projects orthogonally onto pixel i.

• b = −α ln(

pp0

).

• b = Tx is the (mass projection) radiographic transformation.

The description of images (objects and radiographs) as vectors in RN and RM is compu-tationally useful and more familiar than a vector spaces of images. One should keep inmind that this is a particular representation for images which is useful as a tool but isnot geometrically descriptive. The price we pay for this convenience is that we no longerhave the geometry of the radiographic setup (pixelization and voxelization) encoded in therepresentation.

A vector in R3, say (1, 2, 5), is a point in a three-dimensional space with coordinatesdescribed relative to three orthogonal axes. We can actually locate this point and plot it.An image represented in R3, say (1, 2, 5), is not a point in this space. Without furtherinformation about the vector space of which it is a member, we cannot draw this image. Theuse of R3 allows us to perform scalar multiplication and vector addition on images becausethese operations are equivalently defined on R3.

7

103


Model Assumptions

The radiography model we have constructed is based on a number of approximations andassumptions which we list here. This list is not comprehensive, but it does gather the mostimportant concepts.

• Monochromaticity. Laboratory sources generate x-rays with a wide range of ener-gies as a continuous spectrum. We say that such x-ray beams are polychromatic.One method of approximating a monochromatic beam is to precondition the beamby having the x-rays pass through a uniform material prior to reaching the region ofinterest. This process preferentially attenuates the lower energy photons, leaving onlythe highest energy photons. This process is known as beam-hardening. The result isa polychromatic beam with a narrower range of energies. We can consider the beamto be approximately monochromatic, especially if the attenuation coefficient(s) of thematerial, µ, is not a strong function of photon energy.

• Geometric Beam Characteristics. Laboratory sources do not naturally generateplanar x-ray beams. It is more characteristic to have an approximate point sourcewith an intensity pattern that is strongly directionally dependent. Approximate planarbeams with relatively uniform intensity E(0, y, x) can be achieved by selective beamshielding and separation of source and region of interest. In practice, it is common touse the known point source or line source characteristics instead of assuming a planarbeam. The model described here is unchanged except for the computation of T itself.

• Secondary Radiation. Our model uses a single-scatter approximation in which if aphoton undergoes a Compton scatter, it is removed from the analysis. In fact, x-rayscan experience multiple scatter events as they traverse the region of interest or otherincidental matter (such as the supporting machinery). The problematic photons arethose that scatter one or more times and reach the detector. This important secondaryeffect is often approximated by more advanced models.

• Energy-Dependent Attenuation. The attenuation coefficient µ, which we havetaked to be constant, is not only somewhat material dependent but is also beam energydependent. If the beam is truly monochromatic this is not a problem. However, fora polychromatic beam the transmitted total energy will depend on the distribution ofmass along a path, not just the total mass.

• Other Attenuation Mechanisms. We have included only Compton scattering in themodel. Four other mechanisms (outlined in the supplementary material) contribute tothe attenuation. While Compton scattering is the dominant contributor, photoelectricscattering will have some effect. It becomes important at the lower energies of interestand for materials of relatively high atomic number – such as calcium which is concen-trated in bone. The major effect of ignoring photoelectric scattering is quantitativemass uncertainty.

8

104


• There are a number of Detector-Related Effects which affect radiograph accuracy.Energy detection efficiency can be a function of beam intensity, photon energy andeven pixel location. Detectors are subject to point-spread effects in which even aninfinitesimally narrow beam results in a finitely narrow detection spot. These types ofeffects are usually well-understood, documented and can be corrected for in the data.Detection is also prone to noise from secondary radiation, background radiation, orsimply manufacturing variances.

Additional Resources

There is a variety of online source material that expands on any of the material presentedhere. Here are a few starting points.

https://www.nde-ed.org/EducationResources/CommunityCollege/Radiography/cc_rad_index.htm

http://web.stanford.edu/group/glam/xlab/MatSci162_172/LectureNotes/01_Properties%20&%20Safety.pdf

http://radiologymasterclass.co.uk/tutorials/physics/x-ray_physics_production.html

Several physical processes contribute to absorption and scattering of individual photonsas they pass through matter. Collectively, these processes alter the geometry and intensityof a beam of such photons. What follows is a brief description of each.

• The Photoelectric Effect is the absorption of an x-ray photon by an atom accompa-nied by the ejection of an outer-shell electron. This ionized atom then re-absorbs anelectron and emits an x-ray of energy characteristic of the atom. The daughter x-rayis a low-energy photon which is quickly re-absorbed and is effectively removed fromthe x-ray beam. Photoelectric absorption is the dominant process for photon energiesbelow about 100keV and when interacting with materials of high atomic number.

• Rayleigh Scattering is the process of a photon interacting with an atom withoutenergy loss. The process is similar to the collision of two billiard balls. Rayleighscattering is never the dominant mechanism, at any energy.

• Compton Scattering occurs when an x-ray photon interacts with an electron impartingsome energy to the electron. Both electron and photon are emitted and the photonundrgoes a directional change or scatter. Compton Scattering is the dominant processfor soft tissue at photon energies between about 100keV through about 8MeV.

• Pair Production is the process in which a photon is absorbed producing a positron-electron pair. The positron quickly decays into two 510keV x-ray photons. Pari produc-tion is only significant for photon energies of several MeV or more.

• Photodisintegration can occur for photons of very high energy. Photodisintegrationis the process of absorption of the photon by an atomic nucleus and the subsequentejection of a nuclear particle.

9

105

6.1.3 Summary of Terminology from the Reading Assignment:

It may be useful to have this out as a reference during Lab 2. As this lab follows the lectureon the physics of radiography, much of this material is review from your reading, but isprovided here for clarity and continuity. It is important to keep the notation consistent.There are several details to keep in mind:

Pixel versus Voxel. “Voxel” is the standard term for a general multi-dimensional pixel– a hyperpixel. In this Lab, for consistency of language, we use “voxel” whenever we areconsidering the area of interest (or object) and we use “pixel” whenever we are consideringthe radiograph.

Square Voxel Array. The total number of voxels is N = n2. This is a simplification.There is no theoretical or application limitation that requires the use of a square grid ofsquare voxels. In general the area of interest can be partitioned in any finite fashion. Wehave chosen an n× n grid for computational and visual simplicity.

Voxel and Pixel Size. Voxels are of unit size. This is a choice that makes the com-putations and discussion simpler. In quantitative applications, one must be careful to usecorrect units. Pixel size is given in terms of the voxel size by specifying the scale factorScaleFac. This units simplification does not alter the quality or details of computationalresults – quantitative application results differ only by a multiplicative constant. One canthink of this as working in unfamiliar units.

Limitations on n and m. Both n and m must be non-negative even integers. This isonly because this makes the code tomomap.m easier to compose and compute. If the studentattempts to use an odd integer for n or for m the code will stop and report an error message.

Radiographs Arrangement. Each radiograph has the same number of pixels, m. Thisis another simplification that makes computations simpler. In practice, radiographic anglesare imposed by either rotating the object or rotating the source-detector combination, andtaking individual radiographs at separate times. There is nothing special about measuringradiographic angles in degrees east of south. This is just the way the module code tomomap.mwas written. Notice that for each radiograph, the center point of the object grid projectsperpendicularly to the center point of the radiograph.

Visual Setup. Keep in mind that in applications the detectors are much more distantfrom the area of interest as would be indicated in the figures. All radiographs are typicallytaken equidistant from the object.

106

6.2 Radiography and Tomography Lab 2

107

108


An Abbreviated View of Radiography

Transmission Radiography is the process of measuring and recording changes in a high-energy particle beam (X-rays, protons, neutrons, etc.) resulting from passage through anobject of interest.

Source

x−ray

x−ray beam

Incident Region of InterestDetector

x−ray beam

Transmitted

E(0, y, z) E(Dx, y, z)

ρ(x, y, z)

Figure 1: Typical radiographic experiment. The function E measures intensity of the x-raybeam.

Radiographic Scenarios and Notation

A single-view radiographic setup consists of an area of interest where the object will beplaced, and a single screen onto which the radiograph will be recorded. A multiple-viewradiographic setup consists of a single area of interest experimentally designed so that radio-graphs of this area can be recorded for different locations about the object.

The geometry of radiographic scenarios is illustrated in figures 1 and 2. The notation isas follows.

• Slice of region of interest: n by n array of voxels.• Total number of voxels in each slice is N = n2.• Each voxel has a width and height of 1 unit.• For each radiographic view we record m pixels of data.• The width of each pixel is ScaleFac. If ScaleFac = 1 then pixel width is the same as

voxel width.

1

109

total mass

along

total mass

Pixel

detector

region of interest

Voxel j

beam path k

Sk =∑N

j=1 Tkjxj

fraction of voxel jin beam path k in voxel j

kbeam path k

Figure 2: Object space and radiograph space discretization.

• Number of radiographic angles (views): a.• Total number of pixels in the radiograph: M = am• Angle of the ith view (measured in degrees east of south): θi• Object mass at voxel j is xj• Recorded radiograph value at pixel k is bk

In this Lab we will be constructing matrix representations of the radiographic transfor-mation operators for example scenarios. Recall that an object can be represented by anN -dimensional vector x and a set of radiographs can be represented by an M -dimensionalvector b. What will be the size of the corresponding matrix operator that maps object vectorx to radiograph vector b? Recall the definition of the matrix operator from the radiographydiscussion. Notice that particular values for x and b are not necessary for computing thematrix of the radiographic transformation!

2

110

Figure 1: The geometry of a single view

radiographic transformation.

xn+1

x2

x3

xn

. ..

.

.

.

. . . xn2

. . .x1

b1 b2 bm

x2n

x2n+1

x2n−1xn−1

Angles are measured in

degrees east of south

Figure 2: The Geometry of a multiple view

radiographic transformation showing view

1, view 2, and view a.

b2m

bm+1

ba·m

xn+1 x2n+1x1

x2n−1

x2n

xn−1

θ2

x2

x3

xn

.. .

.

.

.

. . . xn2

. . .

b1 b2 bmbm+2

A First Example

Let’s look at a specific scenario. For the setup pictured below, we have:

x3

x2 x4

b2

b1

x1

• Total number of voxels: N = 4 (n = 2).

• Total number of pixels: M = m = 2

• ScaleFac =√

2

• Number of views: a = 1

• Angle of the single view: θ1 = 45◦

Recalling that Tkj is the fraction of voxel j which projects perpendicularly onto pixel k,

the matrix associated with this radiographic setup is T =

[1/2 1 0 1/21/2 0 1 1/2

]. Be sure and

check this to see if you agree! Hence, for any input vector x, the radiographic output is

b = Tx. Find the output when the object is the vector x =

100510

.

3

111

.

4

112

In Class Example:

We will first do an example together in class. Suppose you have the setup where

• Height and width of image in voxels: n = 2 (Total voxels N = 4)

• Pixels per view in radiograph: m = 2

• ScaleFac =√

2


• Angle of the views: θ1 = 45◦, θ2 = 135◦

1. Sketch this setup.

2. Calculate the matrix associated with the setup.

3. Repeat step (b) using the code tomomap. You will need to use the function tomomap.mfound in blackboard under Lab 2 code.

5

113

.

6

114

Radiography and Tomography Lab 2

Now it’s your turn! Whatever you do not finish, you will need to finish outside ofclass. The due date for this lab is 1 week after we have started it, unless theinstructor chances the due date. If you are working with a partner, please onlysubmit ONE lab with both of your names! There are 2 sides to this lab. CompleteParts I and II.

Note: Some of the following exercises will ask you to use Matlab or Octave to computeradiographic transformations. You will need to use the function tomomap.m found in black-board under Lab 2 code.

Part I: Finding the matrix associated with a radiographic transformation, andusing the matrix to determine radiographic outputs.

Your instructor may assign one or more of the following scenarios.

1. Suppose you have the setup where

• Height and width of image in voxels: n = 2 (Total voxels N = 4)• Pixels per view in radiograph: m = 2• ScaleFac = 1• Number of views: a = 2• Angle of the views: θ1 = 0◦, θ2 = 90◦

(a) Sketch this setup.

(b) Calculate the matrix associated with the setup.

(c) Find the radiographs of the following objects.

x=6

4 3

1

y=1

0 1

0

2. Suppose you have the setup where

• Height and width of image in voxels: n = 2 (Total voxels N = 4)• Pixels per view in radiograph: m = 4• ScaleFac =

√2/2

• Number of views: a = 1• Angle of the views: θ1 = 45◦

(a) Sketch this setup.1

1continue on next page

7

115


(c) Repeat step (b) using the code tomomap. How you can use this is first by openingthe mfile tomomap.m in MatLab or drag it into Octave. The code to do this isT=full(tomomap(n,m,th,ScaleFac)) so for example if n = 4, m = 4, and theangles are 0, 20, and 90, with a scale factor of 1, you would writeT=full(tomomap(4,4,[0 22.5 90],1))Note to write

√2, write sqrt(2).

3. Extra Credit Example: (this one requires some geometric maneuvering )Suppose you have the setup where

• Height and width of image in voxels: n = 2 (Total voxels N = 4)• Pixels per view in radiograph: m = 2• ScaleFac = 1• Number of views: a = 2• Angle of the views: θ1 = 0◦, θ2 = 45◦

(a) Sketch this setup.


(c) Repeat step (b) using the code tomomap. How you can use this is first by openingthe mfile tomomap.m in MatLab or drag it into Octave. The code to do this isT=full(tomomap(n,m,th,ScaleFac)) so for example if n = 4, m = 4, and theangles are 0, 20, and 90, with a scale factor of 1, you would writeT=full(tomomap(4,4,[0 22.5 90],1))

Part II: Radiographs of Linear Combinations of Objects

Take the two objects in exercise 1 above to be x (left object) and y (right object).

x=6

4 3

1

y=1

0 1

0

For two of the assigned transformations from Part I answer the following questions.

1. What is the radiograph of the object 3x?2. What is the radiograph of the object 0.5y?3. What is the radiograph of the object 3x + 0.5y?4. What do you notice? Can you generalize your observations?

8

116

7 Linear Transformations

7.1 Motivation for Linear Transformations

Terminology: A Transformation is just another word for . This function willtransform one space to another space. In Calc 3, this occurred when we used a changeof variables. For example, we can transform rectangular coordinates to polar through atransformation. In Figure 2 we see that this change of variables transformation takes roundthings and makes them rectangular.7

T

r

θ

x

y

Figure 2:

Another type of transformation is when we project from one space to a lower dimensionalspace. For example, we can project a points in R2 to their component along the y axis as inFigure 3.

T y

x

y

Figure 3:

In Lab #2, we found that when we apply a radiographic transformation to a linear combi-nation, αu+ βv, we get a linear combination of radiographs, αTu+ βTv out. This propertyis useful because we may wonder what is an object made of. If we know part of what isbeing radiographed (and what the corresponding radiograph should be), we can subtractthat away to be able to view the radiograph of the part we don’t know.

7Some Images from Heather Moon and Beezer’s A First Course in Linear Algebra

117

Example 7.1. Suppose we expect an object to look like what we see on the left in Figure 4but the truth is actually like we see on the right in Figure 4.

Figure 4:

We then expect that the radiograph that comes from an object, xexp like we see on the leftin Figure 4 to be a certain radiograph, call it bexp. But, we take a radiograph of the actualobject, x, and we get the radiograph b. Now, we know for sure that the expected parts arein the object. We can then remove bexp from the radiograph so that we can dig deeper intowhat else is present. Thus, we want to know about the radiograph, bunexp, of the unexpectedobject, call this xunexp, that is present. So we compute this radiograph like this:

Another reason this might be helpful comes in possible changes in an object.

Example 7.2. Suppose you radiograph an object as in Figure 4 and find that the radiographis bexp, but weeks later, you radiograph the same object (or so you think) and you get aradiograph that is 1.3 times the radiograph bexp. This could mean that the object now looksmore like one of the objects we see in Figure 5.

Figure 5:

Again, we can see that the density is proportionally larger, possibly meaning the object grew.

118

Being able to perform the type of investigation above with our radiographic transformationis very useful. It turns out that this property is useful beyond the application of radiography,so useful that we define transformations with this property.

7.2 Introduction to Linear Transformations

Definition 7.1. Let V and W be vector spaces and let T be a function that maps vectorsin V to vectors in W . We say that T : V → W is a linear transformation if both of thefollowing are satisfied:1) For any vectors u1, u2 ∈ V , T (u1 + u2) =2) For any scalar α and vector u ∈ V , T (αu) =

Note: You can combine these two steps into a 1-Step Check by showing thatT (αu1 + βu2) = .

Terminology: We can also call T a linear or a homomorphism.

Method: To check if T : V → W is a linear transformation, take 2 arbitrary elements in V,and an arbitrary scalar and check that 1) T (u1+u2) = T (u1)+T (u2), and 2) T (αu1) = αT (u1)

Note: The radiographic transformation is an example of a linear transformation.

Example 7.3. Consider the vector space V = R3. Determine whether T is a linear trans-

formation when T : V → W is defined as follows. T

abc

=

(a+ b+ c

).

119

Example 7.4. Is the map f : R→ R where f(x) = 2x+ 1 a linear transformation?

(a) Yes, and I am very confident(b) Yes, but I am not very confident(c) No, but I am not very confident(d) No, and I am very confident

Example 7.5. f(x) = mx+ b is a linear transformation only if what is true:

(a) m = 0 and b = 0(b) m = 0 and b can be anything(c) b = 0 and m can be anything(d) m and b can be anything!(e) I like cats!

Theorem 7.1. Let V and W be vector spaces. If T : V → W is a linear transformation,then T (0) = .

Proof. Let V and W be vector spaces and let T : V → W be a linear transformation. Noticethat 0 ∈ V and 0 ∈ W . (Note also that these two vectors called 0 need not be the samevector.) We also know that, for any scalar α, T (0) = T (α0) = αT (0). We can use thisequation to solve for T (0) and we get that either α = 1 or T (0) = 0. Since we know that αcan be any scalar, T (0) = 0 must be true.

Quick Check for Linear Transformations:We can determine whether or not T (0) = . If not, T cannot be linear.If yes, you still have to check that T (u1 + u2) = T (u1) + T (u2) and T (αu) = αT (u).

Example 7.6. Determine which of the following is a linear transformation.

a) Define f : R3 → R2 by f(v) = Mv + x, where M =

(1 2 11 2 1

)and x =

(10

)

120

b) Define T : P2 → P1, where by T (ax2 + bx+ c) = 2ax+ b.

Example 7.7. Define T (v) = Av, where A =

[−1 00 1

]. Then T (v).... [Hint: let v =

[xy

]]

(a) reflects v about the x-axis.(b) reflects v about the y-axis.(c) rotates v clockwise π

2radians about the origin.

(d) rotates v counterclockwise π2

radians about the origin.(e) None of the above

7.2.1 Motivation for Matrix Forms of Linear Transformations:

Functions are relatively easy for us to understand, but it would be nice to be able to encodethem into a matrix form so computers can understand and use them better. Matrices tendto be a good way to store information in a computer. They are also, at times, easier to workwith (as they were when we solved systems of equations). Coding a linear transformationbased on the formula can at times be very tedious. So, we want to be able to use matricesas a tool for linear transformations as well. Let’s look at an example that suggests that thismight be a good idea.

Example: Define T : R2 → R3 by Tx = Mx, where M is a 3 × 2 matrix. T is a lineartransformation. We can show this using properties of matrix multiply. Let x, y ∈ R2 and letα, β be scalars. Then

T (αx+ βy) = M(αx+ βy)

= Mαx+Mβy

= αMx+ βMy

= αTx+ βTy.

This example shows us that matrix multiply defines a linear transformation. So, what if wecan define all linear transformations with a matrix multiply? That would be really great!Issue: We can’t just multiply a vector, in say P2, by a matrix. What would that mean?

121

Key: Represent it using a Coordinate Vector

122

7.2.2 ICE 4 -Linear Transformations

1) Suppose “N” on the left is written in regular 12-point font. Find a matrix A that willtransform N into that letter on the right, which is written in ‘italics’ in 16-point fond.

A =

123

2) After class, Eva, Archer and their kitty friends Gilbert and Mako are wondering how

letters placed in other locations in the plan would be transformed under A =

[1 1/30 4/3

]. If

an “E” is placed around the “N,” the kitties argued over four different possible results forthe transformed E’s on the next page. Which choice below is correct, and what? If none ofthe four options are correct, what would the correct option be, and why?

124

125

126

8 Coordinate Vectors

In our typical 3D space, we talk about vectors that look like In our typical 3D space, we

talk about vectors that look like

xyz

. And we see that the coordinates are x, y, and z

respectively. What this means is that the vector points from the origin to a point that is xunits horizontally, y units vertically, and z units up from the origin (as in the Figure below.The truth is that when we say that, we are assuming that we are working with the standardbasis for R3. (This makes sense because, it is the basis that we usually think about, x-axisperpendicular to the y-axis, forming a horizontal plane and the z axis perpendicular to thisplane.)

xyz

y

x

z

So, using the standard basis E = {

100

,

010

,

001

}, we can write

xyz

= x

100

+ y

010

+ z

001

.

Notice that the coordinates are the scalars in this linear combination. That’s what we mean

by coordinates. Notice the coordinates of the vector v =

123

in the standard basis are

, , & respectively.

127

Now, let us consider a different basis for R3, B =

111

,

101

,

001

. Let’s find

the coordinates for v =

123

in this new basis. These are found by finding the scalars

α1, α2, and α3 so that

123

= α1

111

+ α2

101

+ α3

001

. Going through the

motions to solve for α1, α2, and α3, we find that α1 = 2, α2 = −1, and α3 = 2.

Thus, we can represent the vector v in coordinates according to the basis B as

[v]B =

2−1

2

.

Notice that we indicate that we are looking for coordinates in terms of the basis B by usingthe notation, [v]B. This means if we are given a vector space V and bases B1 = {v1, v2, . . . , vn}

and B2 = {u1, u2, . . . , un}, then if w = α1v1 + α2v2 + . . .+ αnvn, we have [w]B1 =

α1

α1...αn

.

But, if w = β1u1 + β2u2 + . . .+ βnun, we have [w]B2 =

β1

β1...βn

.

In our example from R3, notice [v]B =

2−1

2

but with respective to the standard basis, [v]E3 =

123

.

Important: Notice also that coordinate vectors look like vectors in Rn for some n.

128

Let’s do another example from a vector space that isn’t already in vector form:

Suppose we have v = 3 + x − x2 ∈ P2. Given the standard basis for P2,B = {1, x, x2}, wecan think of v = 3 + x − x2 = 3(1) + 1(x) − 1(x2). We define 3, 1,−1 as the B-coordinatesof v. Now we can represent 3 + x− x2 as a vector in R3:

This vector is called the B−coordinate vector of v or the coordinate vector relative to B andis denoted as [v]B.

Example 8.1. What is the coordinate vector of 3 + x− x2 relative to the standard basis ofP3, S = {1, x, x2, x3}?

Key Idea: I like to think of the coordinate vector of v relative to a basis as the “instructionmanual” for how to put the basis elements together to create v.

Example 8.2. Let V = {ax2 + (b)x + (c) ∈ P2| a + b − 2c = 0}. One basis for V isB = {−x2 + x, 2x2 + 1} (check this on your own).

a) Since v = 3x2 − 3x ∈ V , we can write v as a coordinate vector with respect to B.

b) Now suppose we know the coordinates for a vector w are [w]B =

(2−1

). We can use

this to determine what w actually is in V.

129

Let’s now consider representations of a vector when we view the vector space in terms of two

different bases.

Example 8.3. Let V =

{(a b− a

a+ b a+ 2b

)∣∣∣∣ a, b ∈ R}. Archer has found one base for

V : B1 = {[1 −11 1

],

[0 11 2

]}. Eva has used the basis to construct another basis for V:

B2 = {[1 02 3

],

[1 −20 −1

]}.

a) Find a vector such that [w]B1 =

[12

].

b) Write find the coordinates of the vector your found in terms of Eva’s basis.(That is findthe instruction manual to build it from Eva’s basis.)

130

Reading Example:

Example 8.4. Let V = {ax2 + (b− a)x+ (a+ b)|a, b ∈ R}. Find a basis for V and find thecoordinate vector for v = 3x2 + x+ 7 ∈ V with respect to this basis.

V = {a(x2 − x+ 1) + b(x+ 1)| a, b ∈ R} = span{x2 − x+ 1, x+ 1}.So a basis for V is

B = {v1 = x2 − x+ 1, v2 = x+ 1}.This means that dimV = 2 and so vectors in V can be represented by vectors in R2. Noticethat v = 3x2 + x+ 7 ∈ V . You actually should be able to check this (a = 3, b = 4). We canwrite v in terms of the basis of V as v = 3v1 + 4v2. We can check this as follows

3v1 + 4v2 = 3(x2 − x+ 1) + 4(x+ 1) = 3x2 + x+ 7.

Thus, the coordinate vector for v is [v]B =

(34

).

131

132

9 Matrix Forms of Linear Transformations:

Now that we have discussed coordinate vectors in more detail, let’s go back to the reasonwhy we went down that bunny trail.

Recall, we wanted to be able to define a linear transformations with a matrix, but the issueis that we can’t just multiply a vector, in say P2, by a matrix.

The Key is to use Coordinate Vectors!

Given a basis for V , we are able to represent any vector v ∈ V as a coordinate vector inRn, where n = dimV . Suppose B = {v1, v2, . . . , vn} is a basis for V , then we find thecoordinate vector [v]B by finding the scalars, αi, that make the linear combination v =α1v1 + α2v2 + . . .+ αnvn and we get

[v]B =

α1

α2...

αn

∈ Rn.

After our next section, we will talk a little bit more about the theory behind why we con-struct the matrix of a linear transformation (and how we create a linear transformation thatchanges basis representation) later.

Our goal is to create a matrix that does what we want: [T (v)]BW = M [v]BV .

So given a Linear Transformation T : V → W , we first need to convert vectors in V intovectors in Rn. To do this we create the coordinate vector for each basis element in thestandard basis. We then take the image of each basis element from V, T (v) and write it asa coordinate vector in BW . We do this one basis element at a time and construct a matrixso that

M [v1]BV = [T (v1)]BWM [v2]BV = [T (v2)]BW

...

M [vn]BV = [T (vn)]BW .

Notice that

133

[v1]BV =

100...0

and [v2]BV =

010...0

, etc....

So, M [v1]BV gives the first column of M , M [v2]BV gives the second column of M , . . ., andM [vn]BV gives the nth column of M . Thus, we have that

M =

| | |

[T (v1)]BW [T (v2)]BW . . . [T (vn)]BW

| | |

Steps for finding the Matrix Form of a Linear Transformation:

To find a matrix form for the linear transformation T : V → W :

Step 1: Find bases for the domain (in this example, this is V) and co-domain (in thisexample, this is W).

Step 2: For each element of the basis of the domain (in this example, this is V), BV , findits image after apply T to it.

Step 3: Write the image as a coordinate vector with respect to the basis for co-Domain (inthis example, this is W), BW .

Step 4: Create M using the coordinate vectors you created in Step 3: M =

Useful, but optional step: Check that your matrix works to see if you get the same thingas you would with the original transformation.

134

Example 9.1. Find a matrix form for the linear transformation T : P1 → P2 where T (ax+b) = ax2 + (a+ b)x− b.

First of all, since we are going from a domain of V = P1 which is a dimensionalvector space to a codomain of W = P2 which is a dimensional vector space. Thematrix we create should have the dimensions × .

Step 1: Find bases for V and W.BV =

BW

Step 2: For each element of the basis of V, BV , find its image after apply T to it.

Step 3: Write the image as a coordinate vector with respect to the basis for W, BW .

Step 4: Create M using the coordinate vectors you created in Step 3: M =

Check that your matrix works to see if you get the same thing as T (8x+ 4)

135

Example 9.2. Let’s try to find a matrix representation for the linear transformations inExercise 7.2. That is, define a matrix representation for the linear transformation: T :

R3 → R is defined as follows. T

abc

=

(a+ b+ c

).

Example 9.3. The linear transformation T : R2 → R2 where T (x, y) = (x + 2y, x − 2y),

can be written as a matrix transformation T (x, y) = A

[xy

]where

(a) A =

[x 2yx −2y

]

(b) A =

[1 21 −2

]

(c) A =

[1 −21 2

]

(d) It can’t be written in matrix form.(e) Archer is my favorite.

Example 9.4. The linear transformation T (x, y) =

[−1 00 1

], can be written as

(a) T (x, y) = (x, y)(b) T (x, y) = (y, x)(c) T (x, y) = (−x, y)(d) T (x, y) = (−y, x)(e) None of the above

136

Things get a little stranger when you look at subspaces. When you have this type of situation,you must first find a basis for the subspace of V.

Example 9.5. Find a matrix representation for the Linear Transformation

h : V → P1, where V =

{(a b c0 b− c 2a

)∣∣∣∣ a, b, c ∈ R}⊆M2×3 and

h

(a b c0 b− c 2a

)= ax+ c.

137

9.0.3 Radiographic Connection

When working with radiographic transformations, we found a matrix using tomomap. But, our

objects weren’t vectors in RN that could be multiplied by the matrix we found. Let’s use the above

information to explore, through an example, what was really happening. Let V = I2×2, the space

of 2 × 2 objects. Let T be the radiographic transformation with 6 views having 2 pixels each.

This means that the codomain is the set of radiographs with 12 pixels. To figure out the matrix

M representing this radiographic transformation, we first change the objects in V to coordinate

vectors in R4 via the isomorphism T1. So T1 is defined by:

x1

x2

x3

x4

x1

x2

x3

x4

V

T1

R4

After performing the matrix multiply, we will change from coordinate vectors in R12 back to radio-

graphs via T2. So T2 is defined by by:

b1b2b3b4b5b6b7b8b9b10b11b12

WR12

b2b1b4b3b6b5b8b7b10b9b12b11

T2

Our radiographic transformation is then represented by the matrix M (which we called T inthe labs). M will be a 12× 4 matrix determined by the radiographic set up. We’ve computedM several times in previous labs, but the real mathematics was all a bit hand-wavy and so nowwe see that really, what we have is that T maps from V to W by taking a side excursion throughcoordinate spaces and doing a matrix multiply.

Note: Typically, to simplify notation, we write T (v) = Mv when we actually mean [T (v)]BW =

M [v]BV . This is understandable by a mathematician because we recognize that when two spaces

are isomorphic, they are “essentially” the same set. The only difference is that the vectors look

different. In this class, we will maintain the notation discussed in this section being aware that this

is just to get used to the ideas before relaxing our notation.

138

9.0.4 ICE 5 -Matrix Linear Transformations

1. Let’s try to find a matrix representation for the linear transformation in Exercise7.3.That is, Define a matrix representation for the linear transformation: T : P2 → P1,where T (ax2 + bx+ c) = 2ax+ b.

139

2. Let V = {ax2 + (b − a)x + (a + b)| a, b ∈ R} and let W = M2×2. Consider the

transformation T : V → W defined by T (ax2 +(b−a)x+(a+b)) =

(a b− a

a+ b a+ 2b

).

Construct the Matrix Representation of T. Remember you will need to find bases forV and W!

140

10 Properties of Transformations

10.1 Injective and Surjective Transformations

Recall, a linear transformation is just a function with special properties. 8

In Lab # 3, we will see several properties of linear transformations that are useful to recog-nize. We will see that it is possible for two objects to produce the same radiograph. This canbe an issue in the case of brain radiography. We would like to know if it would be possiblefor an abnormal brain to produce the same radiograph as a normal brain. We also will seethat it was possible to have radiographs that could not be produce from any object. Thisbecomes important in being able to recognize noise or other corruption in a given radiograph.Again, it turns out that these properties are not only important in radiography. There aremany other scenarios (some application based and some theoretical) where we need to knowif a transformation has these properties. So, let’s define them.

Due to the press for time in this course and the “applied” focus for the course, we will notdwell as much on proving transformations have these properties. Later we will have a morecomputational way to check these properties. I will provide some of the arguments thoughfor completeness and for reference.

Definition 10.1. Let V and W be vector spaces. We say that the transformation T : V → Wis injective if the following property is true:Whenever T (u) = T (v) it must be true that = .

A transformation that is injective is also said to be one-to-one (1-1) or an injection.

Idea:

8Image used with permission from https://www.probabilitycourse.com

141

https://www.probabilitycourse.com

Process to show Injectivity: Start with two arbitrary elements in the range of T andsuppose they are equal, that is suppose T (u) = T (v) and work backwards to see if u = v. Inother words, can we find two possible input values that map to the same output? If yes, Tis NOT injective.

Notice, in Figure 8, we see that the transformation represented on the left is 1-1, but thetransformation represented on the right is not because both v1 and v2 map to w4, but v1 6= v2.

v1

v4

v5

v6

v3

v2 w3

w4

w5

w6

w7

w2

w1 v1

v4

v5

v6

v3

v2 w3

w4

w5

w6

w7

w2

w1

Figure 6:

Example 10.1. Let T : R2 → M3×2 be the linear transformation defined by T

(ab

)=

a −bb a+ b0 −a

. Determine whether it is one-to-one.

Proof. T is injective/(one-to one). Notice that if

T

(ab

)= T

(cd

)

Then

a −bb a+ b0 −a

=

c −dd c+ d0 −c

Matching up entries, gives us a = c,−b = −d, b = d, a + b = c + d, 0 = 0, and −c = −d.Thus, (

ab

)=

(cd

).

So, by definition, T is one-to-one.

142

Example 10.2. Let T : P2 → R2 be defined by T (ax2 + bx+ c) =

(a− bb+ c

). Notice that T

is a linear transformation. Is T an injection?

Proof. T is not injective, notice that both x+ 4 and 5−x2 map to

[−15

], that is T (x+ 4) =

T (5− x2) =

[−15

].

But how could we determine this? Here is another way. Suppose T (ax2 + bx + c) =

T (ex2 + fx+ g), this implies (after apply T) that

[a− bb+ c

]=

[e− ff + g

].

This means (after matching the entries) that

{a− b = e− fb+ c = f + g

We can set up the augmented equation to solve this:

(1 −1 0 e− f0 1 1 f + g

)

Notice we have a free variable which means we have the freedom to play a bit. At this point,we should be suspicious that this transformation is NOT one-to-one, but we need to find anoutput that has 2 different inputs that can map to it. So we can pick any values for e, f,and g like e = 1, f = 2, g = 3 (but they could be anything.)

Now we need to solve:

{a− b = e− fb+ c = f + g

⇒{a− b = 1− 2

b+ c = 2 + 3⇒{a− b = −1

b+ c = 5

So now we have this system to solve:(1 −1 0 −10 1 1 5

)

Solving this gives usa = −1 + bb = free baby!c = 5− b.

So we can pick two different values for b. In the example above we picked b = 1 which meansa = 0, c = 4 which gives the vector x+ 4. We then picked b = 0 which means a = −1, c = 5which gives us the vector −x2 + 5.

143

Definition 10.2. Let V and W be vector spaces. We say that the transformation T : V → Wis surjective if every element in W is mapped to. That is, if w ∈ W , then there exists av ∈ V so that = .

A transformation that is surjective is also said to be onto or a surjection.

Idea: Dr. Harsy is going to do her absolute worst to present you with the most obscureelement of the codomain. Are you certain that this function will be able to map to thiselement no matter what Dr. Harsy presents you with?

Process to show surjectivity: Pick an arbitrary element in co-domain, find an elementthat maps to it.

Example 10.3. Consider the function f : R→ R where f(x) = x2. Note that this is neitherinjective nor onto (and isn’t a linear transformation either). Why?

Example 10.4. Determine which of the maps below are surjective and which maps are injective.

v1

v4

v5

v3

v2 w3

w4

w5

w2

w1

v6

v1

v4

v5

v3

v2 w3

w4

w5

w2

w1

v6

Figure 7:

v1

v4

v5

v6

v3

v2 w3

w4

w5

w6

w7

w2

w1 v1

v4

v5

v6

v3

v2 w3

w4

w5

w6

w7

w2

w1

Figure 8:

144

Example 10.5. Let T : R2 → M3×2 be defined by T

(ab

)=

a −bb a+ b0 −a

. We already

showed this linear transformation was one-to-one in Ex 10.5. Is it onto?

Proof. Notice that,

w =

1 11 11 1

∈M3×2.

But, there is no v ∈ R2 so that T (v) = w because every output has a 0 in the (3, 1) entry.

Example 10.6. Let T : R2 → P1 be defined by T

(ab

)= 2ax + b. Is this map 1-to-1 or

onto?

Proof. T is 1-to-1:

Suppose T

(ab

)= T

(cd

). Then implies 2ax + b = 2cx + d. Matching up like terms

gives us that

{2a = 2c

b = dSolving this, gives us a = c and b = d. That is

(ab

)=

(cd

).

Therefore T is one-to-one.

T is also onto:We want to pick an arbitrary representative of a vector in the codomain, P1, say ax + b

and we want to find a vector in the domain, R2,

[cd

]such that T (

[cd

]) = ax + b. Let

w = ax + b ∈ P1, and Notice also that v =

[a2

b

]∈ R2 maps to ax + b. Thus T (v) = w.

Therefore, T is onto.

145

146

10.1.1 ICE 5 -Injective and Surjective Linear Transformations

For the next 4 questions, try to construct the following transformations. If you can, justifyor prove that your transformation has the property you claim it has or give an argumentwhy it is impossible.

1. If possible, create a transformation that maps from R3 to P1 that is surjective.

2. If possible, create a transformation that maps from R3 to P1 that is injective.

147

3. If possible, create a transformation that maps from P1 to R3 that is injective.

4. If possible, create a transformation that maps from P1 to R3 that is surjective.

5. What do you notice? Given T : V → WIn order for T to be injective, the dim V needs to be than dim W.In order for T to be surjective, the dim V needs to be than dim W.

148

10.2 Bijections and Isomorphisms

We see above that sometimes a linear transformation can be both injective and surjective.In this subsection we discuss this special type of linear transformation.

Definition 10.3. We say that a linear transformation, T : V → W , is bijective if T is bothinjective and surjective. We call a bijective transformation a bijection or an isomorphism.

Definition 10.4. Let V and W be vector spaces. If there exists a bijection mapping betweenV and W , then we say that V is isomorphic to W and we write V ∼= W .

Example 10.7. Notice that 10.5 we found that T was a bijection. This means that P1∼= R2.

Notice also that dimP1 = dimR2. This is not a coincidence.

Lemma 10.1. Let V and W be vector spaces. Let B = {v1, v2, . . . , vn} be a basis for V .T : V → W is an injective linear transformation if and only if {T (v1), T (v2), . . . , T (vn)} isa linearly independent set in W .

Proof. As with every proof about linear dependence/independence, we will assume the fol-lowing linear dependence relation is true. Let α1, α2, . . . , αn be scalars so that

α1T (v1) + α2T (v2) + . . .+ αnT (vn) = 0.

Then because T is linear, we know that

T (α1v1 + α2v2 + . . .+ αnvn) = 0.

But, we also know that T (0) = 0. That means that

T (α1v1 + α2v2 + . . .+ αnvn) = T (0).

And, since T is 1-1, we know that (by definition)

α1v1 + α2v2 + . . .+ αnvn = 0.

Finally, since B is a basis for V , B is linearly independent. Thus,

α1 = α2 = . . . = αn = 0.

Thus, {T (v1), T (v2), . . . , T (vn)} is linearly independent.Now suppose that T is linear, let B = {v1, v2, . . . , vn} be a basis for V , and suppose{T (v1), T (v2), . . . , T (vn)} ⊂ W is linearly independent. Suppose u, v ∈ V so that T (u) =T (v). So, T (u− v) = 0. Since u, v ∈ V , there are scalars α1, α2, . . . , αn and β1, β2, . . . , βn sothat

u = α1v1 + α2v2 + . . .+ αnvn and v = β1v1 + β2v2 + . . .+ βnvn.

149

ThusT ((α1 − β1)v1 + (α2 − β2)v2 + . . .+ (αn − βn)vn) = 0.

This leads us to the linear dependence relation

(α1 − β1)T (v1) + (α2 − β2)T (v2) + . . .+ (αn − βn)T (vn) = 0.

Since {T (v1), T (v2), . . . , T (vn)} is linearly independent, we know that

α1 − β1 = α2 − β2 = . . . = αn − βn = 0.

That is, u = v. So, T is injective.

Note: Notice that Lemma 10.1 tells us that if V is n-dimensional then basis elements of Vmap to basis elements of an n-dimensional subspace of W . In particular, if dimW = n also,then we see a basis of V maps to a basis of W . This is very useful and leads to the followinguseful result.

Theorem 10.1. Given (finite dimensional) vector spaces V and W , then V ∼= W if andonly if dimV = dimW .

Proof. Suppose dimV = dimW . Suppose also that a basis for V is BV = {v1, v2, . . . , vn}and a basis for W is BW = {w1, w2, . . . , wn}. Then we can define T : V → W to be thelinear transformation so that

T (v1) = w1, T (v2) = w2, . . . , T (vn) = wn.

We will show that T is an isomorphism. Now, we know that if w ∈ W , then w = α1w1 +α2w2 + . . . + αnwn for some scalars α1, α2, . . . , αn. We also know that v = α1v1 + α2v2 +. . .+ αnvn ∈ V . Since T is linear, we can see that

T (v) =T (α1v1 + α2v2 + . . .+ αnvn)

=α1T (v1) + α2T (v2) + . . .+ αnT (vn)

=α1w1 + α2w2 + . . .+ αnwn = w.

Thus, T is onto. Now, suppose that T (v) = T (u) where v = α1v1 + α2v2 + . . . + αnvn andu = β1v1 + β2v2 + . . .+ βnvn are vectors in V . Then we have

T (α1v1 + α2v2 + . . .+ αnvn) =T (β1v1 + β2v2 + . . .+ βnvn)

α1T (v1) + α2T (v2) + . . .+ αnT (vn) =β1T (v1) + β2T (v2) + . . .+ βnT (vn)

α1w1 + α2w2 + . . .+ αnwn =β1w1 + β2w2 + . . .+ βnwn

(α1 − β1)w1 + (α2 − β2)w2 + . . .+ (αn − βn)wn = 0.

150

Notice that this last equation is a linear dependence relation for the basis BW . Since BW islinearly independent, we know that

α1 − β1 = 0

α2 − β2 = 0

...

αn − βn = 0.

That is to say u = v. Thus, T is injective. And, therefore, since T is both injective andsurjective, T is an isomorphism. Now, since there is an isomorphism between V and W , weknow that V ∼= W .Now we will prove the other direction. That is, we will show that if V ∼= W then dimV =dimW . First, let us assume that V ∼= W . This means that there is an isomorphism,T : V → W , mapping between V and W .Suppose, for the sake of contradiction, that dimV 6= dimW . Without loss of generality,assume dimV > dimW . (We can make this assumption because we can just switch V ’sand W ’s in the following argument and argue for the case when dimV < dimW .) LetBV = {v1, v2, . . . , vn} be a basis for V and BW = {w1, w2, . . . , wm} be a basis for W . Thenm < n. We will show that this cannot be true. Lemma 10.1 tells us that since T is one-to-one, the basis BV maps to a linearly independent set {T (v1), T (v2), . . . , T (vn)} with nelements. But by Theorem 5.1, we know that this is not possible. Thus, our assumptionthat n > m cannot be true. Again, the argument that tells us that n > m also cannot betrue is very similar with V ’s and W ’s switched. Thus, n = m. That is, dimV = dimW .

Careful: If we allow the dimension of a vector space to be infinite, then this is not neces-sarily true. In this class, we restrict to discussions of finite dimensional vector spaces only.

This theorem gives us a tool for creating isomorphisms, when they exist. It also tells us thatisomorphisms exist between two vector spaces so long as they have the same .

Example 10.8. Suppose we have a linear transformation T : P2 → R2. Using dimen-sional analysis, which of the following properties could T have: One-to-One, Onto, and/orIsomorphism?

151

Example 10.9. Suppose we have a linear transformation matrix, A, what dimensions willA need to have for it to possibly represent a bijective linear transformation?

Example 10.10. Let V = M2×3 and W = P5. We know that V ∼= W because both are6-dimensional vector spaces. Indeed, a basis for V is

BV =

{(1 0 00 0 0

),

(0 1 00 0 0

),

(0 0 10 0 0

),

(0 0 01 0 0

),

(0 0 00 1 0

),

(0 0 00 0 1

)}

and a basis for W is BW = {1, x, x2, x3, x4, x5}. Create a bijection T that maps V to W .

Using the Theorem 10.1 we define T as follows

T

(1 0 00 0 0

)= , T

(0 1 00 0 0

)= , T

(0 0 10 0 0

)=

T

(0 0 01 0 0

)= , T

(0 0 00 1 0

)= , T

(0 0 00 0 1

)= .

Notice that if we have any vector v ∈ V , we can find where T maps it to in W . Since v ∈ V ,we know there are scalars, a, b, c, d, e, f so that

v = a

(1 0 00 0 0

)+b

(0 1 00 0 0

)+c

(0 0 10 0 0

)+d

(0 0 01 0 0

)+e

(0 0 00 1 0

)+f

(0 0 00 0 1

).

That is,

v =

(a b cd e f

).

Thus, since T is linear,

T (v) = a(1) + b(x) + c(x2) + d(x3) + e(x4) + f(x5) = a+ bx+ cx2 + dx3 + ex4 + fx5.

152


This lab will motivate some of the computational techniques we will use to determine prop-erties of transformations in addition to putting these properties into a real world context.

11.1 Radiography and Tomography Intro to Lab 3

Solutions from Lab 2

Operator A (from Example 1 in Lab 2):

The Transformation for this map is T =

1 1 0 00 0 1 10 1 0 11 0 1 0

.

Operator B (from Example 3 in Lab 2):


12

1 0 12

12

0 1 12

1 12

12

00 1

212

1

.

Operator C (from Example 4 in Lab 2):


0 12

0 012

12

0 12

12

0 12

12

0 0 12

0

.

b2

x1 x3

x2 x4

b 4b 3

b1

b4

x1 x3

x2 x4

b 1

b 2

b3

b 1

x1 x3

x2 x4b 4

b 3

b 2

Figure 9: Setups of Operators A, B, and C

153

Common Misconceptions at this Point in the Lab:

• Misconception: Each radiographic view corresponds to a different transformation.Truth: Multiple radiographic views are combined into a single radiographic image andis the product of a single transformation acting on our object. The scenario details(number and placement of views, voxelization of the object, etc.) and the domain andcodomain descriptions In×n(R) and Im×1(R) should make this clear.

• Misconception: Objects cannot have negative valued entries.Truth: Objects are vectors in I2×2(R), with entries in the field of Real numbers. Ourparticular application suggests that realistic objects must have non-negative entries.However, not all meaningful objects need be non-negative. For example, two non-negative objects, u and v, can have a difference object, d = u− v, which has negativeentries. And d is a very important object!

• Misconception: Radiograph pixel widths can be individually chosen.Truth: In our scenarios, each pixel has the same width as every other pixel. Objectvoxels have unit width and radiograph pixels have width ScaleFac.

• Misconception: Objects are matrices.Truth: The space of radiographic objects In×n(R) is isomorphic to Mn×n(R).

154


Radiography and Tomography: Lab #3

In this activity, you will explore some of the properties of radiographic transformations.

In Lab #2 you found several radiographic transformation operators. The object imageconsisted of four (or in one case, sixteen) voxels and was represented as a vector in R4.The radiographic image was represented as a vector in RM , one entry for each radiographicpixel. The price we pay for this representation is that we no longer have the geometry ofthe radiographic setup encoded in the representation. The use of representations in Rn is acomputational tool and not geometrically descriptive of vector spaces of images. We wantto reiterate that this is only a representation and that these images are not vectors in RM .Because of these new image representations, each transformation could be constructed as amatrix operator in MM×4(R).

In Class Example:

Consider the radiographic setup described below.

b2

x1 x3

x2 x4

b 4b 3

b1

• Height and width of image in voxels:n = 2 (Total voxels N = 4)• Pixels per view in radiograph: m = 2• ScaleFac = 1• Number of views: a = 2• Angle of the views: θ1 = 0◦, θ2 = 90◦

The transformation matrix for this map is T =

1 1 0 00 0 1 10 1 0 11 0 1 0

.

1

155


Answer the following questions about this radiographic setup, justifying your conclusions.

Task 1:

1. Are any nonzero objects invisible to this operator? If so, give an example. We say thatan object is nonzero if not all entries are zero. We say that an object is invisible if itproduces the zero radiograph.

2. Describe the set of all invisible objects. This could involve an equation that the entrieswould have to satisfy or a few specific objects that could be used to construct all othersuch objects. Be creative.

2

156


3. Show that the set of all invisible objects is a subspace of the vector space of all objects.

4. Find a set of objects that spans the space of all invisible objects. Is your set linearlyindependent? If not, find a linearly independent set of objects that spans the space ofinvisible objects. The linearly independent set you have found is a basis for the spaceof invisible objects.

3

157


Task 2:

5. Is it possible for two different objects to produce the same radiograph? If so, give anexample.

6. Choose any two objects that produce the same radiograph and subtract them. Whatis special about the resulting object?

4

158


Task 3

7. Are there radiographs (in the appropriate dimension for the problem) that cannot beproduced as the radiograph of any object? If so, give an example.

8. Describe the set of radiographs that can be produced from all possible objects. Thismay require similar creativity.

5

159


9. The set of all possible radiographs is a subset of a vector space. What is this vectorspace? Show that the set of all possible radiographs is a subspace of this vector space.

10. Find a basis for the set of all possible radiographs. That is, find a linearly independentset of radiographs that spans the space of possible radiographs.

6

160


Lab 3 Questions

Now it’s your turn! For each of the two setups (B and C) below, Repeat Tasks 1, 2, and 3of the Lab (the steps are reprinted on this page for your convenience). In looking at moreexamples you will get a better sense of the underlying principles at work. We will use yourobservations to develop deeper linear algebra theories. Whatever you do not finish in class,you will need to finish outside of class. The due date for this lab is 1 week after we havestarted it, unless the instructor chances the due date. If you are working with a partner,please only submit ONE lab with both of your names!

Setup B

b4

x1 x3

x2 x4

b 1

b 2

b3

The Transformation for this map is

T =

12

1 0 12

12

0 1 12

1 12

12

00 1

212

1

• Height and width of image in voxels:n = 2 (Total voxels N = 4)• Pixels per view in radiograph: m = 4• ScaleFac =

√2/2

• Number of views: a = 1• Angle of the views: θ1 = 45◦

Setup C

b 1

x1 x3

x2 x4b 4

b 3

b 2


0 12

0 012

12

0 12

12

0 12

12

0 0 12

0

.

• Height and width of image in voxels:n = 2 (Total voxels N = 4)• Pixels per view in radiograph: m = 2• ScaleFac = 2• Number of views: a = 2• Angle of the views: θ1 = 90◦, θ1 = 315◦

7

161


Questions for Each Setup

1. Are any nonzero objects invisible to this operator? If so, give an example. (See yourLab 3 handout for the definitions: nonzero object and invisible object.)

2. Describe the set of all invisible objects by finding a basis for this space.

3. Is it possible for two different objects to produce the same radiograph? If so, give anexample.

4. Choose any two objects that produce the same radiograph and subtract them. Whatis special about the resulting object?

5. Are there radiographs (in the appropriate dimension for the problem) that cannot beproduced as the radiograph of any object? If so, give an example.

6. Describe the set of radiographs that can be produced from all possible objects.

7. Find a basis for the space of all possible radiographs. That is, find a linearly indepen-dent set of radiographs that spans the space of possible radiographs.

8. Do you think this transformation is injective? Why or why not?

9. Do you think this transformation is onto? Why or why not?

Question: In this lab, you found bases for two different vector spaces (invisible objects, andpossible radiographs). What reasons might you be interested in these bases, in the contextof radiography?

Bonus question: Can you find a radiographic setup for square objects with 4 voxels so thatthere are no invisible objects? What properties must the associated linear transformationhave?

8

162

12 Transformation Spaces

In Lab # 3, we saw more properties of linear transformations that were useful to recognize other

than injectivity and surjectivity. Some radiographic transformations cannot “see” certain objects

or we say that those objects are “invisible” to the transformation. In this application, we really

want to know if something might be present yet invisible to the transformation we are using. In

the case of brain scans, it would be most unhelpful if we cannot see certain abnormalities because

they are invisible to the radiographic setup. If something we want to look for in a brain scan is

invisible to our current setup, we can adjust the setup to “see” the object we are looking for. Say,

for example, we know that the object on the right in Figure 10 is invisible to our radiographic

transformation. Then when it is present along with what we expect (Figure from Example 1 from

our Linear Tranformation notes), we get the same radiographic transformation bexp and we might

go along our merry way, not knowing that something unexpected is present in our object so that

instead of what is seen in our Example 1 figure, we actually have what is on the left in Figure 10.

Figure 10:

12.1 Nullspace

Wanting to know which objects are “invisible” to a transformation extends beyond the application

of Radiography and Tomography. So, we define the space of all “invisible” objects below.

Definition 12.1. The nullspace of a linear transformation, T : V → W , is the subet of Vthat map to 0 ∈ W . That is, null(T ) =

Sometimes this is called the Kernel of T, or ker(T).

Definition 12.2. We say that the nullity of a linear transformation, T , is the dimensionof the subspace null(T ).

To find the nullity: Find a basis for the nullspace (a spanning set that is linearly inde-pendent) and count the number of vectors.

163

Example 12.1. Let V = {ax2 + 2bx + b| a, b ∈ R} ⊆ P2. Define F : V → P1 by F(ax2 +2bx+ b) = 2ax+ 2b. Find the nullspace of F .

In Example 12.1, the nullity is because there are no elements in the basis of nullFand we say that F has a nullspace, or that the nullspace of F is trivial.

Example 12.2. Define f :M2×2 → R4 by f

(a bc d

)=

ab+ abc

. Find null(f).

In Example 12.2 the nullity is because there is element in the basis for the

nullspace.

Also notice that there is more than one element of the nullspace.

That means, since f

(0 00 1

)= f

(0 00 2

)=

(0 00 0

), but

(0 00 1

)6=(

0 00 2

),

f is not .

164

Note: The above examples are indeed examples of linear transformations (See Linear Trans-formation Notes).

Comment: The name “nullspace” seems to imply that this set is a .In fact, we have discussed basis and treated it as if it is a vector space in Lab 3 and otherexamples. The next theorem justifies this treatment.

Theorem 12.1. Given vector spaces V and W and a linear transformation T : V → W ,the nullspace null(T ) is a of V .

Proof. By definition of null(T ), we know that null(T ) ⊆ V . We also know that the zero vectoralways maps to 0. Thus 0 ∈ null(T ). Now, let α and β be scalars and let u, v ∈ null(T ).Then T (u) = 0 and T (v) = 0. Thus, since T is linear, we have

T (αu+ βv) = αT (u) + βT (v) = α · 0 + β · 0 = 0.

So, αu+ βv ∈ null(T ). Therefore, null(T ) is a subspace of V .

12.1.1 Nullspace of a Matrix

Recall any linear transformation has a matrix representation. So the Matrix Spaces are definedanalogously to linear transformations spaces. And Matrix Reduction makes the calculations easier.

Definition 12.3. The nullspace, null(M), of an m × n matrix M is the nullspace of the corre-sponding transformation, T , where T (x) = Mx for all x ∈ Rn.That is, null(M) =

Example 12.3. Given the matrix M =

1 1 12 1 −1−1 0 2

, Determine null(M).9:

9

1 1 1 02 1 −1 0−1 0 2 0

R2=−2r1+r2−→

R3=r1+r3

1 1 1 00 −1 −3 00 1 3 0

R1=r2+r1,R2=−r2−→

R3=r2+r3

1 0 −2 00 1 3 00 0 0 0

.

165

12.2 Range Spaces

When considering a transformation, we want to know to which vectors we are allowed to apply thetransformation. In the case of a Radiographic transformation, we wonder what is the shape andsize of objects/images that the particular radiographic transformation uses. This was all part ofour radiographic setup. As with most functions, this set is called the . In linear algebra,we consider only sets that are vector spaces. So, it is often referred to as the space.There is also a space to which all of the vectors in the domain space map. This space is definednext.

Definition 12.4. We say that the codomain of a linear transformation, T : V →W , is the vectorspace to which we map. That is, the codomain of T : V →W is .

In Examples 12.1 and 12.2 the codomains are P1 and R4, respectively. The codomain tends to bemuch less interesting for a given problem than the set of all thing mapped to. Often not all thevectors in the codomain are mapped to. If they were, then we would say that T is .

Definition 12.5. We say that the range space of a linear transformation, T : V → W , is thesubspace of the codomain W that contains all of the outputs from V under the transformation T .That is, ran(T ) =

Definition 12.6. We say that the rank of a linear transformation, T , is the dimension of thesubspace ran(T ).

To find the rank: Find a basis for the range space of T (a spanning set that is linearly indepen-dent) and count the number of vectors.

Theorem 12.2. Let V and W be vector spaces and let T : V → W be a linear transformation.Then ran(T ) is a of W .

So the range is a space.

Proof. By definition of ran(T ), we know that null(T ) ⊆W . Since V is a vector space, 0 ∈ V . Thezero vector always maps to 0, thus T (0) = 0 ∈ W . Thus 0 ∈ ran(T ). Now, let α and β be scalarsand let u,w ∈ ran(T ). This means there exists a v1, v2 ∈ V such that T (v1) = u and T (v2) = wThus, since T is linear, we have

T (αv1 + βv2) = αT (v1) + βT (v2) = α · u+ β · w.

So, there exists a v ∈ V , v = αv1 + βv2, such that T (v) = αu+ βw. Therefore αu+ βw ∈ ran(T )and thus, ran(T ) is a subspace of W .

166

Example 12.4. Define F : V → P1, where V = {ax2+2bx+b| a, b ∈ R} ⊆ P2. by F(ax2+2bx+b) =2ax+ 2b. Show that the range of F is P1.

Since ran(F) = P1, the rank of F is .And in fact, this means F is an map!Note: As we have seen the codomain need not be the range. This is only true if our map is onto.

Example 12.5. Define f : M2×2 → R4 by f

(a bc d

)=

ab+ abc

. Find ran(f). What is the

rank of f?

Note: In the example above, the codomain is and ran(f) 6= . That meansthere are elements in R4 that are not mapped to through f . That is, f is not .

12.2.1 Column/Range Space for Matrix

Definition 12.7. The range space, ran(M) for an m× n matrix is the set ran(M) =This is often called the Column Space of M, denoted Col(M).

167

Note: To find Col(M), we want to find all b ∈ Rm so that there is an v ∈ Rn so that Mv = b.

Process: To find a basis for the column space of a matrix M:1) Use elementary row operations to write M in RREF.2) Identify the columns that have pivots.3) The basis for Col(M) will be the span of the columns of the original M associated with thepivot columns.

Example 12.6. Given the matrix M =

1 1 12 1 −1−1 0 2

, Determine ran(M). We have Row

reduced M below:

1 1 1 02 1 −1 0−1 0 2 0

R2=−2r1+r2−→

R3=r1+r3

1 1 1 00 −1 −3 00 1 3 0

R1=r2+r1,R2=−r2−→

R3=r2+r3

1 0 −2 00 1 3 00 0 0 0

Aside: An alternate way to solve Example 12.6, is to row reduce the augmented matrix below:

1 1 1 a2 1 −1 b−1 0 2 c

R2=−2r1+r2−→

R3=r1+r3

1 1 1 a0 −1 −3 −2a+ b0 1 3 a+ c

R1=r2+r1,R2=−r2−→

R3=r2+r3

1 0 −2 −a+ b0 1 3 2a− b0 0 0 −a+ b+ c

.

We then look at the requirements for this system to be consistent and use this to find our basis.Note that our basis spans the same set that we found in Ex 12.6.

168

12.3 Injectivity and Surjectivity Revisited

Let’s consider this discussion again from the point of view of radiography. We saw that some trans-formations had the property where two objects could give the same radiograph. This particularradiographic transformation would not be . Furthermore, we found that if two objectsproduce the same radiograph, that there difference would then be invisible. That is, the differenceis in the of the radiographic transformation. Thus if there is an object that is invisibleto the radiographic transformation, any scalar multiple of it will also be invisible. This means thattwo different objects are invisible, producing the same radiograph, 0. Therefore, the radiographictransformation would not be injective.

Theorem 12.3. A linear transformation, T : V →W , is injective if and only if .

Proof. Let T : V → W be a linear transformation. First we will prove that if T is injective thennull(T ) = {0}. Since T is an linear transformation, T (0) = 0. Because T is an injection, only onevector can map to 0. Thus null(T ) = {0} and there is nothing else that can be in this set.

On the other hand, suppose null(T ) = {0}. We want to show T is an injection and we will provethis by showing that the contrapositive statement is true. That is, we will prove if T is not injectivethen null(T ) 6= {0}. Suppose T is not injective. This means there exists at least 2 vectors, say uand v both in V such that u 6= v, but T (u) = T (v). Thus T (u)−T (v) = 0. By linear transformationproperties, 0 = T (u)−T (v) = T (u− v). Thus (u− v) ∈ null(T ) and therefore null(T ) 6= {0}. Sincewe have show that the contrapositive statement is true, the original statement is also true and thus,null(T ) = {0} =⇒ T is injective.

Therefore T is injective ⇐⇒ null(T ) = {0}

New process to determine Injectivity: We now have a computational way to determine if alinear transformation is injective. Just find the Nullspace (or nullity)!

Recall, also, that we found that there were radiographs that could not be produced from an objectgiven a certain radiographic transformation. This means that there is a radiograph in the codomainthat is not mapped to from the domain. If this happens, the radiographic transformation is not

.

Theorem 12.4. Let V and W be vector spaces and let T : V → W be a linear transformation. Tis surjective if and only if ran(T ) = .

Proof. Suppose ran(T ) = W then, by definition of ran(T ) if w ∈ W , there is a v ∈ V so thatf(v) = w. Thus T is onto. Now, if T is onto, then for all w ∈ W there is a v ∈ V so thatT (v) = w. That means that W ⊆ ran(T ). But, by definition of T and ran(T ), we already knowthat ran(T ) ⊆W . Thus, ran(T ) = W .

169

Corollary 12.1. T : V → ran(T ) is a bijection if and only if null(T ) = {0}.

Proof. Note since the codomain is equal to the range for this transformation, T is onto.

Suppose T is one-to-one and suppose that u ∈ null(T ). Then T (u) = 0. But, T (0) = 0. So, sinceT is 1-1, we know that u = 0. Thus, null(T ) = {0}.

Now, suppose null(T ) = {0}. We want to show that T is 1-1. Notice that if u, v ∈ V satisfy

T (u) = T (v)

thenT (u)− T (v) = 0.

But since T is linear this gives us that

T (u− v) = 0.

Thus, u− v ∈ null(T ). But null(T ) = {0}. Thus, u− v = 0. That is, u− v. So, T is 1-1.

These theorems give us tools to check injectivity and surjectivity. Let’s see how...

Example 12.7. Define F : V → P1, where V = {ax2 + (3a − 2b)x + b| a, b ∈ R} ⊆ P2. byF(ax2 + 2bx+ b) = 2ax+ 2b. Show that V ∼= P1 using Example 12.1 and 10.4.

170

Example 12.8. Define f :M2×2 → R4 by f

(a bc d

)=

ab+ abc

. Use Example 12.2 and 10.5

to determine whether this map is injective or surjective.

12.4 The Rank-Nullity Theorem

In each of the last examples of the previous section, we saw that the following theorem holds:

Theorem 12.5. The Rank Nullity Theorem: Let V and W be a vector spaces and let T : V →W be a linear transformation. Then the following is true

dimV = + .

Recall, dimV= the number of vectors in a basis for V.

Proof. Let B = {v1, v2, . . . , vn}.First, we consider the case when ran(T ) = {0}. Then, ran(T ) has no basis, so rank(T ) = 0. Wealso know that if v ∈ V then T (v) = 0. Thus, B is a basis for null(T ) and nullity(T ) = n. Thus,rank(T )+nullity(T ) = n.Next, we consider the case when null(T ) = {0}. In this case, null(T ) has no basis so nullity(T ) = 0.Now, we refer to Lemma 10.1 and Theorem 12.3. We then know that {T (v1), T (v2), . . . , T (vn)}is linearly independent and we also know that span {T (v1), T (v2), . . . , T (vn)} = ran(T ). Thus,{T (v1), T (v2), . . . , T (vn)} is a basis for ran(T ) and rank(T ) = n. Thus, rank(T )+nullity(T ) = n.Finally, we consider the case where rank(T ) = m and nullity(T ) = k. Let

BN = {v1, v2, . . . , vk}

171

be a basis for null(T ). And let

BR = {T (vk+1), T (vk+2), . . . , T (vk+m)}

be a basis for ran(T ). Notice that we know that none of the elements of B are zero (for otherwise thisset would not be linearly independent). So, we know that none of T (v1), T (v2), . . . , T (vk) ∈ BR. Wealso know that span {T (v1), T (v2), . . . , T (vn)} = ran(T ) so there must be m linearly independentvectors in {T (v1), T (v2), . . . , T (vn)} that form a basis for ran(T ). So, our choice of vectors for thebasis of BR makes sense.Our goal is to show that m + k = n. That is, we need to show that B = {v1, v2, . . . , vk+m}. Weknow that {v1, v2, . . . , vk+m} ⊆ B. We need to show that if v ∈ B then v ∈ {v1, v2, . . . , vk+m}.Suppose v ∈ B. Then T (v) ∈ ran(T ). Suppose v /∈ null(T ) (for otherwise, v ∈ BN .) Then there arescalars α1, α2, . . . , αm so that

T (v) = α1T (vk+1) + α2T (vk+2) + . . .+ αmT (vk+m).

Soα1T (vk+1) + α2T (vk+2) + . . .+ αmT (vk+m)− T (v) = 0.

Using that T is linear, we get

T (α1vk+1 + α2vk+2 + . . .+ αmvk+m − v) = 0.

Thusα1vk+1 + α2vk+2 + . . .+ αmvk+m − v ∈ null(T ).

So eitherα1vk+1 + α2vk+2 + . . .+ αmvk+m − v = 0

orα1vk+1 + α2vk+2 + . . .+ αmvk+m − v ∈ span BN .

Ifα1vk+1 + α2vk+2 + . . .+ αmvk+m − v = 0

Then v ∈ span BT , but this is only true if v ∈ BT because B is linearly independent and v ∈ B andall the elements of {vk+1, vk+2, . . . , vk+m} are also in B. Now, if

α1vk+1 + α2vk+2 + . . .+ αmvk+m − v ∈ span BN

then v ∈ BN again because v ∈ B and so are all the elements of BN .Thus, v ∈ {vk+1, vk+2, . . . , vk+m}. So, we have that B = {vk+1, vk+2, . . . , vk+m}. Thus k +m = n.That is, nullity(T )+rank(T ) = n.

Important: A quick result of this theorem says that we can separate the basis of V into thosethat map to 0 and those to a basis of ran(T ). More specifically, we have the following result.

172

Corollary 12.2. Let V and W be vector spaces and let T : V →W be a linear transformation. IfB = {v1, v2, . . . , vn} is a basis for V , then {T (v1), T (v2), . . . , T (vn)} = {0} ∩ Br where Br is a basisof ran(T ).

Proof. We can see in the proof of Theorem 12.5 that B was split into two sets {v1, v2, . . . , vk}(a basis for null(T )) and {vk+1, vk+2, . . . , vn}, (where {T (vk+1), T (vk+2), . . . , T (vn)} is a basis forran(T )).

The Rank Nullity Theorem is useful in determining rank and nullity, along with proving resultsabout subspaces.

Example 12.9. Given a linear transformation T :M2×5 → P4.How do we know that T cannot be one-to-one?10

.

10 Notice, we didn’t know anything about T except for the spaces from and to which it maps.

173

Reading Example:

Example 12.10. Define g : V → R3, where V = P1 by g(ax + b) =

ab

a+ b

. Show that g is

injective and find the range space.

Notice that null(g) = {ax+ b| a, b ∈ R, g(ax+ b) = 0}

= {ax+ b| a, b ∈ R,

ab

a+ b

=

000

} = {ax+ b| a = 0, b = 0} = {0}.

Thus, g is injective.Now we find the range space.

ran(g) = {g(ax+ b)| a, b ∈ R} =

ab

a+ b

∣∣∣∣∣∣a, b ∈ R

= span

101

,

011

Notice that since rank(g) = 2 and dimR3 = 3, R3 6= ran(g) and thus g is not onto.Notice also that dimV = 2, nullity(g) = 0, and rank(g) = 2.

174

12.4.1 ICE 7: Column Space, Row Space, Nullspace

Let A =

1 0 2 3 10 2 −1 1 20 1 0 1 13 −1 0 2 2

. The Row Reduced Echelon form of A is

1 0 0 1 10 1 0 1 10 0 1 1 00 0 0 0 0

1. A maps from a dimensional space to a dimensional space.

2. True or False: The column space of A is a subspace of R4.(a) True and I am very confident.(b) True, but I am not very confident.(c) False, but I am not very confident.(d) False and I am very confident.

3. True or False: The nullspace of A is a subspace of R4.(a) True and I am very confident.(b) True, but I am not very confident.(c) False, but I am not very confident.(d) False and I am very confident.

4. What is the dimension of the column space of A?(a) 0(b) 1(c) 2(d) 3(e) 4

5. What is the dimension of the nullspace of A?(a) 0(b) 1(c) 2(d) 3(e) 4

6. What is the rank of the linear transformation represented by the matrix A?

(a) 0(b) 1(c) 2(d) 3(e) 4

175

7. Which columns would form a basis for the column space of A?

where A =

1 0 2 30 2 −1 10 1 0 13 −1 0 2

and the Row Reduced Echelon form of A is

1 0 0 10 1 0 10 0 1 10 0 0 0

.

(a) All four(b) The first three(c) Any three(d) Any two

8. The column space of a matrix A is the set of vectors that can be created by taking all linear

combinations of the columns of A. Is the vector b =

[−412

]in the column space of the matrix

A =

[1 23 6

]?

(a) Yes, since we can find a vector x so that Ax = b.

(b) Yes, since −2

[13

]−[26

]=

[−412

]

(c) No, because there is no vector x so that Ax = b.

(d) No, because we can’t find α and β such that α

[13

]+ β

[26

]=

[−412

]

e) More than one of the above.

9. The row space of a matrix A is the set of vectors that can be created by taking all linearcombinations of the rows of A. Which of the following vectors is in the row space of the

matrix A =

[1 23 6

]?

(a) x =[−2 4

]

(b) x =[4 8

]

(c) x =[0 0

]

(d) None of the above(e) More than one of the above

10. The column space of the matrix A =

[1 23 6

]is

(a) the set of all linear combinations of the columns of A.(b) a line in R2.

(c) the set of all multiples of the vector

[13

]

(d) All of the above(e) None of the above

176

13 Inverse Transformations

Example 13.1. If T : V →W and M is the matrix representation of T . What can you say aboutdimension of M if T is injective?

a) M has the same number of rows and columnsb) The number of rows of M is greater than or equal to the number of columnsc) The number of rows of M is less than or equal to the number of columnsd) Nothing, but I love Linear Algebra!

Example 13.2. If T : V →W and M is the matrix representation of T . What can you say aboutdimension of M if T is onto?

a) M has the same number of rows and columnsb) The number of rows of M is greater than or equal to the number of columnsc) The number of rows of M is less than or equal to the number of columnsd) Nothing, but I love Linear Algebra!

13.1 Connection to Tomography

In this class, we’ve actually talked about an application: Radiography/Tomography. Well, wehaven’t talked about the Tomography part. We will get there. First note that nobody ever reallycomputes the radiograph. This is done using a machine that sends x-rays through something. Butwhat people want to be able to do is to figure out what the object being radiographed looks like.This is the idea behind Tomography. So, we do need to be able to find the object that was producedby a certain transformation.Suppose we know that T : Rn → Rm is the transformation given by T (v) = Mv where M ∈Mm×n.Suppose also that we know that the vector w ∈ Rm is obtained by Tv for some v, but we don’tknow which v. How would we find it? Basically we want a way to work backwards!

13.2 The Inverse Matrix, when m = n

Recall, an inverse function of T, is a function such that T ◦T−1(x) = x. As we learned in Calculus,it can be difficult to construct inverse functions and sometimes they don’t even exist! But whenwe have a bijective transformation, T : V → W , we can construct an inverse transformationT−1 : W → V .Sometimes it can be difficult to construct these inverse linear transformations, but this isn’t toobad when we have a matrix transformation.

Definition 13.1. Let M be an n× n square matrix. We say that M has an inverse (and call itM−1) if and only if

MM−1 = M−1M = In×n,

where In×n is the square identity matrix. If M has an inverse, we say that it is invertible

177

Algorithm to find the inverse of A.Row reduce the augmented matrix: [A|I], if A is row equivalent to I, then row reducing gives us[I|A−1]. Otherwise, A does not have an inverse.

We can find the inverse of a matrix, A, in Octave by typing:

Example 13.3. Find M−1 when M =

(1 30 1

).11

Example 13.4. Find the inverse of M =

(1 22 4

)

Again, we begin by reducing the augmented matrix(

M e1 e2

)as follows:

(1 2 1 02 4 0 1

)→(

1 2 1 00 0 −2 1

).

This cannot be reduced further. From the reduced echelon form, we see that there is no vector v sothat Mv = e1 nor is there a vector v so that Mv = e2. That is, there is no matrix M−1 so thatMM−1 = I2×2, the identity matrix. So, M does not have an inverse.

Lemma 13.1. The inverse of A =

[a bc d

]is A−1 = 1

ad−bc

[d −b−c a

]

Proof. Notice 1ad−bc

[d −b−c a

]·[a bc d

]= 1

ad−bc

[a bc d

]·[d −b−c a

]=

[1 00 1

]

11

(1 3 1 00 1 0 1

)→(

1 0 1 −30 1 0 1

)

178

Now, let’s look at a bigger example.

Example 13.5. Find the inverse of M =

1 1 −21 2 12 −1 1

.

No matter the size of M , we always start by reducing the augmented matrix(

M e1 e2

).

1 1 −2 1 0 01 2 1 0 1 02 −1 1 0 0 1

→

1 1 −2 1 0 00 1 3 −1 1 00 −3 5 −2 0 1

→

1 0 −5 2 −1 00 1 3 −1 1 00 0 14 −5 3 1

→

1 0 −5 2 −1 00 1 3 −1 1 00 0 1 −5/6 1/2 1/6

→

1 0 0 3/14 1/14 5/140 1 0 1/14 −5/14 −3/140 0 1 −5/14 3/14 1/14

.

Using the above understanding, we see that

M

3/14 1/14 5/141/14 −5/14 −3/14−5/14 3/14 1/14

=

1 0 00 1 00 0 1

.

Thus, we know that the inverse is M−1 =

3/14 1/14 5/141/14 −5/14 −3/14−5/14 3/14 1/14

.

Theorem 13.1. Properties of Invertible Matrices

1. If A is invertible, then A−1 is invertible and (A−1)−1 =

2. If A is invertible, then AT is invertible and (AT )−1 =

3. If A and B are n× n invertible matrices, then A ·B is invertible and (AB)−1 =

Proof. (1) Notice A−1A = AA−1 = I so (A−1)−1 = A.(2) By transpose properties, (A−1)TAT = AT (A−1)T = (A(A−1))T = IT = I.(3) Using the associative property of matrix multiplication, (AB)(B−1A−1) = A(BB−1)A−1 =A(I)A−1 = AA−1 = I and (B−1A−1)(AB) = B−1(A−1A)B = B−1(I)B = B−1B = I.

.

Theorem 13.2. If A is an n× n invertible matrix, then for each b ∈ <n, the equation Ax = b hasthe unique solution x = A−1b.

Proof. Suppose A is an n × n invertible matrix and let b ∈ Rn and consider A−1b ∈ Rn. NowA(A−1b) = b so a solution does exist. We now want to show this solution is unique. To do thiswe will let u be an arbitrary solution to Ax = b and show that u = A−1b. So if Au = b thenA−1Au = A−1b⇒ u = AA−1b. Thus Ax = b has the unique solution x = A−1b.

179

The Invertible Matrix Theorem:Suppose A is a square n× n. The following statements are either all true or all false:

1. A is invertible

2. A has n pivot positions

3. The Nullspace of A is trivial (The equation Ax = 0 has only the trivial solution.)

4. The columns of A form a linearly independent set.

5. The columns of A span Rn

6. The linear transformation represented by A is injective.

7. The linear transformation represented by A is surjective.

8. AX = b has a unique solution

9. AT is an invertible matrix.

10. There is an n× n matrix C such that CA = I

11. There is an n× n matrix D such that AD = I

Proof: Omitted. Some of these properties we have already proved. Please see suggested textbookif interested in some of the details.

180

14 The Determinant of a Matrix

In this section, we discuss how to calculate a determinant of a matrix and discuss some informationthat we can obtain by finding the determinant. First, note that this section is not a comprehensivediscussion of the determinant of a matrix. There are geometric interpretations that we will notdiscuss here. Instead this section will present the computation and tough on why we would everconsider finding the determinant. At first, this computation seems lengthy and maybe even morework than its worth. But, we will use the determinant to help us decide the outcome when solvingsystems of equations or when solving matrix equations. That is, it is often nice for us to know ahint about the solution before we begin the journey through matrix reduction (especially if we haveto do these by hand and if they are big). With that, we begin by discussing the properties thatwe want the determinant of a matrix to have. These properties are all related to matrix reductionsteps.

The Properties of the Determinant

Let α be a scalar. We want the determinant of an n×n matrix, A, to satisfy the following properties:

• det(αA) = αn det(A).

• det(AT ) = det(A).

• If B is obtained by performing the row operation, Rk = αrj+rk on A, then det(B) = det(A).

• If B is obtained by performing the row operation, Rk = αrk on A, then det(B) = α det(A).

• If B is obtained by performing the row operation, Rk = rj and Rj = rk on A, then det(B) =−1 · det(A).

• If A is in echelon form, then det(A) is the product of the diagonal elements.

We can use these properties to find the determinant of a matrix by keeping track of the determinantas we perform row operations on the matrix. Let us try an example. We will find the determinant

of A =

1 1 12 −1 3−1 1 −2

. Our goal is to reduce A to echelon form all the while keeping track of

how the determinant changes.

1 1 12 −1 3−1 1 −2

R2=−2r1+r2−→

R3=r1+r3

1 1 10 −3 10 2 −1

R2=r2+r3−→

1 1 10 −1 00 2 −1

det(A) det(A) det(A)

R2=−r2−→R3=2r2+r3

1 1 10 1 00 0 −1

−→

R3=−r3

1 1 10 1 00 0 1

−det(A) det(A)

= .

181

We will try one more example before giving another method for finding the determinant. We will

find the determinant of A =

2 2 21 0 1−2 2 −4

. Again, we will reduce A to echelon form all the

while keeping track of how the determinant changes.

2 2 21 0 1−2 2 −4

R1= 1

2r1−→

1 1 11 0 1−2 2 −4

R2=−r1+r2−→

R3=2r1+r3

1 1 10 −1 00 4 −2

det(A) 12 det(A) 1

2 det(A)

R2=−r2−→R3=4r2+r3

1 1 10 1 00 0 −2

−→

R3=− 12r3

1 1 10 1 00 0 1

−12 det(A) 1

4 detA

.

Clearly, there has to be another method because, well, I said that we would want to know thedeterminant before going through all of those steps. Another method for finding the determinantof a matrix is the method called cofactor expansion.Now, if the matrix M is 2× 2, it is much easier to compute the determinant:

Determinant of a 2 by 2 Matrix:

Let M =

(a bc d

)then the determinant is

Notation: detM is also denoted as |M |.

Example 14.1. What is the determinant of

[5 41 3

]

(a) 4(b) 11(c) 15(d) 19

If M is a bigger matrix, then there’s more to do here. Here, we write out the formula given by thismethod.

182

Definition 14.1. Co-factor Expansion Method to Compute and Define the Determinant:Let A = (ai,j) be an n× n matrix and choose any j so that 1 ≤ j ≤ n, then|A| =

∑ni=1(−1)i+jai,j |Mi,j |, where Mi,j is the sub-matrix of A where the ith row and jth column

has been removed.

Note, we can also expand about a column so that we choose an i so that 1 ≤ i ≤ n, then|A| = ∑n

i=j(−1)i+jai,j |Mi,j |, Notice that if n is large, this process is iterative until the sub-matricesare 2× 2. Here are some examples showing what this formula looks like.

∣∣∣∣∣∣

a b cd e fg h i

∣∣∣∣∣∣=

Example 2: Determine |A| for A =

2 2 21 0 1−2 2 −4

.

183

Example 14.2. Let A =

a1,1 a1,2 a1,3 a1,4

a2,1 a2,2 a2,3 a2,4

a3,1 a3,2 a3,3 a3,4

a4,1 a4,2 a4,3 a4,4

, then

|A| = a1,1

∣∣∣∣∣∣

a2,2 a2,3 a2,4

a3,2 a3,3 a3,4

a4,2 a4,3 a4,4

∣∣∣∣∣∣−a1,2

∣∣∣∣∣∣

a2,1 a2,3 a2,4

a3,1 a3,3 a3,4

a4,1 a4,3 a4,4

∣∣∣∣∣∣+a1,3

∣∣∣∣∣∣

a2,1 a2,2 a2,4

a3,1 a3,2 a3,4

a4,1 a4,2 a4,4

∣∣∣∣∣∣−a1,4

∣∣∣∣∣∣

a2,1 a2,2 a2,3

a3,1 a3,2 a3,3

a4,1 a4,2 a4,3

∣∣∣∣∣∣.


5 1 01 3 20 −1 1

?

(a) 0(b) 15(c) 24(d) 26


5 0 00 3 00 0 1

?

(a) 0(b) 9(c) 15(d) none of the above

The determinant of a diagonal matrix is


5 2 −10 3 40 0 1

?

(a) 0(b) 6(c) 15(d) 22

The determinant of a triangular matrix is

184

Now we can add a few more lines to our Big Theorem:

14.1 The Invertible Matrix Theorem

Theorem 14.1. The Invertible Matrix Theorem Suppose A is a square n× n. The followingstatements are either all true or all false:

1. A is invertible

2. A has n pivot positions

3. The Nullspace of A is trivial (The equation Ax = 0 has only the trivial solution.)

4. The columns of A form a linearly independent set.

5. The columns of A span Rn

6. The linear transformation represented by A is injective.

7. The linear transformation represented by A is surjective.



10. There is an n× n matrix C such that CA = I

11. There is an n× n matrix D such that AD = I

12. detA 6=

Proof: Omitted. Some of these properties we have already proved. Please see suggested textbookif interested in some of the details.

185

186

15 Eigenvalues and Eigenvectors

Recall, Linear algebra is basically the study of multivariate linear systems and transformations and,in my opinion, is at its core trying to solve Two Fundamental Problems:

1. Solving Ax = b, and

2. Diagonalizing a matrix A (AKA Eigenvalue Problems).

The first problem relates to exploiting linear methods to solve complex and dynamical systems andsituations. Basically, it is using one of the best problem-solving techniques mathematicians use,what I like to call the “Wouldn’t it be nice if...” approach. That is, real life is a mess and we oftenhave to deal with really complex functions (if we are even lucky enough to have a function at all!)which are difficult to manage. So instead we use linear functions (lines and planes) to approximateor model the complex, real-world situation, which is much easier.

The second problem relates to simplifying our system so that we can more easily solve or approxi-mate systems. The fact that some systems don’t have solutions leads directly into the mathematicalfield of Numerical Analysis and we will dive into some basic numerical analysis in this course. Be-cause so many situations in life can be modeled linearly, Linear Algebra shows up in many topicsincluding (but not exhaustively) “Markov chains, graph theory, correlation coefficients, cryptology,interpolation, long-term weather prediction, the Fibonacci sequence, difference equations, systemsof linear differential equations, network analysis, linear least squares, graph theory, Leslie popu-lation models, the power method of approximating the dominant eigenvalue, linear programming,computer graphics, coding theory, spectral decomposition, principal component analysis, discreteand continuous dynamical systems, iterative solutions of linear systems, image processing, andtraffic flow.”1

15.1 Eigenvalue and Eigenvectors Motivation

This next section of the class is primarily focused on the second fundamental problem we like touse Linear Algebra to solve. Before we begin, we first discuss some motivation for why we like andseek to find eigenvalues, eigenvectors, and eigenbases.

187

15.1.1 ICE 8: Motivation for Eigenvalues and Vectors

1. Compute the product

[1 22 1

]·[11

]

a)

[1 22 1

]b)

[33

]c)[3 3

]d)

[42

]e) None of the above.


[1 22 1

]2

·[11

]

a)

[33

]b)

[66

]c)

[99

]d)

[1212



[1 22 1

]4

·[11

]

a)

[2727

]b)

[8181

]c)

[243243

]d)

[729729


4. For any integer n, what will this product be?

[1 22 1

]n·[11

]

a)

[3n3n

]b) 3n

[11

]c) n3

[11

]d) 3n

[nn

]e)

[33

]n

188

Wouldn’t it be nice if...

189

190

15.2 Motivating Application for Eigenvalues and Eigenvectors

15.2.1 Review of Heat States in a (Vector) State Space

Recall, we in our notes on Vector Space, we introduced the application of diffusion welding. Amanufacturing company uses the process of diffusion welding to adjoin several smaller rods into asingle longer rod. The diffusion welding process leaves the final rod heated to various temperaturesalong the rod with the ends of the rod having the same temperature. Every acm along the rod, amachine records the temperature difference from the temperature at the ends to get heat states.We can represent the heat states of the rod as a vector space.

Heat States in One DimensionNow consider the function f : [a; b]→ R that measures, along a rod of length l = b− a, the differ-ence in temperature from the temperature at the ends of the rod (f(a) = f(b) = 0). The quantityf(x) is the temperature of the rod at position x for any x in the interval [a, b].

Notice that the temperature distribution evolves in time.1. Places where the temperature is higher than the surrounding area will .2. Places where the temperature is cooler than the surrounding area will .3. As the heat diffuses, temperature differences disappear first.4. The long term behavior is that temperatures smooth out approaching f(x) = .

We want to explore this application further and formulate our problem in a finite dimensionalvector space. We will use our knowledge of linear algebra to compute the heat state at any latertime as the heat diffuses.

In the previous part of the discussion we modeled the heat along a bar by a continuous functionf , which we call the heat state of the bar. We will discretize such a heat state f by sampling thetemperature at m locations along the bar. If we space the m sampling locations equally, then for∆x = L

m+1 = b−am+1 , we can choose the sampling locations to be a+ ∆x, a+ 2∆x, . . . , a+m∆x. (We

will assume that the heat state is known (and fixed) at the endpoints so we don’t need to samplethere!) Then, the discretized heat state is the following vector u in Rm.

[u1, u2, ..., um] = [f(a+ ∆x), f(a+ 2∆x), . . . , f(a+m∆x)].

Notice that f(a) = f(b) = 0, where b = a + (m + 1)∆x. Also, if uj = f(x) for some x ∈ [a, b]then uj+1 = f(x + ∆x) and uj−1 = f(x − ∆x). The figure below shows a continuous heat stateon the left and a heat state with sampling points marked on the right (although the initial and

191

final points on the heat state are marked in the figure, they are not part of the vector u). A noteabout notation: in these notes we do not use special notation for vectors (such as bold typeface orarrows).

To complete our vector space development (we need the set of heat states to form a vector spaceso that we can use Linear Algebra in this setting), we must define heat state addition and scalarmultiplication.

1. We define scalar multiplication in the usual componentwise fashion. Scalar multiplicationresults in a change in only.

2. We define vector addition in the usual componentwise fashion. Addition can result in changesin both amplitude and .

192

15.2.2 The Heat Equation

The heat equation, or diffusion equation, (with fixed unit diffusion coefficient) is

∂f

∂t=∂2f

∂x2.

1. A heat state vector f is a function of both position and time, though we often do not explicitlywrite f(x, t) for clarity sake.

2. The rate of change of temperature at any location x is proportional to the second derivativeof the temperature with respect to position.

3. Temperature is changing fastest at sharp peaks or dips in temperature where the curvatureis large.

4. Regions where temperature is changing slowly are where |∂2f(x)/∂x2| is small. This occursunder two different conditions:

(a) Where the temperature profile has nearly slope.

(b) Near points of the temperature profile.

5. The sign of the second derivative ensures that the heat state function does approach the zerofunction.

15.2.3 The Discrete Heat Equation, in x

Because derivatives are linear, intuitively we believe that it should be possible to get a good discreteapproximation. So, we discretize the derivatives, using the definition of derivative

∂f

∂x= lim

h→0

f(x+ h)− f(x)

h= lim

h→0

f(x)− f(x− h)

h.

Notice that in a discrete setting, it is not possible for h → 0. The smallest h can be is ∆x. Let’suse this to make the matrix representation for the second derivative operator, which we call D2.That is, D2u approximates ∂2f/∂x2.

∂2f

∂x2=

∂

∂x

∂f

∂x= ∂

∂x

(limh→0

f(x+h)−f(x)h

)≈ ∂

∂x

(f(x+ ∆x)− f(x)

∆x

)

= limh→0

[f(x+h)−f(x)

∆x − f(x+∆x−h)−f(x−h)∆x

h

]≈ f(x+∆x)−f(x)−f(x+∆x−∆x)+f(x−∆x)

(∆x)2

≈ 1(∆x)2

[f(x+ ∆x)− 2f(x) + f(x−∆x)]

Notice that we have used both a forward and backward difference definition of the derivative in orderto make our approximation symmetric. This helps us keep the later linear algebra manageable. Inour discretization, we have

∂2uj∂x2

=1

(∆x)2(uj+1 − 2uj + uj−1).

193

Example 15.1. Write the matrix representation for the second derivative operator, D2 with respectto the standard basis using D2(uj) = 1

(∆x)2(uj+1 − 2uj + uj−1) for {u1, u2, ..., u6}12

15.2.4 The Discrete Heat Equation, in t

The heat state u also changes over time. So we can write u = u(t), with a time dependence. That

means we have from above ∂2u(t)∂x2

= D2u(t).

Now, we use a similar argument to discretize ∂u∂t .

∂u

∂t≈ u(t+ ∆t)− u(t)

∆t.

We want to find the diffusion operator E : Rm → Rm, for a fixed ∆t, so thatu(t + ∆t) = Eu(t) (That is, u(t + ∆t) is obtained via a matrix multiply). We put the abovediscretizations into the heat equation to obtain:

u(t+ ∆t)− u(t)

∆t=

1

(∆x)2D2u(t).

So,

u(t+ ∆t) = u(t) +∆t

(∆x)2D2u(t) =

(I +

∆t

(∆x)2D2

)u(t).

Let E be defined as the matrix I + ∆t(∆x)2

D2. Then we have u(t+ ∆t) = Eu(t).

12 Note: This means that if we want to take a discrete second derivative, we do the matrix multiply D2u.

194

Example 15.2. Compose the matrix representation of E. Note E = In+ ∆t(∆x)2

D2 and let δ = ∆t(∆x)2

(which is a constant)

E =

1 0 0 00 1 0 0 . . .0 0 1 0

.... . .

+ δ ∗

−2 1 0 01 −2 1 0 . . .0 1 −2 1

.... . .

=

1− 2δ δ 0 0δ 1− 2δ δ 0 . . .0 δ 1− 2δ δ

.... . .

Technical Detail: We need 0 < δ ≡ ∆t(∆x)2

≤ 14 for computational stability, though the proof is

beyond the scope of this module. ∆x is fixed so we need to take small enough time steps ∆t to getthis inequality. Hence as we let ∆x→ 0, we are also implicitly forcing ∆t→ 0.

So u(t+ ∆t) = E · u(t) =

1− 2δ δ 0 0δ 1− 2δ δ 0 . . .0 δ 1− 2δ δ

.... . .

· u(t)

Consider the meaning of the row and column representations of E.

1. How can we interpret the values in column k of E?Answer: The kth column shows how the heat at time t distributes to heat at time t+ ∆t atlocation k in the heat state. In particular, fraction 1− 2δ of the heat at location k remainsat location k and fraction δ of the heat moves to each of the two nearest neighbor locationsk + 1 and k − 1.

2. How can we interpret the values in row k of E?Answer: The kth row shows where the heat at time t + ∆t came from. In particular,fraction 1− 2δ of the heat at location k was already at location k and fraction δ came fromeach of the location’s two nearest neighbors.

3. How can we interpret the fact that all but the first and last columns (and rows) sum to unity(1)?Answer: Based on the above discussion, we see that this guarantees that no heat is lostfrom the system except at the end points. Heat is redistributed (diffused) not lost.

15.2.5 Heat State Evolution

Now, we consider the evolution over more than one time step.

u(t+ ∆t) = Eu(t)

u(t+ 2∆t) = Eu(t+ ∆t) = E(Eu(t)) = E2u(t)

This means k time steps later,u(t+ k∆t) =

195

Example 15.3. We found the matrix representation of the heat diffusion operator to be

E =

1− 2δ δ 0 0δ 1− 2δ δ 0 . . .0 δ 1− 2δ δ

.... . .

.

We see that if we are want to find the kth heat state, u(k∆t) in the heat diffusion, we need to use

u(k∆t) = Eku(0),

where u(0) is the initial heat state. Let’s explore what happens when we do these calculations.

1. What is the equation for the heat state 2 steps in the future? 1000 time steps in the future?

2. Note if we find E2 or (E · E), we get

E2 =

(1− 2δ)2 + δ2 2δ(1− 2δ) δ2 0 . . .2δ(1− 2δ) (1− 2δ)2 + δ2 2δ(1− 2δ) δ2 0

δ2 2δ(1− 2δ) (1− 2δ)2 + δ2 2δ(1− 2δ) . . .0 δ2 2δ(1− 2δ) (1− 2δ)2 + δ2

......

...... . . .

3. Does it look like raising E to higher and higher powers is an easy way to find the heat stateat some time far in the future (for example, 1000 time steps away)?

4. Does it look like multiplying over and over by the matrix E is an easy way to find the heatstate at some time far in the future?

5. If you had to perform this kind of operation, what characteristics would you prefer a diffusionoperator to have?

6. On the next page, you will see several 12 Diffusion Models for Heat States. Pick 3 that seemto have the easiest, most predictable diffusion patterns.

196

Heat Equation for Linear Algebra Lab 1

1. In the following picture, there are 12 different initial heat states (in orange) and their corre-sponding diffusions (colors varying from orange to dark blue). Group the pictures based onthe diffusion behavior. Briefly list the criteria you used to group them. We will use thesedescriptions in the next discussion so try to be as clear as possible.

1

197

198

15.3 Eigenvalue and Eigenvectors Motivation Continued

In our motivating application example, we discovered that multiplying a matrix A multiple timescan often cause us to have very complex matrix. So when we want to discover how a state, like inheat diffusion, changes after time, we want to find u(t+ n∆t) = Enu(t). Wouldn’t it be nice if wehad special vectors which had the property Eu = λu where λ is a constant/scalar? If this were thecase then u(t + n∆t) = Enu(t) = λnu(t). This will mean that our heat states will only change inamplitude in the heat diffusion (cooling) on a rod. This means that when we apply the diffusionoperator to one of these heat states, we get a result that is a scalar multiple of the original heatstate. Mathematically, this means we want to find vectors v such that:

Ev = λv, (5)

for some scalar λ. In other words, these vectors, v, satisfy the matrix equation

(E − λI)v = 0. (6)

Notice this is a homogeneous equation so we know that this equation has a solution. This meansthat either there is a unique solution (only the trivial solution) or infinitely many solutions. If webegin with a zero heat state (all temperatures are the same everywhere along the rod) then thediffusion is very boring (my opinion, I know) because nothing happens. It would be nice to finda nonzero vector satisfying the matrix Equation (6) because it gets us closer to the possibility ofhaving a basis of these vectors. By the invertible matrix theorem, we know that this equation hasa nonzero solution as long det(E − λI) = . Let’s remind ourselves why we want such abasis. In the case of heat states, we recognize that if B = {v1, v2, . . . , vm} is a basis of these specialvectors so that Evi = λivi and u0 is our initial heat state, we can write u in coordinates accordingto B. That is, there are scalars α1, α2, . . . αm so that

u0 = α1v1 + α2v2 + . . .+ αmvm.

Then, when we apply the diffusion operator to find the heat state, u1 a short time later, we get

u1 =Eu0 = E(α1v1 + α2v2 + . . .+ αmvm)

=α1Ev1 + α2Ev2 + . . .+ αmEvm

=α1λ1v1 + α2λ2v2 + . . .+ αmλmvm.

So, if we want to find uk for some time step k, far into the future, we get

uk =Eku0 = α1Ekv1 + α2E

kv2 + . . .+ αmEkvm

=α1λk1v1 + α2λ

k2v2 + . . .+ αmλ

kmvm.

With this we can predict long term behavior of the diffusion. (You will get a chance to do this inthe exercises later.)

199

15.4 Eigenvalue and Eigenvectors Definition

This property that the special vectors in a heat diffusion have is a very desirable property elsewhere.So, in Linear Algebra, we give these vectors a name.

Definition 15.1. Let V be a vector space. Given a linear operator(transformation) L : V → V ,with corresponding square matrix, and a nonzero vector v ∈ V . If Lv = λv for some scalar λ, thenwe say v is an eigenvector of L with eigenvalue λ.

As with the heat states, we see that eigenvectors (with positive eigenvalues) of a linear operatoronly change amplitude when the operator is applied to the vector. This makes repetitive applica-tions of a linear operator to its eigenvalues very simple.

Note: Since every linear operator has an associate matrix, we will treat operators/transformationsL : V → V as a matrix for the rest of this section.

Process: To find the eigenvalues and eigenvectors of a matrix, we need only solve the equationdet(L− λI) = 0. That is, the scalars λ so that det(L− λI) = 0.

Definition 15.2. We call the equation det(L−λI) = 0. the characteristic equation and det(L−λI) the characteristic polynomial of L.

Example 15.4. Find the eigenvalues of the matrix M =

(5 −36 −4

).

200

Example 15.5. Find the eigenvectors associated with the eigenvalues found in Ex 15.4 (-1 and 2)

of M =

(5 −36 −4

).

Definition 15.3. Given an eigenvalue λ for matrix A, we sometimes call the nullspace of A− λIthe space corresponding to λ. A basis for an eigenspace is called an basis

Example 15.6. What is the eigenspace for λ = 2 for M =

(5 −36 −4

)from Ex 15.4?

201

Example 15.7. Find the eigenvalues for M =

5 1 30 0 10 0 1

.

Theorem 15.1. The eigenvalues of a triangular or diagonal matrix are the entries of its maindiagonal.

Proof. We will prove this for a triangular matrix since a diagonal matrix is a triangular matrix.Without loss of generality, assume A is an upper triangular matrix. Therefore

A− λI =

a11 − λ a12 a13 a14 a15 . . . a1n

0 a22 − λ a23 a24 . . . . . . a2n

0 0 a33 − λ a34 . . . . . . a3n...

. . ....

0 0 . . . 0 . . . 0 ann − λ

.

Since the determinant of a triangular matrix is the product of the diagonal entries. Solving det(A−λI) = 0 gives the characteristic polynomial (a11 − λ)(a22 − λ) . . . (ann − λ) = 0 And thus the rootsof this equation (the eigenvalues) are a11, a22, . . . , ann.

Lemma 15.1. If 0 is an eigenvalue of A, then A is not invertible!

Proof. If 0 is an eigenvalue of A, then for λ = 0, 0 = det(A − λI) = det(A − 0) = det(A). Ifdet(A) = 0, then A is not invertible by The Invertible Matrix Theorem (Theorem 14.1).

202

Example 15.8. Find an eigenbasis for the eigenspace associated with the eigenvalue 1 for

M =

5 1 30 0 10 0 1

.

203

204

15.5 ICE 9: Eigenvalue and Eigenvectors

Two Sided.

1. Consider A =

[1 12 2

]

(a) Find the eigenvalues for A.

(b) Find the eigenspaces associated to each eigenvalue.

205

2. Consider A =

[−2 43 −1

]13

(a) Find the eigenvalues for A.

(b) Find the eigenspaces associated to each eigenvalue.

13Note if A =

[−2 43 1

]we have irrational evalues, which are ok, but messier.

206

15.6 Eigenbasis

Recall, when we are searching for eigenvectors and eigenvalues of an n×n matrix M , we are reallyconsidering the linear transformation T : Rn → Rn defined by T (v) = Mv. Then the eigenvectorsare vectors in the domain of T that are scaled (by the eigenvalue) when we apply the linear trans-formation T to them. That is, v is an eigenvector with corresponding eigenvalue λ ∈ R if v ∈ Rnand T (v) = λv.

So far, we have agreed that it would be nice if we could find a set of “simple vectors” that formeda basis for the space of heat states.

That is, this will allow us to predict long term behavior of a system in a rather simple way. Butwe can only do this if we have a basis consisting of eigenvectors. If we can find such a basis, wecall this basis a special name, we say it is an “eigenbasis”. That is, an eigenbasis is just a basis forRn made up of eigenvectors. We define an eigenbasis more formally here. In this section, we wantto explore when it is the case that we have an eigenbasis.

Definition 15.4. Given an n × n matrix M . If M has n linearly independent eigenvectors,v1, v2, . . . , vn then B = {v1, v2, . . . , vn} is called an eigenbasis of M for Rn.

First let’s recall the following lemma.

Lemma 15.2. If the dimension of V is n, then any set of n linear independent vectors of V is abasis for V.

Proof. If the dimension of V is n, then a set of n linear independent vectors will be a minimumspanning set for V and thus is a basis.

Goal: We want to construct an eigenbasis for M by finding a large enough set of linearly independenteigenvectors. So that the number of eigenvectors is equal to the dimension of M.

Example 15.9. Determine if M =

2 0 00 3 10 0 3

has an eigenbasis for R3.

So we want to find , linearly independent eigenvectors for M.

We begin by finding the and .That is, we want to know for which nonzero vectors v and scalars λ does Mv = λv.Eigenvalues =

207

λ1 = 2 In this case we are solving the matrix equation (M − 2I)v =

0 0 00 1 10 0 1

abc

= 0. Row

reducing gives us

0 0 0 00 1 1 00 0 1 0

→

0 1 1 00 0 1 00 0 0 0

→

0 1 0 00 0 1 00 0 0 0

.

λ2 = 3 Now, in this case we are solving the matrix equation (M−3I)v =

−1 0 00 0 10 0 0

abc

= 0.

Row reducing gives us

−1 0 0 00 0 1 00 0 0 0

→

1 0 0 00 0 1 00 0 0 0

.

To try to find an eigenbasis, take the union of each basis for each eigenspace. In this case our setis:

So can we find an eigenbasis of M for R3?

208

Comment: Notice that in each example above, the set made by taking the union of each basis foreach eigenspace is a linearly independent set. This theorem justifies why this union will be linearlyindependent.

Lemma 15.3. Let M be a matrix and let v be an eigenvector with eigenvalue λ. Then for anyscalar c, cv is an eigenvector with eigenvalue λ.

Proof. Given a matrix M with eigenvalue λ with eigenvector v, then any vector in the eigenspaceassociated with λ will be an eigenvector for M. Thus any scalar multiple of v will also be in theeigenspace and thus will be an eigenvector associated with λ.

Theorem 15.2. Let M be a matrix and let v1 and v2 be eigenvectors with eigenvalues λ1 and λ2

respectively. If λ1 6= λ2 then {v1, v2} is linearly independent.

Proof. Suppose v1 and v2 are nonzero eigenvectors with eigenvalues λ1 and λ2 respectively. Supposealso that λ1 6= λ2. Then Mv1 = λ1v1 and Mv2 = λ2v2. By Lemma 15.3, αv1 and βv2 areeigenvectors with eigenvalues λ1 and λ2 respectively. Let’s look at two cases 1.) λ1 = 0 and λ2 6= 0(the case when λ2 = 0 and λ1 6= 0 is proved similarly so we won’t prove it) and 2.) λ1 6= 0 andλ2 6= 0:

Case 1: If λ1 = 0 then by definition, v1 ∈ nullM . But since λ2 6= 0, v2 /∈ null(M).Suppose that αv1 + βv2 = 0. Then M(αv1 + βv2) = 0.But this means that βMv2 = 0. So β = 0. But then αv1 = 0 and so α = 0 and {v1, v2} islinearly independent.

Case 2: If λ1 6= 0 and λ2 6= 0 and αv1 + βv2 = 0.Then M(αv1 + βv2) = 0 tells us that αλ1v1 = −βλ2v2.Then αv1 = λ2

λ1βv2. Thus, λ2

λ1βv2 is an eigenvector with eigenvalue λ1 then, λ2

λ1βv2 is an

eigenvector with eigenvalue λ2. So

Mλ2

λ1βv2 = λ2βv2 and M

λ2

λ1βv2 =

λ22

λ1βv2.

So,

λ2βv2 =λ2

2

λ1βv2 and (λ1 − λ2)βv2 = 0.

Since, λ2 6= λ1 and v2 6= 0, we see that β = 0. Thus,

M(αv1 + βv2) = 0

implies αλ1v1 = 0. So, α = 0. Therefore {v1, v2} is linearly independent.

209

210

16 Diagonalization

If we have a basis made up of eigenvectors, life is good. Well, that’s maybe a bit over reaching.What we found was that if B = {v1, v2, . . . , vn} is an eigenbasis for Rn corresponding to the diffusionmatrix E then we can write any initial heat state vector v ∈ Rn as v = α1v1 + α2v2 + . . .+ αnvn.Suppose these eigenvectors have eigenvalues λ1, λ2, . . . , λn, respectively. Then with this decom-position into eigenvectors, we can find the heat state at any later time (say k time steps later)by multiplying the initial heat state by Ek. This became an easy computation with the abovedecomposition because it gives us, using the linearity of matrix multiplication,Ekv = Ek(α1v1 + α2v2 + . . . + αnvn) = α1λ

k1v1 + α2λ

k2v2 + . . . + αnλ

knvn. We can then apply our

knowledge of limits from Calculus here to find the long term behavior. That is, the long termbehavior is lim

k→∞α1λ

k1v1 +α2λ

k2v2 + . . .+αnλ

knvn. Depending on the eigenvalues. We also discussed

how it would be nice to use a change of basis to transform E so that it acts like a diagonal matrix.Recall diagonal matrices are easy to raise to higher powers.

For example if D =

a 0 0 ...0 b 0 ...0 0 c ...0 0 0 ......

......

. . .

, Dk =

There is a formal method for this, called Diagonalization.

Definition 16.1. Matrices A and B are similar if there exists an invertible matrix P such thatA =

Definition 16.2. An n× n (square) matrix A is diagonalizable if there exists n× n matrices Dand P where D is diagonal and P is invertible such that A = .That is, A is similar to a diagonal matrix.

Why is diagonalization so useful?Suppose A is diagonalizable, that is, A = PDP−1. Calculate A3.

211

Example 16.1. Suppose A = PDP−1 where D =

[1 00 2

]and P =

[2 10 −1

]. Calculate A3.

Theorem 16.1. An n × n matrix A is diagonalizable if and only if A has n linearly independenteigenvectors. In fact, A = PDP−1, where D is a diagonal matrix, if and only if the columns of Pare n linearly independent eigenvectors of A and the diagonal entries of D are the eigenvalues of Athat correspond respectively to the eigenvectors of P.

Note: In other words, A is diagonalizable if and only if there are enough eigenvectors to form abasis of Rn. If such a basis exists, we call it an eigenvector basisor eigenbasis.

Proof. It is helpful for notation to first notice that if P is any n×n matrix with columns v1, v2, ..., vnand if D is a diagonal matrix with diagonal entries λ1, λ2, ..., λn, then

AP = A[v1 v2 . . . vn

]=[Av1 Av2 . . . Avn

](7)

and

PD =[v1 v2 . . . vn

]·

λ1 0 0 . . . 00 λ2 0 . . . 0

0 0. . . 0 0

0 0 . . . 0 λn

=

[λ1v1 λ2v2 . . . λnvn

](8)

First suppose A is diagonalizable. So there exists invertible P and diagonal D such that A =PDP−1 ⇒ AP = PD Thus by 7 and 8,

[Av1 Av2 . . . Avn

]=[λ1v1 λ2v2 . . . λnvn

]

Setting the columns equal to each other gives us that Avi = λivi for i = 1, ..., n. Since P is in-vertible, {v1, v2, ..., vn} are linearly independent and non-zero by The Invertible Matrix Theorem(Thm 14.1). Furthermore, λ1, λ2, ..., λn are eigenvalues with associated eigenvectors v1, v2, ..., vn.Therefore A has n linearly independent eigenvectors (and thus has an eigenbasis -woo!).

On the other hand, suppose A has n linearly independent eigenvectors, v1, v2, ..., vn. Define

P :=[v1 v2 . . . vn

]

212

. And put

D :=

λ1 0 0 . . . 00 λ2 0 . . . 0

0 0. . . 0 0

0 0 . . . 0 λn

. ThenAP =

[Av1 Av2 . . . Avn

]

andPD =

[λ1v1 λ2v2 . . . λnvn

]

. By definition of eigenvalues and eigenvectors for A, AP = DP and since the columns of P are thelinearly independent eigenvectors, by the Invertible Matrix Theorem (Thm 14.1), P is invertible.Therefore A = PDP−1 and thus is diagonalizable.

Process to Diagonalize a Matrix, If Possible

1. Find the eigenvalues of the matrix.Solve det(A− λI) = 0. That is, solve |A− λI| = 0.

2. Find a spanning set of linearly independent eigenvectors of the matrix.For each e-value, λi, find a basis for Null(A− λiI).Take the union of all bases for all of the eigenspaces.

3. Determine if you have enough eigenvectors from Step 2, that you can span Rn.Note you can only proceed if there are the same number of eigenvectors as the dimension ofA (n) if there are not enough eigenvectors, the matrix is NOT diagonalizable).

4. Construct P from the eigenvectors from Step 2.

5. Construct D from the corresponding eigenvalues (order matters!).

6. Check to make sure P and D work by checking if A = PDP−1, but it is easier tojust check if AP =

213

Example 16.2. Diagonalize M =

(5 −36 −4

)if possible. Note in Ex. 15.6, we found that M has

eigenvalues 2 and -1 with associated eigenspaces spanned by E2 =span{[11

]} and E−1 =span{

[12

]}.

Example 16.3. Diagonalize M =

2 0 00 3 10 0 3

from Example 15.9 if possible.

214

Example 16.4. Diagonalize A =

4 0 −22 5 40 0 5

, if possible. One eigenvalue is 4 and Eva has found

that the E-space associated with λ = 4 to be E4 =span{

−120

} 14

14null(A− 4I) = null(

0 0 −12 1 40 0 1

)→ we solve

0 0 −1 : 02 1 4 : 00 0 1 : 0

→

1 12 0 : 0

0 0 1 : 00 0 0 : 0

So the Eigenspace associated with 4 is = {

xyz

|x = −y2 , z = 0} = {

−y2y0

|y ∈ R} =span {

−1210

} =span

{

−120

}

215

216

16.1 ICE 10 -Diagonalization

1. Eva and Archer want to Diagonalize a 2 by 2 matrix A. Dr. Harsy has computed the follow-

ing eigenvalues and associated eigenvectors for A: λ1 = 3 with associated eigenvector {[11

]}

and λ2 = 16 with associated eigenvector

[1−1

]. Which of the following would work for the

diagonalization of A:

a) P =

[1 11 −1

], D =

[16 00 3

]

b) P =

[1 1−1 1

], D =

[16 00 3

]

c) P =

[1 11 −1

], D =

[3 00 16

]

d) P =

[1 1−1 1

], D =

[3 00 16

]


2. Now Eva and Archer want to Diagonalize a 3 by 3 matrix A. Dr. Harsy has computed thefollowing eigenvalues and associated eigenvectors for A: λ1 = 3 with associated eigenbasis

{

110

} and λ2 = 16 with associated eigenbasis {

01−1

,

111

}. Which of the following would

work for the diagonalization of A:

a) P =

1 1 01 1 10 1 −1

, D =

3 0 00 3 00 0 16

b) P =

1 1 01 1 10 1 −1

, D =

3 0 00 3 00 0 16

c) P =

1 1 01 1 10 1 −1

, D =

16 0 00 16 00 0 3

d) P =

1 1 01 1 10 1 −1

, D =

3 0 00 16 00 0 16


217

3. Now Eva and Archer want to Diagonalize a 3 by 3 matrix A. Dr. Harsy has computed thefollowing eigenvalues and associated eigenvectors for A: λ1 = 3 with associated eigenbasis

{

110

} and λ2 = 16 with associated eigenbasis {

01−1

}. Which of the following would work

for the diagonalization of A:

a) P =

1 1 01 1 10 1 −1

, D =

3 0 00 16 00 0 16

b) P =

1 1 01 1 10 1 −1

, D =

3 0 00 3 00 0 16

c) P =

1 1 01 1 10 1 −1

, D =

16 0 00 16 00 0 3

d) More than one of the above.

e) A is not diagonalizable.

4. Suppose A = PDP−1, write an expression for A8 involving P and D.

218

17 Markov Chains

It turns out, we already kinda know about Markov Chains 15

In our Heat Equation Discussion about the motivation for eigenvectors, we discussed how we couldeasily predict long term behavior of our heat state by calculating Ekv0, where E is our diffusionmatrix and v0 is our initial heat state. This can get really difficult to calculate, especially if E is alarge matrix. On the other hand, if we had a basis made up of eigenvectors, life is easier. Whenwe have such a basis, our matrix E is diagonalizable. That is we can write E = PDP−1. ThenEkv0 = PDkP−1v0. We can then also apply our knowledge of limits from Calculus here to find thelong term behavior. That is, the long term behavior is

limk→∞

Ekv0 = limk→∞

PDkP−1v0 =

limk→∞

α1Ekv1 + α2E

kv2 + . . .+ αmEkvm = lim

k→∞α1λ

k1v1 + α2λ

k2v2 + . . .+ αmλ

kmvm

We see that this limit really depends on the size of the eigenvalues.

We can see how this equation allows us to find different heat states for a given rod. That is,u2 = Eu1, u3 = Eu2 = E2u1, u4 = Eu3 = E3u1, etc. This process of transitioning between “states”of a system over discrete time steps has many applications including:

• Movement of people between regions

• States of weather

• Movement between positions on a board games

• Your score in betting games like Blackjack

• Movements between webpages

• Positions of the runners on base and the number of outs in baseball

• Population changes of an ecosystem

• Subscription changes for amenities

Example 17.1. We will divide the class into 3 groups, A,B, & C.

• 1/3 of group A goes to group B and 1/3 of group A goes to group C.

• 1/4 of group B goes to group A and 1/4 of group B goes to C.

• 1/2 of group C goes to group B.

What happens after a few state transitions?

15Some Examples from When Life is Linear and http://www.slideshare.net/leingang/

lesson-11-markov-chains

219

http://www.slideshare.net/leingang/lesson-11-markov-chains

http://www.slideshare.net/leingang/lesson-11-markov-chains

17.1 Introduction to Markov Chains

Definition 17.1. A Markov chain or Markov Process is a process in which the probability of thesystem being in a particular state at a given time period depends only on its state at previous time period.Note: There is NO PAST in a Markov Chain, just the here and now!

Common questions about a Markov Process:

• What is the long term behavior of the process?

• Is there a long-term stability to the process?

• What is the probability of transitions from state to state over multiple observations

• Can we hit an equilibrium state?

Example 17.2. Suppose on any given class day you wake up and decide whether to come to class.If you went to class the time before, you’re 70% likely to go today, and if you skipped the last class,you’re 80% likely to go today. Some questions you may ask include:If I go to class on Tuesday, how likely am I to go to class on Thursday? Next Thursday?Assuming the class is infinitely long (oh my!), what portion of the class will I approximately attend?

To answer these questions, we first will set up a Transition Matrix or Stochastic Matrix.

Definition 17.2. Suppose a system has n possible states. The Transition or Stochastic Matrixfor this system is given by T = (tij) where for each i and j, tij is the probability of switching fromstate j to state i.

What is our transition matrix for Ex 17.2: Skipping Class?If you went to class the time before, you’re 70% likely to go today, and if you skipped the last class,you’re 80% likely to go today.

What is our transition matrix for Ex. 17.1: Markov Dance?Recall, 1/3 of group A goes to group B and 1/3 of group A goes to group C. 1/4 of group B goesto group A and 1/4 of group B goes to C. 1/2 of group C goes to group B.

220

What properties do these matrices have?

Properties of Stochastic Matrices:

• All entries are

• The columns add up to

Example 17.3. Is our Diffusion Matrix from our Heat Lab a transition matrix?

E =

1− 2δ δ 0 0 . . .δ 1− 2δ δ 0 . . .0 δ 1− 2δ δ...

......

. . .

Definition 17.3. A Transition Matrix (for a corresponding Markov chain) is called regular if forsome k ≥ 1, T k has all strictly positive entries. Note that this means there is a positive probabilityof eventually moving from every state to every state.

Example: T =

[1 00 1

]is a matrix but not .

Is A a Transition Matrix? Is it Regular?

A =

.5 .5 .5.5 0 .50 .5 0

, A2 =

.5 .5 .5.25 .5 .25.25 0 .25

, A3 =

.5 .5 .5.375 .25 .375.125 .25 .125

.

Definition 17.4. The state vector for a Markov chain with n possible state at a given time step k

is denoted xk =

pk1pk2pk3...pkn

where pki is the probability that the system is in state i at time-step k. These vectors are also calledvectors since they add up to 1 and are nonnegative.

In order to find our state vectors for our two examples, we need more information.For our skipping class example, we need to decide what is our initial vector state. What would bea good choice for this?

221

Find the initial state for our Markov Activity if we started with 20 students in A, 10 in B, and 10in C.

Theorem 17.1. If T is the transition matrix for a Markov chain, the then the state vector for the(k + 1)th time step, denoted , can be determined from by

Proof: Omitted. See supplementary textbook.

Example 17.4. Suppose you go to class on Tuesday, what’s the probability that you will go to thenext class? What about the class 3 days from now?

Connection: Remember, if we want to calculate T k for large k’s (Lay’s Linear Algebra Book saysfor k > 30), we may want to compute each xk from the previous state rather than calculate T k.

222

Even better, it would be nice if we can T!

17.2 Steady State Vectors

As we mentioned before, it may be useful to determine the end behavior of a system. In otherwords, we want to determine lim

k→∞T kx0.

Theorem 17.2. If T is an n× n regular stochastic matrix, the T has a unique vector u. That isif x0 is the initial state, the lim

k→∞T kx0 = . That is, our Markov chain of states {xi}

converges to u.


Definition 17.5. The vector u described above is called a steady-state vector.

What does this mean?xi+1 = Txi so for large i...

Theorem 17.3. The steady-state vector u is the unique probability vector satisfying the matrixequation:

That is, u is an for the associated = !!


So how can we find a steady-state vector?

Example 17.5. Find the steady-state vector for the Skipping Ex. 17.2. Recall, T =

[0.7 0.80.3 0.2

]

223

17.3 Application to Google’s Page Rank Model

There are billions of web pages in the interwebs. How can we possibly determine the quality of apage? Google’s Page Rank Model was developed by Google founders Larry Page and Sergey Brinin order to solve this problem. They determined the popularity of a web page by modeling Internetactivity. That is, the frequency of time spent of time spent on a page yields that page’s PageRank.So what is this model? Does Google have spies (probably cats) tracking your every move on theInternet?

Like every model, we start with some assumptions.PageRank Model Assumptions:

• Google treats everyone as a random surfer and assumes you will randomly choose links tofollow.

• If there are links on the page you currently are on:

– There is a 85% chance you will follow a hyperlink on a page that you are currently on.

– There is a 15% chance you will jump to any web page in the network (with uniformprobability),

• If there are no links on the web page you are currently on (this page is called a “danglingnode”), you are equally likely to jump anywhere on the Internet.*note you can go right back to your current page which is necessary so we can have a regulartransition matrix*

2 4

1 3

5

6

224

Example 17.6. Below is a directed graph representing six web pages representing links to otherpages. For Example Page 5 links to 6 and 3. 6 is a dangling node and you have equal probabilityto any page in the network.a) Create the Transition Matrix for this Markov process.

2 4

1 3

5

6

b) Notice T is regular, so we can find a unique steady state vector. Let’s find this vector.

225

17.4 Application to Board Games

Example 17.7. Consider the board game above from your textbook, When Life is Linear. Hereare the rules: You start on square 1 and you win when you reach square 9. On each turn, you rolla die. Rolling a 1 or 2 means you do no move and stay on your current square. Rolling 3 or 4 youmove ahead one square If you roll a 5 or 6, you move ahead 2 squares. You must climb the ladderif you are in square 6 and slide down the snake if you land on square 4.

a) Find the transition matrix for this Board Game.

226

b) Use Markov Chains to determine the minimum moves it could take to win the game.

c) How many moves is it necessary for it to be 50% likely that someone has won?

227

228

17.5 Ice 11 -Markov Chains

1. A small, isolated town has two grocery stores, Archer’s Market and Eva’s Shoppe.While some customers are completely loyal to one store or another, there is anothergroup of customers who change their shopping habits each month. Of the shopperswho favor Archer’s Market one month, only 70% will still shop there the followingmonth, while Eva’s Shoppe retains 78% of its customer base each month. Everyone inthe town shops at one of the two stores, and no one from out of town ever shops ateither store. If Archer’s Market currently has 2500 customers and Eva’s Shoppe has1900 customers, how many customers will Archer’s Market have next month?

(a) 418(b) 1750(c) 2168(d) 3080

2. Referring to the scenario in the previous question, what will the product[0.70 0.220.30 0.78

]·[25001900

]tell us?

(a) This product will tell us the percentage of customers that will switch from onestore to the other store next month.

(b) This product will tell us the number of customers who will shop at each store nextmonth.

(c) This product will tell us the total number of customers who switched stores thismonth.

(d) This product doesn’t have any meaning.

229

3. Continuing the scenario from the previous questions, what does the (2, 1)-entry of the

matrix

[0.70 0.220.30 0.78

]3

represent?

(a) This represents the probability that a customer will switch from Archer’s Marketto Eva’s Shoppe between months 3 and 4.(b) This represents the probability that a customer will switch from Eva’s Shoppe toArcher’s Market between months 3 and 4.(c) This represents the probability that a customer who currently shops at Archer’sMarket will be shopping at Eva’s Shoppe three months from now.(d) This represents the probability that a customer who currently shops at Eva’sShoppe will be shopping at Archer’s Market three months from now.

4. Suppose we found that the steady-state (equilibrium) vector for this shop example is[11261526

]. What does the steady-state vector mean in the context of Archer’s Market and

Eva’s Shoppe?

(a) In the long-run, the probability of staying at Archer’s Market will be 11/26 andthe probability of switching to Eva’s Shoppe will be 15/26.(b) In the long-run, the probability of shopping at Archer’s Market will be 11/26 andthe probability of shopping at Eva’s Shoppe will be 15/26.(c) In the long-run, Archer’s Market will approach 11/26 of the market share, whileEva’s Shoppe will approach 15/26 of the market share.(d) I like cats!

5. Verify that

[11261526

]is a steady state vector for this Markov Process.

230

18 Orthogonality

So far, we have talked about how lovely life is when we have a matrix (often representing adynamical system) that is diagonalizable. Unfortunately, we don’t always have this property.Sometimes we have matrices that don’t have a complete and spanning eigenbasis and oftenwe don’t have square matrices. Luckily we have a way to deal with these less than idealsituations. This requires us to look at the matrix ATA (which will be square) and createsomething called an orthogonal basis. So first let’s talk about orthogonal vectors.

18.1 Orthogonal Vectors

Recall, when we discussed the Massey Method for sports ranking, we defined the followingtwo terms:

Definition 18.1. The inner product or dot product between two vectors u,v ∈ Rn, de-noted u · v = uT · v = vT · u = u1 · v1 + u2 · v2 + u3 · v3 + ...+ un · vn

Definition 18.2. The length or norm of a vector v∈ Rn is denoted ||v|| =√v2

1 + v22 + ...v2

n

We now have a few more definitions which are necessary for

Definition 18.3. Vectors u and v∈ Rn are orthogonal if u · v =

Definition 18.4. Given a set of vectors S = {u1, u2, u3, ..., um}, We say S is an orthogonalset if

Example 18.1. Show that S = {

−21−1

,

1−1−3

,

47−1

} is an orthogonal set.

What if we add

022

?

231

Example 18.2. Which of the following sets of vectors is not an orthogonal set?

(a) (1, 1, 1), (1, 0,-1)(b) (2, 3), (-6, 4)(c) (3, 0, 0, 2), (0, 1, 0, 1)(d) (0, 2, 0), (-1, 0, 3)(e) (cos θ, sin θ), (sin θ,− cos θ)

Theorem 18.1. Suppose u and v∈ Rn, then u ·v = 0 if and only if ||u+v||2 = ||u||2 + ||v||2.

Proof. Let u and v be orthogonal vectors, so by definition u · v = 0. Then ||u + v||2 =|u||2 + 2u · v + ||v||2= |u||2 + 0 + ||v||2.

Let’s verify this theorem for

−21−1

and

1−1−3

(Recall, we showed that these two matrices were orthogonal in Ex 1.)

u + v =

||u + v||2 =

||u||2 =√

22 + 12 + 122

= 6 and ||v||2 =√

12 + 12 + 322

= 11

||u||2 + ||v||2 =

Note:In general, ||u + v|| 6= ||u||+ ||v||

18.2 Orthogonal Spaces

Definition 18.5. Let V be a vector space. We say a vector u is orthogonal to V if u ·v = 0for every vector v ∈ V .

Definition 18.6. The set of all such vectors u that are orthogonal to V is called the or-thogonal complement of V and is denoted by .

Note: V ⊥ is a vector space.

232

Theorem 18.2. Let B = {v1, v2, ...vn} be a basis for V then u ∈ V ⊥ if and only if u · vi = 0for all vi ∈ B.

Heuristic Proof: If a vector is orthogonal to all of the “building blocks” (basis vectors), thenit is orthogonal to everything you can build/span with those vectors. Feel free to look attextbook for more details.

Example 18.3. Suppose V =span{

1102

,

111−1

,

2011

}. How can we find a basis for V ⊥?

233

Example 18.4. Suppose V =span{

−21−1

,

1−1−3

,

47−1

}. Find a basis for V ⊥.

Theorem 18.3. Given a subspace W ⊆ V , V = W + W⊥. This means any v ∈ V can bewritten as v = + .

18.3 Orthogonal Bases

Theorem 18.4. A set of nonzero orthogonal vectors is a linearly independent set of vectors.

Proof. Suppose {u1,u2, ...,un} is a set up nonzero orthogonal vectors. If 0 = a1u1 + a2u2 +...+ anun, then dotting both sides with u1 gives us0 = 0 ·u1 = a1u1 ·u1 + a2u2 ·u1 + ...+ anun ·u1. Since u1 is orthogonal to ui for i = 2, ..., nui · u1 = 0 for i = 2, ..., n. Thus, 0 = a1u1 · u1 + 0 + ...+ 0. Since u1 6= 0, the only solutionto this equation is if a1. We can continue to do this same method for ui for i = 2, ..., n toshow a1 = a2 = ... = an = 0. Therefore our set is linearly independent.

Definition 18.7. A basis made up of orthogonal vectors is called an orthogonal basis.

Why is this type of basis useful? Orthogonal sets are automatically linearly independent andalso because we will have a nice way to write any arbitrary vector as a linear combinationof our orthogonal basis. Orthogonal bases are also very stable.

234

Theorem 18.5. Suppose V has an orthogonal basis {v1,v2, ...,vn}, then any vector, u ∈ Vcan be written as u = a1v1 + a2v2 + ...+ anvn where

ai =vi · u||vi||2

=vi · uvi · vi

Proof. Let u ∈ V and suppose V has an orthogonal basis {v1,v2, ...,vn}. Then ∃a1, a2, ..., ansuch that u = a1v1 + a2v2 + ...+ anvn. Using a method similar to Thm 18.4, dotting bothsides by nonzero v1 gives us u · v1 = a1v1 · v1. Since v1 · v1 6= 0, we can solve for a1. Thus

a1 =v1 · uv1 · v1

. Similarly we can solve for ai for i = 2, .., n and determine that ai =vi · uvi · vi

.

Example 18.5. 16 Consider the set from Example 18.1: {

−21−1

,

1−1−3

,

47−1

}.

How do we know this is a basis for R3?

Write

3−15

as a linear combination of this basis.

16Ex from Holt’s Text

235

236

18.4 Projections

In this section, we will consider a particular type of linear transformation, a projection.Before we generalize this concept to subspaces, lets recall how we defined projections inCalculus III or Physics.

Definition 18.8. Let u,v ∈ Rn, with v 6= 0. The projection of u onto v is given by

projvu =u · v||v||2 v = (

u · vv · v )v.

Note: This may remind you of the coefficientsfrom our previous section which allowed us towrite any vector as a linear combination of anorthogonal basis.

We would like to generalize this for a subspace. Typically, we use the notation πV to denotethe linear transformation that projects vectors to a space V . Let’s first start with an examplein R3 to understand the idea.

Example 18.6. Suppose we have a vector u ∈ R3 and we want to project u onto anothervector v ∈ {(x, y, 0)|x, y ∈ R} (the x-y plane). In reality, we are projecting onto the linethat is parallel to v: L = {αv : α ∈ R}. That is, we want to apply the transformationπL : R3 → L. Then we are looking for the vector projv(u) shown in the figure on the leftbelow. Notationally, we write πL(u) = projv(u).

Now, suppose we want to project u onto the vector space spanned by two vectors, sayV = span {(1, 1, 1), (1, 0, 0)}. Then, we want to apply πV : R3 → V to get projV (u) shownin the figure on the right in the figure above.

237

In terms of the Linear Algebra language that we have been using, to find the projection ofa vector u onto a space V = span {v1, v2, . . . , vn}, we first find the part (or component) ofu that is orthogonal to the basis elements of V (thus, orthogonal to all vectors in V ). Wecall this component u⊥. Then, the projection, projV (u) is the vector that is left over afterwe subtract the orthogonal part: projV (u) = u− u⊥.Definition 18.9. Let V be a nontrivial subspace with an orthogonal basis {v1, v2, ...vn} then

the projection of u onto V is given by projV (u) =u · v1

||v1||2v1 +

u · v2

||v2||2v2 + ...+

u · vn||v1n||2

vn.

Note that this is just writing u as a linear combination of the orthogonal basis of V. SeePrevious Notes.

Theoretical Note: projV (u) does not depend on the choice of orthogonal basis for V.

18.5 The Gram-Schmidt Process for finding an Orthogonal Basis

Now we will discuss a method for finding an orthogonal basis for an arbitrary Vector Space.

Gram-Schmidt Process: Given a basis {b1, b2, ...bn} for a vector space V, we can createan orthogonal basis for V given by {v1, v2, ...vn} by the following process:v1 = b1

v2 = b2 − projv2b2

v3 = b3 − projv1b3 − projv2b3

v4 = b4 − projv1b4 − projv2b4 − projv3b4

...vn = bn − projv1bn − projv2bn − projv3bn − · · · − projvn−1

bn

Gram-Schmidt Process -Alternate Representation:v1 = b1

v2 = b2 −b2 · v1

v1 · v1

v1

v3 = b3 −b3 · v1

v1 · v1

v1 −b3 · v2

v2 · v2

v2

v4 = b4 −b4 · v1

v1 · v1

v1 −b4 · v2

v2 · v2

v2 −b4 · v3

v3 · v3

v3

...

vn = bn −bn · v1

v1 · v1

v1 −bn · v2

v2 · v2

v2 −bn · v3

v3 · v3

v3 − · · · −bn · vn−1

vn−1 · vn−1

vn−1

238

Computational Comment: The Gram-Schmidt process can suffer from significant round-off error. As we compute the orthogonal vectors, some dot vi · vj produces may not be closeto zero when |i − j| is large. This will cause a loss of orthogonality. There is a modifiedversion of Gram-Schmidt which requires more operations but is more numerically stable.Givens Rotations or Householder Reflections are two such methods, but they are beyond thescope of this course.

Example 18.7. Find an orthogonal basis for V =span {

−101

,

341

,

416

}

Note Eva has found v1 · v1 = 2,v2 · v2 = 24,b2 · v1 = −2,b3 · v1 = 2,b3 · v2 = 24.

239

Definition 18.10. A set of vectors {v1, v2, v3, ..., vn} is orthonormal if the set is orthogonaland ||vi|| = 1 for i = 1, ..n.

Normalizing Process: Given an orthogonal set {v1, v2, v3, ..., vn}, we can normalize thesevectors by dividing each vector by its norm.

When completing the Gram-Schmidt Process by hand, it is easier to normalizeeach orthogonal vector as you go!

Octave/MatLab: Matlab has a build in function that will orthogonalize the columns ofa basis for V. Create a matrix A of its basis elements. Then use [Q,R] = qr(A). Q givesyou a matrix with orthonormal columns. You should know how to do this process by handthough.

Example 18.8. Normalize your orthogonal basis from Ex 18.7: {

−101

,

242

,

3−33

}

240

Definition 18.11. A is an orthogonal matrix if its columns create an orthonormal set.

Theorem 18.6. If A is an n× n orthogonal matrix, then A−1 = AT .

Proof. Let A = aij be an n× n orthogonal matrix, Then AT = aji. Then

AAT =

a11 a12 a13 . . . a1n

a21 a22 a23 . . . a2n...

......

. . ....

an1 an2 an3 . . . ann

·

a21 a21 a31 . . . an1

a12 a22 a32 . . . an2...

......

. . ....

a1n a2n a3n . . . ann

=

a11 · a11 a12 · a21 a13 · a31 . . . a1n · an1

a21 · a12 a22 · a22 a23 · a32 . . . a2n · an2...

......

. . ....

an1 · a1n an2 · a2n an3 · a3n . . . ann · ann

Since the columns are orthogonal and normal, aij · aji = 0 for i 6= j and aij · ·aji = 1 fori = j. Thus AAT = I. Similarly we can show that ATA = I. Therefore A−1 = AT .

You can this on your own by showing that A−1 = AT for A =

−1√

21√6

1√3

0 2√6−1√

31√2

1√6

1√3

18.6 Column Rank

Definition 18.12. If all of the columns of a matrix A are linearly independent (are a basisfor ran(A)), we say A has full column rank.

Example 18.9. A =

1 2 1 00 1 0 10 0 1 30 0 0 10 0 0 0

has full column rank.

Definition 18.13. If all of the rows of a matrix A are linearly independent, we say A hasfull row rank.

Example 18.10. B =

1 2 1 0 00 1 0 1 00 0 1 3 00 0 0 1 0

has full row rank, but NOT full column rank.

Definition 18.14. If all of the rows AND columns of a matrix A are linearly independent,we say A has full rank.

241

Property: If n × m matrix A has full column rank, what is the relationship between nand m?a) n ≥ mb) n ≤ mc) n = md) I love math!

Property: If n×m matrix A has full row rank, what is the relationship between n and m?

a) n ≥ mb) n ≤ mc) n = md) I don’t want this class to ever end!

Property: If n×m matrix A has full rank, what is the relationship between n and m?

a) n ≥ mb) n ≤ mc) n = md) I love cats!

18.7 QR Factorization

In linear algebra it is often useful to represent matrices in another form. We have alreadydiscussed the value of Diagonalizing a matrix, when it is possible. We will not talk aboutall of our different factorizations of matrices, but we will end our time in linear algebradiscussing both the QR and SVD factorizations of matrices

QR Factorization: Let A be an n × m matrix with linearly independent columns (akafull column rank), then A can be factorized as A = QR where Q is an n × m orthogonalmatrix (has orthonormal columns) and R is an m×m upper triangular matrix with positivediagonal entries.

Note: A does not need to be square, but since A has full column rank, we need the numberof columns of A to be than the number of rows of A.

QR Process:

1. Find an orthonormal basis for the columns of A. (using Gram-Schmidt)

2. Find R by computing R = QTA (works because Q is orthogonal so Q−1 = QT ).

242

Example 18.11. Calculate the QR factorization for A =

−1 3 40 4 11 1 6

.

243

18.8 Usefulness of QR Factorization

Diving into Numerical Analysis:Linear Algebra Has Two Fundamental Problems:

1. Solving Ax = b

2. Diagonalizing a matrix A (finding eigenvalues)

Problems: Theoretically perfect algorithms can be very numerically unstable. Errors occurand sometimes are unavoidable and can compound especially for real life applications whenwe have large matrices.

Solution: Orthogonal matrices are the best for numerical stability so we want to use themif we can.

Methods for Solving Ax = b

• Gaussian Elimination/LU Decomposition Advantages:

– Works for any matrix

– Finds all solutions when they exist

– Easy to program (rref!)

– Fast

Disadvantages:

– Can be unstable

– Can’t be used to approximate solutions when systems have no solution (aka leastsquares)

• QR Decomposition (using Gram-Schmidt): Ax = b can be solved by QRx = b→ Rx =QT b which is easier to solve.Advantages:

– Works for any matrix

– Finds all solutions when they exist

– Easy to program

– Rx = QT b is well conditioned

– Finds approximate solutions

Disadvantages:

– Slower than LU

– More complicated

244

19 Singular Value Decomposition

19.1 Spectral Theorem

We have already discussed the value of diagonalizing a matrix. Recall, A is diagonalizableif there exists matrices P and D such that A = . Unfortunately we can’t alwaysdiagonalize a matrix. When can’t we diagonalize a matrix?

Fortunately, we have a useful factorization called Singular Value Decomposition which willwork for all matrices. Next class, we will talk several of the applications of SVD. Before wediscuss these applications, we will first talk about a very nice theorem which is invoked inSVD.

Symmetric Matrices and Orthogonal Matrices have many nice properties. For example, lastclass we learned that if P is orthogonal then P−1 =

Theorem 19.1. If A is symmetric, then the eigenvectors associated with distinct eigenvaluesare orthogonal.

Pf. Omitted. See supplementary text.

Why is this nice?

Theorem 19.2. ATA has nonegative, real, eigenvalues. (Remember ATA is symmetric!)

Definition 19.1. A square matrix A is orthogonally diagonalizable if there exists anorthogonal matrix P and diagonal matrix D such that A = PDP−1.

Note: This is the same idea as diagonalization, just normalize your eigenvectors (since bythe previous theorem, they are orthogonal!).

Theorem 19.3. The Spectral Theorem: A matrix is orthogonally diagonalizable if andonly if A is symmetric.

Pf. Omitted. The complete proof is rather difficult and is not included here. Often under-graduate linear algebra textbooks do not include the complete proof since half of the proofis easier to prove using complex numbers.

This means if A is symmetric then, A is !

245

Example 19.1. Orthogonally diagonalize ATA for A =

2 11 00 1

. Eva found that the eigenspace

associated with λ = 1 is {[−1√

52√5

]}.

19.2 Singular Value Decomposition

Idea Behind SVD: For any m×n matrix A, ATA is symmetric which means it is orthogo-nally diagonalizable! From Theorem 19.2 on the previous page, we know the eigenvalues forATA are nonegative. This allows us to order our eigenvalues in decreasing order:

λ1 ≥ λ2 ≥ λ3 ≥ ... ≥ λn ≥ 0

Definition 19.2. The singular values of A are the square roots of the eigenvalues ofATA and are denoted {σ1, σ2, ...σn}.That is σi =

√λi.

246

Note: The singular values of A are the lengths of the vectors Av1, Av2,.. Avn where the vi’sare eigenvectors.

Theorem 19.4. Let {v1, v2, ...vn} be an orthonormal basis of ATA arranged in order ofeigenvalues: λ1 ≥ λ2 ≥ ...λn. And suppose A has r nonzero singular values. Then{Av1, Av2, ...Avr} is an orthogonal basis for the range/column space of A and the rank of Ais r.

Pf. Omitted. Please see supplementary text for details.

Theorem 19.5. Singular Value Decomposition: Suppose A is an n×m matrix withrank r. Then there exists an n×m matrix Σ (see below) for which the diagonal entries in Dare the first r singular values of A σ1 ≥ σ2 ≥ σr > 0 and an n× n orthogonal matrix U andm×m orthogonal matrix V such that A = UΣV T

Σ is a matrix that is the same size as A. We have two cases for Σ:

First Let D be the diagonal matrix with ordered singular values:

D =

σ1 0 . . . 00 σ2 . . . 0...

......

0 0 . . . σr

Case 1: n ≥ m: (A is a tall matrix)

Σ =

[D0

]Case 2: n ≤ m: (A is a wide matrix)

Σ =[D 0

]

Note: For case 2: we can sometimes make life easier for ourselves by considering B = AT .Then B is a matrix with Case 1: and if B = UΣV T then A = BT = (UΣV T )T =

Octave Code for SVD: [U,S,V] = svd(B)

Theorem 19.6. Every n×m matrix A has a singular value decomposition.

Heuristic Proof: Follow the construction from Thm 19.5 by completing the steps below.

247

Steps for Singular Value Decomposition:

1. Orthogonally Diagonalize ATA to find V (V will be our P when ATA = PDP T ).

2. Find Σ using the singular values of A.

3. Find U in 2 Steps

(a) Find the first entries by ui =1

σiAvi

Why? Our goals is to find U so that A = UΣV T aka AV = UΣ. The ith column of AVis Avi and the ith column of UΣ is σiui. So we need Avi = σiui. Thusui = 1

σiAvi

(b) Fill out U by extending to an orthonormal basis of Rn. Basically you can do this by

adding the normalized vectors from null(AT ). Here is why: The first vectors you find

using ui =1

σiAvi, actually give you a basis for the range space of A (aka the column

space). The column space is a subspace of R3 and remember our theorem which states

V = W⊕W⊥ for any subspace W. So in our case, R3 = (ran(A))

⊕(ran(A))⊥. It

turns out (ran(A))⊥ = null(AT ) (proof omitted).

Example 19.2. Find the singular value decomposition (SVD) for A =

2 11 00 1

.

Recall, in Ex 19.1 we found ATA =

[2√5−1√

51√5

2√5

]·[6 00 1

]·[

2√5

1√5

−1√5

2√5

].

248

Example 19.3. Find the SVD for A =

[−1 1 02 2 1

]

If you can use technology, load up Octave type [U,S,V]=svd(A). To do this by hand, do the follow-

ing:

Step 1: Orthogonally Diagonalize ATA to find V (V will be our P when ATA =PDP T )

Note ATA =

−1 21 20 1

·[−1 1 02 2 1

]=

5 3 23 5 22 1 1

. We could TOTALLY Do this, but if we

diagonalize B = AT , we will get a 2 by 2 matrix which is easier. Let’s do this.Let B = AT .

Then BTB = AAT =

[−1 1 02 2 1

]·

−1 21 20 1

=

[2 00 9

]= M

We first find the eigenvalues of M. Because it is a diagonal matrix, we know the eigenvaluesare λ1 = 9, λ2 = 2 (Note that I ordered them from largest to smallest.)

Next we find the eigenspaces:

E9 = null(M − 9I) = null(

[−7 00 0

])⇒

(−7 0 00 0 0

). So x = 0, y = y. So E9 =span{

[01

]}

We need to normalize this vector, but it already has length 1 so we don’t have to do anything else.

E2 = null(M − 2I) = null(

[0 00 7

])⇒

(0 0 00 7 0

). So x = x, y = 0. So E2 =span{

[10

]}

We also need to normalize this vector, but it already has length 1 (neat!).

Thus, BTB = PDP T =

[0 11 0

]·[9 00 2

]·[0 11 0

](Note in this example P T = P ).

Thus the V for B is P =

[0 11 0

]. So V T = P T = P .

Step 2: Find Σ using the singular values of A.

Since B is a 3× 2 matrix, Σ for B is a 3× 2: Σ =

3 0

0√

20 0

. Remember we put the singular

values for the B along the diagonals (the square roots of the eigenvalues.)

We are almost done! Go to the next page!

249

Step 3: Find U in 2 Steps:

1) Find the first entries by ui =1

σiAvi.

2) Then Fill out U by extending to an orthonormal basis of Rn by finding thenullspace of AT .

Since we are using B, we use ui =1

σiBvi:

u1 = 13

−1 21 20 1

·[01

]= 1

3

221

=

232313

.

u2 = 1√2

−1 21 20 1

·[10

]= 1√

2

−110

=

−1√

21√2

0

.

So we have found the first two entries of U for B. We need to fill out U by finding vectors in

null(BT ) = null(

[−1 1 02 2 1

])⇒

(−1 1 0 02 2 1 0

)⇒(

1 0 14

00 1 1

40

)So x = −z

4, y = −z

4, z = z.

So the nullspace is

−1−14

We need to normalize this vector:

||

−1−14

|| =

√(−1)+(−1)2 + 42 =

√18, so our 3rd vector is

−1√

18−1√

184√18

.

Hurray we have completed U! U =

23−1√

2−1√

1823

1√2

−1√18

13

0 4√18

Thus B = UΣV T =

23−1√

2−1√

1823

1√2

−1√18

13

0 4√18

3 0

0√

20 0

[0 11 0

]

Last Step because we were using B = AT :If B = UΣV T , and BT = A, then A = BT = (UΣV T )T = V ΣTUT .

So A = V ΣTUT =

[0 11 0

] [3 0 0

0√

2 0

]

23

23

13−1√

21√2

0−1√18

−1√18

4√18

250

19.3 Applications of SVD17

Singular values appear in many linear algebra applications especially applications like clustering(eigenvectors help with this too!), statistics, signal processing, and other applications that involvebig data. If we had an Applied Linear Algebra II course (oh boy!) we would go into more examplesof these applications, but for now we will just discuss one major application and discuss a moregeneral use of SVD which will give some insight on why it is important for applications using bigdata.As we have seen in our radiography/tomography labs, Matrices can be used to store and transferdigital images. SVDs of matrices can help use complete these processes efficiently. That is, SVDscan help reduce redundant information inside an image. They do this by takinge advantage of thefact that pixels near one another in a picture often have similar greyscale levels. Thus it may bepossible to represent the image using less storage space while still keeping the essential parts of thepicture. Now SVD isn’t always the typical method used in data compression algorithms. There areother methods and versions of SVD which do an even better job. Nevertheless, Singular Value De-composition still gives us an idea about how image compression can be done and why it sometimesleads to blurrier pictures than the original. SVD also is used in Principle Component Analy-sis, a technique to organize analyze multivariate data (and can used for face recognition algorithmsand merging of pictures of faces). Unfortunately, we do not have time to cover this interesting topic.

General Idea: If A = UΣV T , we can use the outer product expansion to expressA = σ1u1v

T1 + σ2u2v

T2 + ...+ σnu1nv

Tn where σ1 ≥ σ2 ≥ ... ≥ σn > 0.

The terms with the largest singular values contain most of the “data” or information about the

image whereas smaller singular values don’t contribute as much. So the idea is, we can sometimes

discard many of these smaller low singular value terms, and still get a good approximation of our

original image. Thus we can have A ≈ σ1u1vT1 + σ2u2v

T2 + ...+ σku1kv

Tk for k < n.

17Images from our text When Life is Linear

251

By the numbers: If an image has m by n pixels, the storage requires m · n numbers.

But if we approximate A with k singular values, A ≈ σ1u1vT1 +σ2u2v

T2 + ...+σkukv1k

T , this requires

only (m+ n) numbers! Savings Galore!

19.4 Approximating Matrix Rank

Most numerical calculations used in solving Ax = b are as reliable as they can be whenwe use the Singular Value Decomposition for A. We have mentioned before that orthogonalmatrices are nice and stable. That means that they do not affect lengths of vectors or anglesbetween vectors, which decreases the likelihood of error. It also means that small changes tothe data will not cause large changes to the solutions to the system. What this means is thatthe U and V T vectors in the SVD of A, don’t have the errors. These errors are stored in Σ.If the singular values of A are super, super small, it is often really difficult to avoid roundofferror. We can approximate or anticipate error by calculating something called the conditionnumber of a matrix. This condition number affects the sensitivity of a solution to Ax = b toerrors/changes in the entries of A -that is it affects the stability of A. This condition numbercan be calculated many different ways, but the most commonly used calculation is the ratio

252

of the largest and smallest singular value of A.

Definition 19.3. If A is an invertible n× n matrix, then the ratio of σ1σn

of the largest andsmallest singular values of A gives us the condition number of A.

Numerical Note: The larger the condition number, the closer the matrix is to being sin-gular (that is, non invertible). For example, condition number of In is 1 and a non-invertiblematrix has an infinite condition number. If a matrix is really closed to be being singular, wesay it is ill-conditioned. Ill-conditioned matrices can become singular if some of the entriesare changed very slightly -which means they are in trouble if we have round-off error!

Example 19.4. Suppose A is 4 by 4 matrix with singular values σ1 = 2, σ2 = 1, σ3 = 10−1

and σ4 = 10−3. What is the condition number of A?

We can use the condition number of a matrix to estimate the accuracy of a solution toAx = b. Supposed the condition number of A is 10k (k > 0) and A and b are accurate to rsignificant digits, then the calculated solution to Ax = b should be accurate to at least r− ksignificant digits.

Definition 19.4. Round-off error also effects a computer’s ability to determine the rank ofa matrix. The numerical rank of a matrix is the perceived rank of a matrix by a computer.

Definition 19.5. The “Machine epsilon” (ε) of a computer gives an upper bound on therelative error due to rounding in floating point arithmetic.

Recall, relative error is |actual− approx

approx|.

Knowing a computer’s Machine epsilon and a matrix’s SVD can help us determine the nu-merical rank of a matrix. Here’s How:

Determining the Numerical Rank of a Matrix: Suppose a computer has a MachineEpsilon, ε and suppose matrix A has singular values σ1 ≥ σ2 ≥ ... ≥ σn. Let b = σ1 · ε · nwhere n is the dimension of A and let k be the largest integer such that σk ≥ b. Then k isthe numerical rank of A.

Example 19.5. Suppose A is 4 by 4 matrix with singular values σ1 = 2, σ2 = 1, σ3 = 10−1 and

σ4 = 10−7 and we have a Machine Epsilon, ε = 10−5. What is the numerical rank of A? What is

the rank of A?

253

SVD can help decrease noise in data: When an image has noise (errors or randomvalues), sometimes a lower rank approximation can improve the quality of the data. Whileyou may not get the perfect completely-non-noise data back, reducing the noise can helpyou get a better underlying picture of what is happening with the data. So sometimes, yourgoal is to reduce the noise.

In some circumstances you know there is noise, but you don’t know how much noise. So howdo we know which rank k approximation we should use to approximate our matrix? Onemethod to help is to plot the singular values of our matrix. If we look at our figure below(from our text When Life is Linear), notice for what value we have a steep drop-off. Thisdrop-off tell us which rank k approximation we should use.

We will now explore some MatLab scripts written by Tim Chartier to further explore these appli-

cations and if we have time do another Radiography and Tomography Lab!

254


In this lab, we will consider the cases when the matrix representation of the radiographictransformation does not have an inverse. (Note: we say that T : V → W has an inverseT−1 : W → V if T−1T = IV , the identity mapping from V to V , and TT−1 = IW the identitymapping from W to W . We say that T : V → W has a one-sided inverse18, P : W → Vif P ◦ T = IV .)

For example, let T : R2 → R3 be given by the matrix

A =

1 10 −10 −1

.

This has a one-sided inverse matrix P given by

P =

(1 2 30 5 6

).

P is the one-sided inverse because PAx = x. That is PA = I2 as shown below.

PA =

(1 2 30 5 6

)

1 10 −10 −1

=

(1 00 1

)

Begin by proving the following theorem.

Theorem 20.1. Let V and W be finite-dimensional vector spaces and T : V → W a lineartransformation. If rank(T ) = dim(V ) then T TT is invertible19.

Hint: What type of matrix is T TT ... and use a result from our notes.

Proof:

18More specifically, this is a left inverse of T .19Note that since V and W are finite-dimensional spaces, we can fix bases for each, and then in those

coordinates, the transformation T can be represented by a matrix. TT is the transpose of this matrix

255

A more formal way to prove this is as follows:

Proof. Suppose V and W are finite-dimensional vector spaces and suppose T : V → W isa linear transformation such that rank(T ) = dim(V ). Thus T : V → W has full rank sincerank(T ) = dim(V ) = n. This means that ran(T ) is a subspace of W with the same dimension

as V . So, we know that if we define the restriction of T T to ran(T) to be T T : ran(T )→ V ,

then T T , having full rank maps onto V . That is, ran(T T ) = V . So, T TT : V → V hasonly a trivial null space and therefore T TT has linear independent columns and thus by theInvertible Matrix Theorem has an inverse.

Here is an alternate proof:

Proof. We can show that null(T TT ) = {0}, then because T TT is square we have that T TTis invertible. Let x ∈ null(T TT ). We have

0 = xT0 = xTT TTx = (Tx)T (Tx) = ‖Tx‖2

which implies Tx = 0, that is, null(T TT ) = null(T ). Now, since rank(T ) = dim(V ),null(T ) = {0} and null(T TT ) = {0}. Therefore T TT has linear independent columns andthus by the Invertible Matrix Theorem has an inverse.

256



(Reconstruction Without an Inverse)

Inverting Radiographic Transformations

In this section, we will consider the following example: We are given a radiograph with24 pixels that was created by applying some radiographic transformation, T to an objectwith 16 voxels.

1. (Review Question) Give a scenario for a radiographic transformation T that fits theabove example. Don’t calculate a matrix T , rather give the following:

- Size of the object: × .

- Number of pixels per view:

- Number of views:

2. Suppose we know the radiograph b and we want to find the object x, where Tx = b.

(a) What properties must the transformation T have so that the one-sided inverse ofT exists?

(b) When N ≤M (as in the example above), what matrix properties must the matrixrepresentation of T have so that it has a one-sided inverse?

3. For ease of notation, we use the same notation for the matrix and the transformation,that is, we call the matrix representation of T , T . Suppose, N ≤ M and a one-sidedinverse, P of T exists. This means that for all objects x with radiographs Tx = b,we have x = Pb. If T were invertible, then P = T−1. But, in the example above, weknow that T is not invertible (it is not even a square matrix!). Using the followingsteps, find the one-sided inverse of T .

1

257


(a) One strategy is to compose T with a linear transformation A : RM → RN so thatthe product is invertible. (Can you see why this would be a good idea?) If Tx = band A is any linear transformation (whose domain is RM), then ATx = Ab. SinceT is one-to-one, we know that in order for the composition A ◦ T = AT to beinvertible, the only vector in ran(T ) that can be in null(A) is the zero vector.(Think about this!) What other properties must A have so that AT is invertible?

(b) Provide a matrix A that has the properties you listed in 3a and such that thecomposition AT is invertible.

(c) Solve for x in the matrix equation ATx = Ab using the A you found, and providea representation of the one-sided inverse P .

4. Putting this all together now, state a necessary and sufficient condition for T to havea one-sided inverse.

2

258


Application to a Small Example

Consider the following radiographic example.

• Total number of voxels: N = 16 (n = 4).

• Total number of pixels: M = 24

• ScaleFac = 1


• View angles: θ1 = 0◦, θ2 = 20◦,θ3 = 40◦,θ4 = 60◦,θ5 = 80◦,θ6 = 100◦.

1. Use tomomap.m to compute T and verify that the one-sided inverse of T must exist.Note that function tomomap returns a transformation matrix in sparse format. To useand view as a full matrix array use the command T=full(T); after constructing T .

2. Compute the one-sided inverse P . Use P to find the object that created the followingradiograph vector (by copying and pasting into Octave or Matlab from the electroniccopy of this document in our class folder on the P: drive):

b=[0.00000

32.00000

32.00000

0.00000

1.97552

30.02448

30.02448

1.97552

2.71552

29.28448

29.28448

2.71552

2.47520

29.52480

29.52480

2.47520

1.17456

30.82544

30.82544

1.17456

1.17456

30.82544

30.82544

1.17456]

Use the following code (with the modifications indicated!):

3

259


P= ; %insert your formula for P here!!

x= ; %insert what x is in terms of other quantities here!!

figure;

imagesc(reshape(x,a,b)); %change a and b to the dimensions of your object!

colormap(gray);

caxis([0 16]);

colorbar

Here are some notes about the commands in the code above:

• reshape(x,a,b) takes x and reshapes it into a a× b matrix.

• imagesc is a more general array plotting command than imshow. While imshowdoes a good job at plotting arrays with square pixel size and nice grayscale, itwill plot small arrays too small to be seen!

• colormap(gray) tells imagesc to use a grayscale plot.

• caxis declares the grayscale limits, in just the same way that imshow does it. Thedifference is that it is a separate command with imagesc.

• colorbar forces the plot to contain a colorbar reference.

4

260


Application to Brain Reconstruction

Now, let’s reconstruct some brain images from radiographic data. This section will guideyou in this process.

1. Collect the necessary provided files and place them in your working Octave/Matlabdirectory.

(a) Data File: Lab5Radiographs.mat

(b) Plotting Script: ShowSlices.m

(c) Octave/Matlab Code: tomomap.m

2. Choose whether to create a new script file (“.m” file) for all of your commands or towork at the Octave/Matlab prompt.

3. Load the provided radiographic data with the following command.

load Lab5Radiographs.mat

This line loads an array variable named B which is a 12960x362 array. Each column is aradiograph corresponding to one horizontal slice of the human head. Each radiographhas 12960 total pixels spread across many views. The first 181 columns are noiselessradiographs and the last 181 columns are corresponding radiographs with a smallamount of noise added.

4. Use the familiar function tomomap.m to construct the transformation operator T corre-sponding to the following scenario (which is the scenario under which the radiographicdata was obtained): n = 108, m = 108, ScaleFac = 1, and 120 view angles: the firstat 1◦, the last at 179◦, and the rest equally spaced in between (hint: use the linspacecommand).

5. Some Octave/Matlab functions do not work with sparse arrays (such as your T ). So,simply just make T a full array with this command:

T=full(T);

6. It is tempting to compute the one-sided inverse P as found in 3c of the first part.However, such a large matrix takes time to compute and much memory for storage.Instead we can use a more efficient solver provided by Octave/Matlab. If we seek asolution to Lx = b, for invertible matrix L, we find the unique solution by findingL−1 and then multiplying it by b. Octave/Matlab does both operations together in anefficient way (by not actually computing L−1) when we use the command x=L\b.

Let’s try this with the first radiograph in the matrix B. That is, we will reconstructthe slice of the brain that produced the radiograph that is represented in the 50thcolumn of B. We want to solve the equation you found in 3c (recall the matrix youfound to be A). Using the A you found in 3c in the following commands

5

261


b=B(:,50);

x=(A*T)\(A*b);

you will find your first brain slice reconstruction. Now to view your reconstruction,use the following commands

figure;

x=reshape(x,108,108);

imshow(x,[0,255]);

(The reshape command is necessary above because the result x is a (108 · 108) × 1vector, but we want the object to be 108× 108 image.)

7. Notice also that x and b could be matrices, say X and B. In this case, each columnof X is the unique solution (reconstruction) for the corresponding column of B. Usethese ideas to reconstruct all 362 brain slices using a single Octave/Matlab command.

Use the variable name X. Make sure that X is an 11664x362 array. Now, the first181 columns are reconstructions from noiseless data and the last 181 columns arereconstructions from noisy data.

8. Run the script file ShowSlices.m which takes the variable X and plots example slicereconsructions. Open ShowSlices.m in an editor and observe the line

slices=[50 90 130];

You can choose to plot any three slices by changing the slice numbers. In the figure,the left column of images are reconstructions from noiseless data, the right column ofimages are reconstructions from the corresponding noisy data. IMPORTANT: Onceyou have computed X you need only run ShowSlices.m to view different slices; runningthe other commands described above is time consumming and unnecessary.

Print out a picture of your favorite set of 3 slices to turn in with your work from thislab.

6

262


Noisy Data and Reconstructions

Congratulations! You have just performed your first brain scan tomography. Using yournew tool, answer the following questions.

1. Are the reconstructions from the noiseless radiographs exact discrete representationsof the brain? Explain, using linear algebra concepts.

2. Find a method for determining the relative amount of noise in the noisy radiographs.It might be useful to know that the noisy radiograph corresponding to the radiographrepresented in the 50th column of B is in the (181 + 50)th column of B. (Note: Thereare many ways you can do this, be creative.) Is there a lot or a little noise added tothe radiographs?

3. Are the reconstructions from the noisy radiographs exact representations of the brain?

4. Compare the degree of “noisiness” in the noisy radiographs to the degree of “noisi-ness” in the corresponding reconstructions. Draw some conclusion about what thiscomparison tells us about the practicality of this method for brain scan tomography.

7

263

264

21 Practice Problems and Review for Exams

The following pages are practice problems and main concepts you should master on eachexam. Core concepts will have heavier weight than regular concepts.

These problems are meant to help you practice for the exam and are often harder than whatyou will see on the exam. You should also look over all ICE sheets, Homework problems,Quizzes, and Lecture Notes. Make sure you can do all of the problems in these sets.

Disclaimer: The following lists are topics that you should be familiar with, and these areproblems that you should be able to solve. This list may not be complete. You are respon-sible for everything that we have covered thus far in this course.

I will post the solutions to these practice exams on our blackboard site. If you find anymistakes with my solutions, please let me know right away and feel free to email at any hour.

Good Luck,Dr. H

265

MA 307 –Review for Exam 1

Remember to look over all Homework problems, Daily Assignments, and Lecture Notes (all are

posted). Make sure you can do all of those problems. You can also practice with the problems

listed as part of this working review. Disclaimer: The following is a list of topics that you should

be familiar with, and a list of problems that you should be able to solve. This list may not be

complete. You are responsible for everything that we have covered thus far in this course.

Main Concepts for Exam 1

Mastery Concept 1: Gaussian EliminationMastery Concept 2: Understanding solutions to systems of equationsMastery Concept 3: Vector Spaces and SubspacesMastery Concept 4: SpanMastery Concept 5: Basis/Linear IndependenceMastery Concept 6: Linear Transformation/Functions Pt 1* Note you may need tofind a matrix representation for a linear transformation (depending on time)

1. Gaussian Elimination

(a) Know how to use Gaussian Elimination to solve systems of linear equations.

i. Know how to write solutions sets of systems

ii. Be able to create a Row Reduced Echelon Form Matrix using Row Op-erations

iii. Be able to do Gaussian Elimination by hand and also by using technologylike Octave/Matlab or your calculator

2. Solutions to systems of equations

(a) Know how to write solution sets for row reduced echelon form.

(b) Know how to write solution sets as spans or parametric forms using free variables.

(c) Know how to recognize a consistent system or inconsistent system.

3. Vector Spaces

(a) Know the 10 properties that set need to satisfy in order to be defined as a vec-tor space -remember these spaces only have 2 operations -addition and scalarmultiplication!

(b) Know how to recognize when a set is not a vector space (usually you should checkthe closure and identity properties)

(c) Be able to list examples of Vectors spaces and understand why they are vectorspaces. Some examples include, but are not limited to:Rn, GF (2)n, Mn×m, Pn, and The space of greyscale images, a spanning set, thesolution space for a homeogeneous system, the range space of a transformation,the nullspace of a transformation,(the last two may show up on Exam 2)...

266

4. Subspaces

(a) Know how to test whether or not a subset of a vector space is a subspace (thatis, the set acts like a self-contained vector space.)

(b) How to show a subset, W, is a subspace:Method 1:

i. Check if 0 ∈ W (that is make sure the additive identity is in W.)

ii. Check that W is closed under linear combinations. That is let u,v be arbi-trary vectors in W and make sure αu + βv ∈ W for arbitrary scalars α, β.

Method 2:

i. Show that the subspace can be written as a span of vectors.

5. Span

(a) Know the definition of a span of vectors :span(v1, v2, ...vn)={a1v1 + a2v2 + ...+ anvn|ai ∈ R}(the set of all linear combinations of v1, ...vn.)

(b) Be able to write a vector space as a span of vectors. Another way to say this isto find a spanning set for a vector space.

i. Process: get a set in the form:{a1v1+a2v2+...+anvn|ai ∈ R} so that there areno restrictions on the scalar coefficients. Often this can be done by creatingan augmented matrix of the requirements of the set and using free variablesto help find your coefficients.

ii. See HW, Spanning Notes

(c) Know the definition of a spanning set :{v1, v2, ...vn} is a spanning set for V if span(v1, v2, ...vn) = V .

(d) Be able to determine whether or not a particular vector is in the span of a set ofvectors.

i. Process: Find coefficients ai such that ~u = a1v1 + a2v2 + ... + anvn, then~u ∈span(v1, v2, ...vn)

ii. See HW Spanning Notes

(e) Be able to show whether or not a set of vectors spans a whole Vector Space (thatis, be able to show a set is a spanning set for a vector space.)

i. Process: Pick an arbitrary vector from the Vector Space and try to find scalarsai such that the vector can be written as a linear combination of the set ofvectors.

6. Linear Independence

(a) Know the definition of linear independence and dependence

267

(b) Know how to determine whether or not a set of vectors is linear independent ordependent. Here is the process to check whether or not {v1, v2, ...vn}:Step 1: Set an arbitrary linear combination of vi’s equal to 0:α1v1 + α2v2 + ...+ αnvn = 0Step 2: See if you can find values for the αi’s that are not all 0. If you can, theset is dependent. If α1 = α2 = ... = αn = 0 is the only possible values for the α’s,then the set is linear independent.

7. Basis

(a) Know the standard basis for Rn, Pn, and Mn×m.

(b) Know the requires/definition of a basis: A set {v1, v2, ...vn} is a basis for V if bothof the following are true:

i. Span({v1, v2, ...vn})=V

ii. {v1, v2, ...vn} is linear independent

(c) Know how to find a basis for a set

8. Introduction to Linear Transformation/Functions

(a) Know the definition of a Linear Transformation:T : U → V is a linear transformation if T (αu + βv) = αT (u) + βT (v) for anyarbitrary linear combination αu+ βv ∈ U .

(b) Know how to test whether or not a Function/Mapping is a Linear Transformation:Process:

i. Method 1: Show that T (αu + βv) = αT (u) + βT (v) for any arbitrary linearcombination.

ii. Method 2: Show that both of the following hold:

A. T (u+ v) = T (u) + T (v)

B. T (αu) = αT (u)

iii. Note: If T (0) 6= 0, T is NOT a linear transformation. But if T (0) = 0, youmust use Method 1 or 2 to show it is a linear transformation.

(c) Know how to find images of linear transformations.(Given T and x, what is T (x)?)

(d) Know how to find a vector whose image is produced by a linear transformation.(Given T and b, what x gives T (x) = b?)

9. Matrix Representation of a Linear Transformation

• Know that every linear transformation can be represented in a matrix form.

• Process for finding the matrix form for a linear transformation (aka the changeof basis matrix) for T : V → W

Step 1: Find the basis elements for the domain (V), codomain (W), and ran(T)(the range space of T)Step 2: Map the basis elements of V to the basis elements of ran(T)268

Step 3: Find the coordinate vectors or our ran(T) elements with respect to thebasis for the codomain (W). That is write our ran(T) basis elements as a linearcombination of the basis elements of the codomain, then list the coefficients intoa column vectorStep 4: Construct your matrix representation by using the coordinate vectors asthe column vectors for M.

• You should know how to check whether your matrix representation does the samething as the transformation. Do this by taking the a domain element and ap-plying T to the element. Write the image of this element as a coordinate vectorof the codomain. check to see if this coordinate vector gives you the same thingas writing your domain element as a coordinate vector and applying your ma-trix representation to it. Then check to see if your vector output matches thecoordinate vector you got earlier.

269

Practice Problems:

1. Give the solution set of each system.

(a)3x+ 2y + z = 1x− y + z = 2

5x+ 5y + z = 0

(b)

x+ y − 2z = 0x− y =−3

3x− y − 2z =−62y − 2z = 3

(c)2x− y − z + w = 4x+ y + z =−1

(d)x+ y − 2z = 0x− y =−3

3x− y − 2z = 0

2. Verify that each is a vector space by checking the conditions.

(a) The collection of 2×2 matrices with 0’s in the upper right and lower left entries.

{(a 00 b

)| a, b ∈ R}

(b) The collection {a0 + a1x+ a3x3 | a0, a1, a3 ∈ R} of cubic polynomials with no

quadratic term.

3. Determine if each set is linearly independent (in the set’s natural vector space).

(a) {

120

,

−110

} ⊆ R3

(b) {(1 3 1), (−1 4 3), (−1 11 7)} ⊆ M1×3

(c) {(

5 41 2

),

(0 00 0

),

(1 0−1 4

)} ⊆ M2× 2

4. Is the vector in the span of the set?

103

{

21−1

,

1−11

}

5. Find a basis for each space. Verify that it is a basis.

(a) The subspace M = {a+ bx+ cx2 + dx3 | a− 2b+ c− d = 0} of P3.270

(b) This subspace of M2×2.

W = {(a bc d

)| a− c = 0}

6. Give two different bases for R3. Verify that each is a basis.

7. Find a basis for, and the dimension of, each space.

(a)

{

xyzw

∈ R4 | x− w + z = 0}

(b) the set of 5×5 matrices whose only nonzero entries are on the diagonal (e.g., inentry 1, 1 and 2, 2, etc.)

(c) {a0 + a1x+ a2x2 + a3x

3 | a0 + a1 = 0 and a2 − 2a3 = 0} ⊆ P3

8. Give a basis for the span of each set, in the natural vector space.

(a) {

113

,

−120

,

0126

} ⊆ R3

(b) {x+ x2, 2− 2x, 7, 4 + 3x+ 2x2} ⊆ P∈9. Verify that each map is a linear transformation.

(a) h : P3 → R2 given by

ax2 + bx+ c 7→(a+ ba+ c

)

(b) f : R2 → R3 given by(xy

)7→

0x− y

3y

More Practice Problems:Go to the online HW system for more practice problems.

271

272








Mastery Concept 1: Gaussian EliminationMastery Concept 2: Understanding solutions to systems of equationsMastery Concept 3: Vector Spaces and SubspacesMastery Concept 4: SpanMastery Concept 5: Basis/Linear IndependenceMastery Concept 6: Linear Transformations/FunctionsMastery Concept 7: Matrix representation for a linear transformationMastery Concept 8: Injective and Surjective Linear TransformationsMastery Concept 9: Rank Nullity TheoremMastery Concept 10: Matrix Spaces: Nullspace, Column Space, Nullity, RankMastery Concept 11: Least SquaresMastery Concept 12: Determinants

New Concepts

Concept 7: Matrix Representation of a Linear Transformation

• Know that every linear transformation can be represented in a matrix form.

• Process for finding the matrix form for a linear transformation (aka the change of basismatrix) for T : V → W

Step 1: Find the basis elements for the domain (V), codomain (W), and ran(T) (therange space of T)Step 2: Map the basis elements of V to the basis elements of ran(T)Step 3: Find the coordinate vectors or our ran(T) elements with respect to the basisfor the codomain (W). That is write our ran(T) basis elements as a linear combinationof the basis elements of the codomain, then list the coefficients into a column vectorStep 4: Construct your matrix representation by using the coordinate vectors as thecolumn vectors for M.

• You should know how to check whether your matrix representation does the same thingas the transformation. Do this by taking the a domain element and applying T to theelement. Write the image of this element as a coordinate vector of the codomain.check to see if this coordinate vector gives you the same thing as writing your domainelement as a coordinate vector and applying your matrix representation to it. Thencheck to see if your vector output matches the coordinate vector you got earlier.

273

Concept 8: Injective and Surjective Linear Transformations

1. Know the definition of an injective linear transformation: if T (u) = T (v) then u = v.

2. Know the other terms for injective maps: one to one, injection

3. Know how to determine if a function is injective. There are 2 Methods:Method 1: Using the definition of injective functions:

(a) Suppose two arbitrary values in the range are equal T (u) = T (v).

(b) Use the properties of the vector space and the definition of the function to showthat the input values are equal (u = v.)

Method 2: Use the Nullspace

(a) Find the Nullspace(T), if Null(T)={0}, T is injective.

4. Know the definition of a surjective linear transformation: For every value in thecodomain, there is a value in the domain that maps to it. That is if T : V → W ,then T is a surjection if for every w ∈ W , there exists a v ∈ V such that T (v) = w.

5. Know the other terms for surjective maps: onto, surjection

6. Know how to determine if a function is surjective. There are 2 Methods Method 1:Use the definition of a surjective map:

(a) Pick an arbitrary value in the codomain.

(b) See if there is an input value that will map to it.

(c) Note often if the map has some restrictions to its output values, the map is notsurjective -exploit these restrictions to show there is no input that can map tothis output.

Method 2: Use the range space

(a) if the rank(T)=dim(codomain), T is surjective

7. Know the definition of a bijection or isomorphism

8. Know how to create an isomorphism between vector spaces (key: Map basis elementsto basis elements.)

9. Know the definition of isomorphic vector spaces.

10. Nullspace and Range Space and Rank Nullity Theorem

(a) Know the definition of the nullspace of a linear transformation, T : V → W :null(T ) = {v ∈ V | T (v) = 0}.

(b) Know the definition of the nullity of a linear transformation, T : the dimension ofnull(T ).

274

(c) Know that null(T ) is a subspace of V.

(d) Know how to use the nullspace to determine whether or not a transformation isinjective

(e) Know how to find the nullspace of a transformation.

11. Range Space

(a) Know the definition of the nullspace of a linear transformation, T : V → W :ran(T ) ={T (v)| v ∈ V }.

(b) Know the definition of the rank of a linear transformation, T : the dimension ofran(T ).

(c) Know that ran(T) is a subspace of W.

(d) Know how to use the rank and range space to determine whether or not a trans-formation is surjective

(e) Know how to find the range space of a transformation.

Concept 9: The Rank Nullity Theorem

1. Know what the domain and codomain of a linear transformation

2. Know the Rank Nullity Theorem: Given a linear transformation T : V → W , dim(V)=Rank+Nullity

3. Know how to use this theorem to determine whether or not transformations can beinjective or surjective

4. Be able to construct injective and surjective maps if they exist.

Concept 10: Matrix Spaces: Nullspace and Range/Column Space

• Know the definition of the nullspace for an m×n matrix M, nulls(M) = {v ∈ Rn|Mv =0}

• If M is an m× n matrix, the Nullspace is a subspace of Rn

• Know the definition of the range space for an m×n matrix M, ran(M)={Mx ∈ Rm| x ∈Rn}

• If M is an m× n matrix, the range space is a subspace of Rm

• Know that the nullity of a matrix to be the dimension of the nullspace and the rankof a matrix to be the dimension of the range space. Furthermore the same ideas forinjective and surjective linear transformations work for Matrix Spaces.

• Know how to find the nullspace and range space for a matrix.

• Know that a shortcut for finding the range space or column space of a matrix: Findthe echelon form of the matrix and pick the original columns that correspond to thecolumns with leading ones.

275

• Know that the number of columns with leading 1’s of a matrix’s echelon form is thesame as the rank for that matrix.

• Know that the number of columns without leading 1’s of a matrix’s echelon form isthe same as the nullity for that matrix.

Concept 11: Least Squares

• Be able to find an approximate solution to an inconsistent solution by solving thenormal equation AT · Ax = Atb.

Concept 12: Determinants

• Know how to find the determinant of a 2 by 2 matrix: | a bc d

| = ad− bc

• Know how to efficiently do cofactor expansion to determine the determinant of a ma-trix: |A| = ∑n

i=1(−1)i+jai,j|Mi,j|,

For example: Let A =

a1,1 a1,2 a1,3 a1,4a2,1 a2,2 a2,3 a2,4a3,1 a3,2 a3,3 a3,4a4,1 a4,2 a4,3 a4,4

, then

|A| = a1,1

∣∣∣∣∣∣

a2,2 a2,3 a2,4a3,2 a3,3 a3,4a4,2 a4,3 a4,4

∣∣∣∣∣∣−a1,2

∣∣∣∣∣∣

a2,1 a2,3 a2,4a3,1 a3,3 a3,4a4,1 a4,3 a4,4

∣∣∣∣∣∣+a1,3

∣∣∣∣∣∣

a2,1 a2,2 a2,4a3,1 a3,2 a3,4a4,1 a4,2 a4,4

∣∣∣∣∣∣−a1,4

∣∣∣∣∣∣

a2,1 a2,2 a2,3a3,1 a3,2 a3,3a4,1 a4,2 a4,3

∣∣∣∣∣∣.

• Know that we can expand across any row or column when completing cofactor expan-sion.

Other Related Concepts;

1. Inverse Matrices

• Know when a Matrix will have an Inverse (AKA the Invertible Matrix Theorem:)Suppose A is a square n × n. The following statements are either all true or allfalse:

(a) A is invertible

(b) A has n pivot positions

(c) The Nullspace of A is trivial (The equation Ax = 0 has only the trivialsolution.)

(d) The columns of A form a linearly independent set.

(e) The columns of A span Rn

(f) The linear transformation represented by A is injective.

(g) The linear transformation represented by A is surjective.

(h) AX = b has a unique solution

(i) AT is an invertible matrix. 276

(j) There is an n× n matrix C such that CA = I

(k) There is an n× n matrix D such that AD = I

(l) detA 6= 0

2. Coordinate Vectors

• If we think of our basis elements as the building blocks for a vector space, ourcoordinate vectors represent the manual (or instructions) on how to put our basiselements together to get a particular vector.

• Given a basis for V , we are able to represent any vector v ∈ V as a coordinatevector in Rn, where n = dimV . Suppose B = {v1, v2, . . . , vn} is a basis for V ,then we find the coordinate vector [v]B by finding the scalars, αi, that make the

linear combination v = α1v1 +α2v2 + . . .+αnvn and we get [v]B =

α1

α2...

αn

∈ Rn.

Practice Problems:

1. For each map below, describe the range space and find the rank of the map.

(a) h : P3 → R2 given by

ax2 + bx+ c 7→(a+ ba+ c

)

(b) f : R2 → R3 given by(xy

)7→

0x− y

3y

2. Verify that this map is an bijection (isomorphism): h : R4 →M2×2 given by

abcd

7→

(c a+ db d

)

3. Find the determinant of the following matrices.

(a)

∣∣∣∣1 42 8

∣∣∣∣ 277

(b)

∣∣∣∣∣∣

2 1 11 1 06 4 1

∣∣∣∣∣∣

(c) det

1 0 −13 1 1−1 0 3

4. Find the coordinate vectors for the standard basis for R3 in terms of the basis: B =

{

123

,

111

,

01−1

}

5. Use the echelon version of A =

1 2 1 02 3 1 −17 11 4 −3

to determine the column rank, row

rank, rank, and nullity for A.

6. Assume that each matrix represents a map h : RmRn with respect to the standardbases. In each case, (i) state m and n (ii) find rangespace (h) and rank of h (iii) findnullspace of nullity, and (iv) state whether the map is onto and whether it is one-to-one.

(a)

[2 1−1 3

]

(b)

0 1 32 3 4−2 −1 2

(c)

1 12 13 1

7. If T is a linear transformation from V → W with rank= 4 and nullity= 2, what canwe say about the dimension of V? of W?

8. If T is a linear transformation from P4 → M2×3 can T be injective? If so, constructan example of such a possible linear transformation.

9. If T is a linear transformation from P4 →M2×3 can T be surjective? If so, constructan example of such a possible linear transformation.

10. If T is a linear transformation from P4 → M2×3 can T be bijective? If so, constructan example of such a possible linear transformation.

11. If T is an isomorphic linear transformation from V → W . What can we say about thedimensions of V and W?

12. Find the nullspace, nullity, range/column space and rank for the following matrices:278

(a) A =

1 0 −4 −3−2 1 13 50 1 5 −1

rref →

1 0 −4 −30 1 5 −10 0 0 0

(b) B =

1 −2 52 4 1−4 0 21 −2 03 1 1

rref →

1 0 00 1 00 0 10 0 00 0 0

13. Find a least squares solution of

−1 22 −3−1 3

x =

412

.

14. Find a least squares solution of

1 −2−1 20 32 5

x =

31−42

.

15. Consider the linear transformation T : R2 → P1 where T (

[ab

]) = bx − a Use these

bases for the spaces: basis for domain: BV = standard basis and basis for codomain:BW = {x, 1} . Represent this linear transformation (find the change of basis matrix:)

Check that this matrix does the same thing as our transformation for

[12

].

16. Consider the linear transformation T : P2 → P1 where T (ax2 + bx+ c) = 2ax+ b

Use these bases for the spaces. Use these bases for the spaces: basis for domain: BV =standard basis for P2 basis for codomain: BW = standard basis for P1 Represent thislinear transformation (find the change of basis matrix:) Check that this matrix doesthe same thing as our transformation for x2 − x+ 1.

More Practice Problems:Go to the online homework system to practice more problems.

279








Mastery Concept 1: Gaussian EliminationMastery Concept 2: Understanding solutions to systems of equationsMastery Concept 3: Vector Spaces and SubspacesMastery Concept 4: SpanMastery Concept 5: Basis/Linear IndependenceMastery Concept 6: Linear Transformations/FunctionsMastery Concept 7: Matrix representation for a linear transformationMastery Concept 8: Injective and Surjective Linear TransformationsMastery Concept 9: Rank Nullity TheoremMastery Concept 10: Matrix Spaces: Nullspace, Column Space, Nullity, RankMastery Concept 11: Least SquaresMastery Concept 12: DeterminantsMastery Concept 13: Eigenvalues and EigenvectorsMastery Concept 14: DiagonalizationMastery Concept 15: Markov ChainsMastery Concept 16: Grahm-Schmidt and Orthogonality

New ConceptsMastery Concept 13: Eigenvalues and Eigenvectors

• Know the definition of eigenvectors, eigenvalues, and eigenspaces:

• Know the how to find the eigenvalues of a matrix A: find the scalars λ so that det(A−λI) = 0.

• Know that the characteristic equation is det(A− λI) = 0

• Know the characteristic polynomial is det(A− λI).

• Know how to find the eigenvalues of a diagonal or triangular matrix (they are just thediagonal entries!)

• Know how to find an eigenvector for an eigenvalue, λ: Solve (A − λI)v = 0 aka: findthe nullspace of (A− λI)

• Know how to find an eigenspace associated with eigenvalue λ: Solve (A − λI)v = 0aka: find the nullspace of (A− λI). Write the space as a span.

280

Mastery Concept 14: Diagonalization

• Know the definition of an eigenbasis for Rn: a set of n linearly independent eigenvectors:v1, v2, . . . , vn

• Know how to determine whether or not a matrix is diagonalizable (and orthogonallydiagonalizable)

• Know that two matrices, A and B, are similar if there exists an invertible matrix Psuch that A = PBP−1

• Know the definition of a diagonalizable matrix: A matrix that is similar to a diagonalmatrix. In other words there exists diagonal D and invertible P such that A = PDP−1

• Know the Process to Diagonalize a Matrix, If Possible

1. Find the eigenvalues of the matrix.

2. Find linearly independent eigenvectors of the matrix.

3. Determine if you have enough eigenvectors from Step 2, that you can span Rn.Note you can only proceed if there are the same number of eigenvectors as thedimension of A (n) if there are not enough eigenvectors, the matrix is NOTdiagonalizable).

4. Construct P from the eigenvectors from Step 2.

5. Construct D from the corresponding eigenvalues (order matters -make sure theymatch the eigenvectors!).

6. Optional: Check to make sure P and D work by checking if A = PDP−1, but itis easier to just check if AP = PD.

• Know the definition of a symmetric matrix: AT = A

• Know that symmetric matrices can be orthogonally diagonalizable: This means A =PDP T where P is orthogonal.

• Know the process for orthogonally diagonalizing ATA or any symmetric matrix (Sameas diagonalization only normalize your eigenvectors.)

Mastery Concept 15: Markov Chains

• Know the definition of a Markov Chain/Process: A process in which the probabilityof the system being in a particular state at a given time period depends only on itsstate at previous time period.

• Know how to construct a transition for a Markov process and be able to use it to findfuture states.

• Know the properties of a transition matrix: All columns add up to 1 and every entryis between 0 and 1. 281

• Know what a steady state vector is for a Markov process: A vector such that Tv = vwhere T is the transition matrix.

• Know how to find a steady state vector: Solve (T − In)v = 0 (find a normalizedeigenvector for the eigenvalue 1.)

Mastery Concept 16: Gram Schmidt and Orthogonality

• Know the definition of Orthogonal matrices, a matrix in which all its columns areorthonormal.

• Know properties of orthogonal matrices like: A−1 = AT

• Know the definition of the projection of u onto v: projvu =u · v||v||2v

• Know the definition of an orthogonal set of matrices and how to check if two vectorsare orthogonal (check u · v = 0)

• Know how to determine the length of a vector: ||v|| = √v · v =√v21 + v22 + ...+ v2n

• Know the definition of an orthonormal set of vectors: a set which is orthogonal and allvectors have length 1.

• Know that a set of nonzero orthogonal vectors is a linearly independent.

• Know the definition of an orthogonal basis: A basis made up of orthogonal vectors

• Be able to use the Gram Schmidt process to create an orthogonal basis for a span ofvectors.

• Gram-Schmidt Process: Given a basis {b1, b2, ...bn} for a vector space V, we cancreate an orthogonal basis for V given by {v1, v2, ...vn} by the following process:v1 = b1

v2 = b2 −b2 · v1v1 · v1

v1

v3 = b3 −b3 · v1v1 · v1

v1 −b3 · v2v2 · v2

v2

v4 = b4 −b4 · v1v1 · v1

v1 −b4 · v2v2 · v2

v2 −b4 · v3v3 · v3

v3

...

vn = bn −bn · v1v1 · v1

v1 −bn · v2v2 · v2

v2 −bn · v3v3 · v3

v3 − · · · −bn · vn−1

vn−1 · vn−1

vn−1

• Know how to write a matrix into its QR Decomposition:

1. Find an orthonormal basis for the columns of A. (using Gram-Schmidt)282

2. Find R by computing R = QTA (works because Q is orthogonal so Q−1 = QT ).

Other Topics -not MasterySingular Value Decomposition

• Know some of the applications of SVD

• Know the difference between numerical rank and true rank

• Steps for Singular Value Decomposition:Step 1: Orthogonally Diagonalize ATA to find V (V will be our P when ATA = PDP T ).

Step 2: Find Σ using the singular values of A.

Step 3: Find U in 2 Steps

Step 3a: Find the first entries by ui =1

σiAvi

Step 3b: Add the normalized vectors from null(AT )

Column Rank and Left and Right Inverses

• Know the The Invertible Matrix Theorem: Restated Suppose A is a square n×n.Then the following are equivalent (if one is true, they all are true)

1. A is invertible


3. detA 6= 0.


5. A has full rank/ the columns of A are linearly independent

6. nul(A) = 0

• Know how to determine if a matrix has full column rank: Every column of the echelonform of the matrix has a leading one.

• Know how to determine if a matrix has full rank: Every column AND row of theechelon form of the matrix has a leading one.

• Know how to determine if a matrix has full column rank: Every column of the echelonform of the matrix has a leading one.

• Know if A has full column rank, (ATA)−1AT is a left inverse of A.

• Know if A has full row rank, AT (AAT )−1 is a right inverse of A.

• Know that while inverses of matrices are unique, but left and right inverses are notunique.

• Know that the set of least-square solutions of Ax = b coincides with the nonempty setof solutions to the normal equation: ATAx = ATb.

283

Practice Problems:

1. Orthogonally diagonalize A =

2 12 −11 00 1

if you can. Can you do this for ATA?

2. Find the singular value decomposition for

2 12 −11 00 1

3. Orthogonally diagonalize

[1 22 1

]

4. Find the singular value decomposition for

[1 22 1

]

5. Find an orthogonal basis for S=span{

1−22

,

1−40

,

50−4

}

6. Find the Q matrix in the QR Decomposition of

1 1 5−2 −4 02 0 −4

7. Find the eigenvalues and eigenspaces for each associated eigenvalue for the followingmatrices:

(a)

[1 20 0

]

(b)

[7 −84 −5

]

(c)

1 2 10 3 10 0 2

8. Diagonalize the following matrices if possible:

(a)

[1 20 0

]

(b)

[7 −84 −5

]

(c)

1 2 10 3 10 0 2

284

9. In a linear algebra class of 100 students, on any given day, some are in class and therest are absent. It is known that if a student is in class today, there is an 85% chancethat he/she will be in class tomorrow, and if the student is absent today, there is a60% chance that he/she will be absent tomorrow. Suppose today there are 76 kids inclass.

(a) Find the transition matrix for this scenario.

(b) Predict the number of students in class five days from now. And predict thepercentage of the class that will be absent.

(c) Find the steady state vector.

More Practice Problems:Go to the online homework system and complete the Study Plan or HW for more practiceproblems.

285

286

22 Homework

Please write your solutions on these homework pages and show enough of your work so thatI can follow your thought process. This makes it easier for me to grade. Also please staplethe homework together before you turn it in. Sometimes I have my stapler, but there is also astapler in my office and at the front desk of the Department of Computer and MathematicalSciences.

Follow the instructions for each question. If I can’t read your work or answer, you will receivelittle or no credit!

Since I have made some adjustments to the class, I do not have all of the homework posted,but will hand it out to you weekly. Your first Homework is on the next page.

287

.

288

Name:

MA 307 – HW 1: -Due Tuesday 1/23 at 3:30pm

Follow the instructions for each question and show enough of your work so that I can follow your

thought process. If I can’t read your work or answer, you will receive little or no credit! Please

submit this sheet with your answers. There are two sides! Show every step of the Gaussian

Elimination and DO NOT USE a computer system to solve them.

This is the only homework, in which you will need to do Gaussian Elimination by hand.

1. Solve the system of equations using Gaussian Elimination by hand. Make sure youcontinue with the process until your augmented system in RREF form (Row ReducedEchelon Fomr). Write the solution set. You must show your steps.

x − y − z = 42x − y + 3z = 2−x + y − 2z = −1

.

1

289

2. Solve the system of equations using Gaussian Elimination by hand. Make sure youcontinue with the process until your augmented system in RREF form (Row ReducedEchelon Fomr). Write the solution set. You must show your steps.

x − 2y − 3z = 24x + y − 2z = 85x − y − 5z = 10

.

2

290

3. Solve the system of equations using Gaussian Elimination by hand. Make sure you continue

with the process until your augmented system in RREF form (Row Reduced Echelon Fomr).

Write the solution set. You must show your steps.

x − 2y − 3z = 24x + y − 2z = 85x − y − 5z = 3

.

3

291

4. Eva and Archer, the Magnificent Math Kitties are wanting to find the coefficients a, b,&c so that the graph of f(x) = ax2 + bx + c passes through the points (1, 2), (−1, 6),&(2, 3). Eva says we can set up a system of equations to solve this problem, but Archersays that we can’t since we have a quadratic function and not a linear function? Whois right? If Eva is correct, set up and use Gaussian Elimination to solve this augmentedmatrix.

4

292

5. A system of equations is called homogeneous if the right-hand-sides of each equationis 0. So it looks something like this:a1x1 + b1x2 +c1x3 + d1x4 = 0a2x1 + b2x2 +c2x3 + d2x4 = 0a3x1 + b3x2 +c3x3 + d3x4 = 0a4x1 + b4x2 +c4x3 + d4x4 = 0

Eva looks at this and says she knows what will always be a solution to a homogeneoussystem of equations. What is it and why?

6. Archer says “A system with more unknowns than equations has at least one solution.”Eva isn’t so sure about this statement. Help her find a counterexample to disprove thestatement Archer made.

5

293

Bonus Question: From The Matrix: Neo: “What is the Matrix?”

a) Trinity: “The answer is out there,... it’s looking for you, and it will find you if you want

it to.”

b) Morpheus: “The Matrix is everywhere. It is all around us. Even now, in this very room...

It is the world that has been pulled over your eyes to blind you from the truth.”

c) Trinity: “The Matrix isn’t real.”

d) Morpheus: “The Matrix is a system, Neo. That system is our enemy.”

e) Morpheus: “Control. The Matrix is a computer-generated dream world built to keep us

under control in order to change a human being into this.”

f) Morpheus:“Unfortunately, no one can be told what the Matrix is. You have to see it for

yourself.”

g) None of the above, instead it is .... (fill in your answer below)

6

294

Math 307: Applied Linear Algebra Lecture Notesharsyram/AppliedLinearAlgebraWorkbook2018_newest.pdf\I...

Documents

Transcript of Math 307: Applied Linear Algebra Lecture Notesharsyram/AppliedLinearAlgebraWorkbook2018_newest.pdf\I...