1

CHAPTER 2. LINEAR SYSTEMS OF EQUATIONS Written by: Sophia Hassiotis

Revised on January 2013 The simplest model in applied mathematics is a system of linear equations such as:

2x1 + 4x2 =2 4x1 + 11x2 =1

A hand-calculation to obtain the two unknown parameters gives x1=3 and x2=-1. These are also the equations of two straight lines in the form x2=mx1+bo where m is the slope, and bo is the intercept at axis x2. Here the equations for the two lines are:

x2 =-0.5 x1+0.5 x2 =-0.3636 x1+0.0909

The plot of these lines is given here. Their intercept is point (3,-1).

Figure 2.1 Intersection of two lines >> x1=0:1:5; >> x2=-0.5*x1+0.5; >> x3=-0.3636*x1+0.0909; >> plot(x1,x2,x1,x3),title('two intercepting lines'),... >> xlabel('x1'),ylabel('x2') In matrix form these equations are written as:

=

12

11442

2

1

xx

OR A x = b

2

Given matrix A and vector b, We can solve for the unknown vector x using Matlab as: x=A\b (if b is a column vector) or x=b/A’ (if b is a row vector) Example: » A=[2 4;4 11]; If we input b as a row vector, the calculations are as follows: » b=[2 1]; » x=A\b’ (note that b’ is the transpose of b, therefore it is a column vector) Answer: x = 3.0000 -1.0000 or >> x=b/A' Answer: x = 3.0000 -1.0000 If we input b as a column vector, the calculations are as follows: >> b=[2;1]; >> x=A\b x = 3.0000 -1.0000 The coordinate (3,-1) is the intersection of the two lines and the solution for the linear system. Equations are called homogeneous if b=0; inhomogeneous if b not equal to 0. A nxn homogeneous system of linear equations has either 1) a unique solution (the trivial solution, x=0) if its determinant is non-zero or 2) an infinite number of solutions if its determinant is zero. Inhomogeneous equations may be consistent (have one or an infinity of solutions) or inconsistent (have no solution). The solution can be unique for a nonsingular set of equations or non-unique for a singular set of equations. For example, the equations shown above are two lines that intercept at a point. They represent an inhomogeneous, consistent system of equations. Since matrix A is nonsingular, the solution is unique.

3

Figure 2.2 Solutions of Inhomogeneous equations

A set of two equations will have infinity of solutions if they represent the same vector or will have no solution if they represent parallel vectors as seen in Figure 2.3(a). In another example let’s take an equation of a plane in the x1,x2,x3 space: ax1+bx2+cx3=d. We plot three such equations as plane S1, S2 and S3 in Figure 2.3b. The intersection of these planes is the solution of these three equations. The three planes can 1) intercept at a point (consistent solution, unique); 2) intercept at a line or represent the same plane (consistent, any of an infinity of solutions will solve the equations); or 3) they may never intercept at a common point, line, or plane (inconsistent, there is no solution).

Figure 2.3 Graphical representation of Inhomogeneous equations

4

Most linear systems have more unknowns than the two or three represented graphically in Figure 2.3. Before attempting to solve a linear system of equations you should first ask if the system has a solution, and if the solution is unique. We can check our equations by checking the rank of a matrix. The rank is the number of independent rows (or columns) in the matrix. If you view each row (or column) as a vector, the set of vectors is linearly independent if none of them can be expressed as a linear combination of any other. KEY THEOREM: Consider the inhomogeneous equations Ax=b where A is mxn. Exactly one of the following possibilities must hold:

(i) If the rank of the augmented matrix [A b] is greater than the rank of A, the system of equations is inconsistent (have no solution).

(ii) If the rank of [A b] is equal to the rank of A, this being equal to the number of unknowns, then the equations have a unique solution.

(iii) If the rank of [A b] is equal to the rank of A, this being less than the number of unknowns, then the equations have infinitely many solutions.

Next, we will investigate how Matlab handles these cases. First, we will investigate the existence of solutions for a square matrix A. Then we will discuss least-squares solutions in over-determined systems (more equations than unknowns) and then we will introduce unconstrained and constrained optimal solutions in the case of under-determined systems (more unknowns than equations).

5

1—Systems with as many equations as unknowns, m=n (square matrices)

Case i. Rank(a,b)>rank(a). System is inconsistent. A solution does not exist. » A=[1 -2 3;1 -2 3;-1 3 -7] A = 1 -2 3 1 -2 3 -1 3 -7 » b=[1 3 0] b = 1 3 0 » rank([A b']) ans = 3 » rank(A) ans = 2 » x=A\b' Warning: Matrix is singular to working precision. x = Inf Inf Inf » det(A) %Note singular A will have a determinate of zero ans = 0 Note that MATLAB does not give a solution to this set of equations since they are inconsistent. Case ii. Rank(a,b)=rank(a)=number of unknowns. A unique solution exists. » A=[1 -2 3;1 2 3;-1 3 1] A = 1 -2 3 1 2 3 -1 3 1 » b=[1 3 0] b = 1 3 0 » rank([A b']) ans =

6

3 » rank(A) ans = 3 » x=A\b' x = 1.6250 0.5000 0.1250 » det(A) ans = 16 » Case iii: rank(a,b)=rank(a)<number of unknowns. Infinity of solutions exist. » A=[1 -2 3;1 0 3;-1 3 -3] A = 1 -2 3 1 0 3 -1 3 -3 » b=[1 3 0] b = 1 3 0 » rank([A b']) ans = 2 » rank(A) ans = 2 » » x=A\b' Warning: Matrix is singular to working precision. x = Inf Inf Inf » det(A) %Note singular A will have a determinate of zero ans = 0 Note that Matlab does not give a solution to these equations although they are consistent and have an infinity of solutions.

7

2—Least Squares: Systems with more equations than unknowns, m>n

Example 1: Minimize the norm || Ax-b ||2 A system with more equations than unknowns (A has more rows than columns) is called over-determined. As an example, an over-determined set of equations results when attempting to fit the data from a laboratory experiment. Let’s first fit the data of Table 2.1 with a straight line of the form z=my+bo where m is the slope and bo is the z-intercept.

Table 2.1 Data to fit y z 1 1 3 2 4 3 5 3

If each point were exactly part of the line, then each of the following equations would be satisfied exactly. z=m(y)+bo

1=m(1)+bo 2=m(3)+bo 3=m(4)+bo 3=m(5)+bo

or, in the matrix form Ax=b

=

3321

15141311

obm

Solving in Matlab for the unknown vector x (note, x1=m, x2=bo) >> A=[1 1;3 1;4 1;5 1]; b=[1 2 3 3]; >> x=A\b' x = 0.5429 0.4857 This gives an equation for the line as: z=0.5429 y + 0.4857 In such cases as the one above, it is impossible to solve Ax=b exactly. The best we can do is to make Ax not equal but as-close-as-possible to b by minimizing the norm (magnitude) of a vector that represents the error Ax-b. Namely, we minimize the

8

quadratic || Ax-b ||2, where the symbol || v ||2 represents the square of the magnitude of vector v and it is equal to vTv. The solution for x which minimizes the error is the least-squares solution to Ax=b. We can express this mathematically as:

Find vector x that minimizes the norm || Ax-b ||2 Expanding the norm as

(Ax-b)T(Ax-b)=xTATAx-2ATbx +b2 and setting the derivative with respect to x to zero we obtain:

2ATAx - 2ATb = 0. From here we find

ATAx= ATb and

x=(ATA)-1 ATb The matrix inv[(A’A)]*A’is called the pseudoinverse. In Matlab we can solve the same over-determined systems by using the definition for pseudoinverse: >> A=[1 1;3 1;4 1;5 1]; b=[1 2 3 3]; >> x=inv(A'*A)*A'*b' x = 0.5429 0.4857 or by using the pseudo-inverse operator pinv >> x=pinv(A)*b' x = 0.5429 0.4857 For a non-square mxn matrix, if rank(A) = min(m, n), then A is full rank . If rank(A)<min (m,n), then A is rank deficient and does not have unique solutions. In the example above, rank(A)=2 which is equal to the number of columns, n. If A were rank deficient, then we would not find a unique solution using the different methods introduced above. Let’s use the same method to fit data to a quadratic. Assume that the data of Table 2.1 is best fitted by a quadratic of the form: z=cy2+dy+e. In this case, if each point were exactly part of the quadratic curve, each of the following equations would be satisfied exactly:

1=c(1)2+d(1)+e

9

2=c(3)2+d(3)+e 3=c(4)2+d(4)+e 3=c(5)2+d(5)+e

or, in the matrix form Ax=b

Solving in Matlab for the unknown x >> A=[1 1 1;9 3 1;16 4 1;25 5 1];b=[1 2 3 3]; >> x=A\b' x = -0.0455 0.8091 0.2000 This gives an equation for the quadratic curve as: z=-0.0455y2 + 0.8091y+0.2 Matlab has its own solvers to fit data with different polynomials. For example, try the statement “polyfit(y,z,n)” which fits y-z data with a polynomial of degree n such that p(y)=p1yn+p2yn-1+…+pny+c >> y=[1 3 4 5];z=[1 2 3 3]; >> p1=polyfit(y,z,1) ans = 0.5429 0.4857 >> p2=polyfit(y,z,2) ans = -0.0455 0.8091 0.2000 To see how well the polynomials fit the data, evaluate each polynomial at the data points and plot the results. >> f1=polyval(p1,y); >> f2=polyval(p2,y); >> plot(y,z,'o',y,f1,'-',y,f2,'-.'),xlabel('y'),ylabel('z') Note that these examples are used here merely to illustrate the theory behind least squares. MATLAB has many functions that can be used to fit data that are more suited to such a task.

1 1 1 9 3 1

1 6 4 1 2 5 5 1

c

d e

1 2

3 3

10

Figure 2.4 Fit of data with a straight line and with a polynomial curve

11

3—Unconstrained and Constrained Optimization: Systems with fewer equations than unknowns, m<n

A system that has more unknowns than equations is an under-determined system and it is solved as an optimization problem. In these cases a unique solution does not exist. We can find one by introducing an objective function and constraints. Example 1: Unconstrained Optimization In this example, solve the set of equations for the four unknown values of x

5x1 + 2x2 +3x3 +4x4 =1

-5x1 + 3x2 +2x3 +7x4 =2 Infinity of solutions exist. We can find unique solutions if we introduce an objective function. For example, of all possible solutions x, we could be seeking the one with the smallest magnitude. This is an example of an unconstrained optimization problem that can mathematically be stated as:

Find x that minimize the norm of the solution: ||x||2 In Matlab we solve with the \ operator » A=[5 2 3 4; -5 3 2 7]; » b=[1 2]; » x=A\b’ x = -0.0182 0 0 0.2727 we can also try solving with the pseudoinverse » y=pinv(A)*b y = -0.0402 0.0879 0.0798 0.1965 let’s calculate the norms: » norm(x) ans = 0.2733 » norm(y) ans = 0.2331

12

The first solution solves the problem. The pseudo-inverse solves the problem also. In addition, the pseudo-inverse minimizes the norm of the vector ||x||2. Example 2: Minimize the norm ||Ax-b|| subject to x>=0 --Constrained Optimization This problem arises when we have more unknowns than equations (under-determined) but we need a solution within constraints. For example, solve the following equations, but make sure that the solution is always positive. Solve:

5x1 + 2x2 +3x3 +4x4 =1 -5x1 + 3x2 +2x3 +7x4 =2

subject to: xi>=0. In Matlab we use the routine lsqnonneg to solve this problem. It finds the least-squares solution subject to the constraint that all solutions must be nonnegative. » A=[5 2 3 4; -5 3 2 7]; » b=[1 2]’; » x=lsqnonneg(A,b) %[used to be nnls(a,b)(non-negative least squares)] x = 0 0 0 0.2769 Matlab has an optimization toolbox that provides routines that solve different optimization problems. We will investigate two more routines, fmincon and fminsearch.

13

Example 3: Optimal design of a rectangular beam using Matlab Use optimization to design a beam for bending and shear. In this example, the bending moment M, and the shear force V are given. The objective is to choose the lightest rectangular section that will carry the load and not surpass the strength of the material.

Remember that the bending and shear stresses are calculated by:I

My=σ , and

ItVQ

=τ .

For a rectangular section of dimensions w x h , we can rework these expressions as

2/hy = ,12

3whI = , and I

Vh8

2

=τ .

Let’s assume that the strength of the material in tension is 30ksi and in shear it is 5ksi. Also, let us choose from sections with width w between 2 and 10 inches and height h between 6 and 20 in. In addition, let us restrict the aspect ratio of h/w between 1.5 and 2.5. In essence, we are solving the optimization problem: Minimize w * h Subject to: My/I <30 ksi Vh^2/8I <5 ksi 2< w<10 6<h<20 1.5<h/w<2.5 We are dealing with a multivariable minimization problem (both, w and h are variables) with nonlinear inequality constraints. The Matlab routine fmincon seems appropriate to solve this problem. First re-write the constraints in the form c(x) ≤ 0:

My/I <30 becomes: 6*M/(x(1)*x(2)^2)-30 Vh^2/8I <5 becomes: 12*V/(8*x(1)*x(2))-5 2< w<10 becomes: x(1)-10 -x(1)+2 6<h<20 becomes: x(2)-20 -x(2)+6 1.5<h/w<2.5 becomes: x(2)/x(1)-2.5 -x(2)/x(1)+1.5]

Then follow the appropriate steps as directed by Matlab to create your subroutine.

14

Step 1: Write an M-file objfun.m for the objective function. function f=objfun(x) %function used for optimal design of beam section. Here,we are minimizing the function f=w*h. Assign x(1)=w and x(2)=h f=x(1)*x(2);

Step 2: Write an M-file constraints.m for the constraints. function [c,ceq]= constraints(x) %inequality constraints for beam Design Optimization M=1200;V=300; c=[6*M/(x(1)*x(2)^2)-30;12*V/(8*x(1)*x(2))-5;x(1)-10;-x(1)+2;x(2)-20;… -x(2)+6;x(2)/x(1)-2.5;-x(2)/x(1)+1.5]; %equality constraints ceq=[];

Step 3: Invoke constrained optimization routine. %beam optimization main program M=1200;V=300;x(1)=6;x(2)=6; x0=[6 6];%starting guess options=optimset('Algorithm','active-set'); [x,fval]=fmincon(@objfun,x0,[],[],[],[],[],[],@constraints,options)

The solution x produced with function value fval is

x = 7.3181 12.2983 fval = 90.0000 The optimal solution is given by w=7.3 and h=12.298. The product w x h is 90.0

15

Example 4: Minimize the energy of a system In most linear systems, the energy of the system can be expressed as a quadratic. Equilibrium is defined as the position vector x that minimizes the energy of the system. In other words, if x is a vector of the system displacements, the solution for x is the equilibrium position. We are thus looking for the x that minimizes the energy E(x)

E(x) = bxAxx TT −21

If A is positive definite, then the quadratic is minimized at the point x that satisfies Ax=b. To minimize the energy E(x), take the derivative with respect to x and set it to zero. Namely, dE/dx= 0. When we perform this we get Ax-b = 0. As an example, let’s assume that there exists a system whose total energy E(x) is written as:

[ ] [ ]

−

−

−=

87

2121

3335

21)( 21 xx

xx

xxxE

To minimize E(x) set dE/dx1=0 and dE/dx2=0. Please perform these operations and show that we recover the set of equations Ax=b. The vector x that solves this set of equations is the one that minimizes the energy of the system. A is a square positive definite matrix. We can use Matlab to solve for x using x=A\b’. Here, x = 7.5000 10.1667 Lets Plot the quadratic with Matlab %this plots a 3d plot of a quadratic; energy of two springs xa=-10:0.2:20; xb=-10:0.2:20; k1=2; k2=3; [x1,x2]=meshgrid(xa,xb); E=0.5*k1*(x1.^2)+0.5*k2*(x2-x1).^2-7*x1-8*x2; meshc(x1,x2,E),xlabel('x1'),ylabel('x2'),zlabel('E') pause contour(x1,x2,E,100),xlabel('x1'),ylabel('x2'),zlabel('E') pause %plot Ax=b with the countour plot A=[k1+k2 -k2;-k2 k2];b=[7 8]; x=A\b'%solving the equations x4=7/5+3/5*xa;%creating data to plot the lines x5=-8/3+xa; plot(x4,xa,x5,xb),xlabel('x1'),ylabel('x2')

16

Notice that the energy is at a minimum at an (x1,x2) of (7.5,10.2).

17

This is a graph of the two lines in Ax=b. Notice the intersection points at ( 7.5,10.2). Instead of solving the equations either algebraically or graphically, we can allow an optimization algorithm to find the minimum of the energy. For this problem we will use the routine fminsearch which finds the minimum of a scalar function of several variables, starting at an initial estimate. This is an unconstrained nonlinear optimization. To use the optimization tools in Matlab, create a function to be minimized. function E=energyoptimization(v) %to used in an optimization search towards the minimization of energy %quadratic k1=2;k2=3; x1=v(1); x2=v(2); E=(0.5*k1*(x1.^2)+0.5*k2*(x2-x1).^2-7*x1-8*x2); Then, use the search algorithm fminsearch to find the minimum. Here, we start the search at the point (x1,x2)=(5,5). >> v=[5 5]; >> a=fminsearch(@energyoptimization,v) a = 7.5000 10.1666

-15 -10 -5 0 5 10 15 20-10

-5

0

5

10

15

20

x1

x2

18

The minimum is found at (7.5,10.2). We have shown that we can obtain the solution (vector x) by 1) finding the minimum of the energy (Principle of Minimization of Energy) either graphically or using optimization algorithms, or 2) by solving the equilibrium equations (Newtonian Mechanics). A geometric interpretation of the system that we just solved is given below. Take two masses suspended on two springs. Let’s find the equilibrium position of the masses by either, minimizing the energy of the system or by satisfying the equations of equilibrium.

1--Minimize the Energy of the System Total Energy E = Strain Energy in the spring (U) – Potential Energy of the Mass (P) where,

U=½ k1 x1^2 + ½ k2 (x2-x1)^2 P = 7x1 + 8x2

The energy can then be written as

E=½ k1 x1^2 + ½ k2 (x2-x1)^2 -7x1 -8x2

To find the position x that minimizes the energy of the system, take the derivative of E with respect to x and solve for x. ∂E/∂x1=0: k1 x1 – k2 x2 + k2 x1 – 7 = 0

∂E/∂x2=0: k2 x2 – k2 x1– 8 = 0 These two equations can be placed in a matrix form as:

19

A x = b 2--Satisfy the equations of equilibrium, SFx=0: Using the free-body diagram, we can write the equations of equilibrium of the two masses as:

-k1 x1 + k2 (x2 – x1) + 7 = 0 -k2 ( x2 – x1) + 8 = 0

These equations can also be presented in matrix form as: A x = b Therefore, minimizing the total energy of the system or solving Newton’s equation of equilibrium results in the same solution for the displacement of the masses (the unique displacement vector x that places the system in equilibrium)

20

Homework 3: Label each set of equations as homogenous/inhomogenous, consistent/inconsistent and solve using MATLAB

1) x –2y+3z=10 -x-2y+3z=20 x+3y-7z =30

2) x –2y+3z=1

x+2y+3z=1 -x+3y+z =1

3) x –2y+3z=0 x +3z=4 -x+3y-3z =0

Solve the equations.

1) 3 x + 5 y – 3 z + 5 =0 – x + 3 y - 10=0 7 x – 7 y – 7 z + 10=0

5 x – 8 y + z + 3 =0 x + 2 y + 5 z =0 2) 2x +5y-3z=10

5 x +y +5z=-5 3) 2x +5y-3z=10

5 x +y -5z=20 where x,y,z > 0

21

Homework 4:

Fit the following data with 4 different polynomials. Plot your results. Can you fit a higher-order polynomial? Would you need to? What is a possible material with this stress-strain curve? Write down the simplest form of an equation that describes your data.

Strain Stress (psi) 0 0 0.0002 950 0.0004 1,900 0.0006 2,300 0.0008 2,500 0.0010 3,200 0.0012 3,300 0.0014 3,500 0.0016 3,700 0.0018 3,850 0.0020 4,000 0.0022 3,800 0.0024 3,600 0.0026 3,500 0.0028 3,400 0.0030 3,300 0.0032 3,200 0.0034 2,800 0.0036 2,500

22

Homework 5: Optimal Design of a Truss

Write an optimization algorithm in Matlab to design for the minimum area of each member of the truss given below. Use sections with a circular cross-sectional area. Use an allowable stress σall of 36ksi and a Young’s Modulus E of 29,000ksi. Note that an axial force F may cause yielding. The stress due to an axial force is F/A where A is the area of the truss member. A compression loading may cause buckling. The stress due to bucking is σ=Fcr/A where Fcr=(π2/l2)EI in which l is the length of the truss member and I is the moment of inertia of the round section. You should check your allowable stress against both, 1) yielding and 2) buckling to design for the area of each truss member.

23

Appendix A

Matlab Examples Using Unconstrained and Constrained Minimization. These examples were taken directly from Matlab/Help/Optimization/Introduction

MATLAB Example(1): fminunc--- Unconstrained Minimization

Consider the problem of finding a set of values [x1, x2] that solves

(4-16)

To solve this two-dimensional problem, write an M-file that returns the function value. Then, invoke the unconstrained minimization routine fminunc.

Step 1: Write an M-file objfun.m. function f = objfun(x) f = exp(x(1))*(4*x(1)^2+2*x(2)^2+4*x(1)*x(2)+2*x(2)+1);

Step 2: Invoke one of the unconstrained optimization routines. x0 = [-1,1]; % Starting guess options = optimset('LargeScale','off'); [x,fval] = fminunc(@objfun,x0,options) This will produce the following output x = 0.5000 -1.0000 fval = 3.6609e-015

MATLAB Example(2): fmincon---Nonlinear Inequality Constraints

If inequality constraints are added to Equation 4-16, the resulting problem can be solved by the fmincon function. For example, find x that solves

(4-57)

subject to the constraints

x1x2 – x1 – x2 ≤ –1.5, x1x2 ≥ –10.

Because neither of the constraints is linear, you cannot pass the constraints to fmincon at the command line. Instead you can create a second M-file, confun.m, that returns the value at both

24

constraints at the current x in a vector c. The constrained optimizer, fmincon, is then invoked. Because fmincon expects the constraints to be written in the form c(x) ≤ 0, you must rewrite your constraints in the form

x1x2 – x1 – x2 + 1.5 ≤ 0, –x1x2 –10 ≤ 0.

(4-58)

Step 1: Write an M-file objfun.m for the objective function. function f = objfun(x) f = exp(x(1))*(4*x(1)^2 + 2*x(2)^2 + 4*x(1)*x(2) + 2*x(2) + 1);

Step 2: Write an M-file confun.m for the constraints. function [c, ceq] = confun(x) % Nonlinear inequality constraints c = [1.5 + x(1)*x(2) - x(1) - x(2); -x(1)*x(2) - 10]; % Nonlinear equality constraints ceq = [];

Step 3: Invoke constrained optimization routine. x0 = [-1,1]; % Make a starting guess at the solution options = optimset('Algorithm','active-set'); [x,fval] = ... fmincon(@objfun,x0,[],[],[],[],[],[],@confun,options)

After 38 function calls, the solution x produced with function value fval is

x = -9.5474 1.0474 fval = 0.0236

25

Appendix B Using the Curve Fitting Tool Find relevant information at the MATLAB HELP, data fitting (Interactive Curve Fitting) >> x=[1 2 3 4 5 6 7 8 9 10]; >> y=[5 8 6 0 2 6 8 1 5 3]; >> plot(x,y)

MATLAB allows you to fit 1) a polynomial, 2) exponential, 3) Fourier, 4) Gausian, 5) Power, 6) Sum of Sine Functions and 7) Weibull. It also allows you to input your own function.

1 2 3 4 5 6 7 8 9 100

1

2

3

4

5

6

7

8

26

Example 1. Fit a Polynomial >> cftool %opens the Curve Fitting Tool Choose Data Choose the x and y data vectors

Create data set and Close Choose Fitting and then, New Fit, Type of fit polynomial, 5th Degree Polynomial, Apply

27

28

Example 2. Fit Sum of Sin Functions Choose Fitting and then, New Fit, Type of fit polynomial, 5th Degree Polynomial, Apply

In-class work Fit the data e=[0 0.01 0.02 0.03 0.04 0.05] s=[0 500 900 1200 950 600 ]