ChE/MatE 166 Advanced Thin Films San Jose State University

Lab 1 Handout - 1

LAB 4: Experiment with a Single Factor: Analysis of Variance (ANOVA)

Learning Objectives 1. Write clear objectives and statement of problem for an experiment. 2. Identify controllable and uncontrollable factors in an experimental set-up. 3. Choose a factor for a single factor ANOVA based on expected outcome and given

time and equipment constraints. 4. Determine the appropriate levels to be researched for a factor based on expected

outcome, equipment control, metrology precision, and time constraints. 5. Design an experiment using proper replication, randomization, and control of

variables. 6. Calculate the variation between levels using a sum of squares method. 7. Calculate the variation using an F-test. 8. Plot data of all the levels to show variation between levels. 9. Organize technical information into a clear and concise formal laboratory report.

Equipment Each team will be using ANOVA techniques to determine the effect of a single variable on the process. The group tasks are:

Team 1: Oxidation Team 2: Wet or Dry Etching of Oxide Team 3: Wet Etching of Al Team 4: Deposition of Al

The equipment and metrology tools are the same as those discussed in Labs 2 and 3. See those lab handouts for more details.

Design of a Single-Factor Experiment In this lab module, you will be designing an experiment using a simple, single factor approach. One way ANOVA (analysis of variance) is a method used to compare two or more data sets to determine if the means are the same or different. This will allow you to compare the data collected when you vary your factor to determine if the factor has a statistically significant impact. Note: The design steps and golf analogy used below are taken from D.C. Montgomery, Design and Analysis of Experiments, John Wiley & Sons, (1997).

1. Statement of Problem The first step in designing an experiment is to clearly state what the problem is or what you are trying to investigate. The statement of your problem is your overall goal for performing the experiment. Some typical goals for experimenting include improving process yield, reducing variability or obtaining closer precision to targeted goals, reducing time or cost, or improving the product.

ChE/MatE 166 Advanced Thin Films San Jose State University

Lab 1 Handout - 2

Example: My golf score varies wildly from game to game. I want to determine what factors affect my golf score and quantify the influence each factor has on the overall score.

2. Determination of Variables In order to design a controlled experiment, all the factors that can influence the overall outcome must be assessed. Variables can be classed into two groups: controllable variables (ones you can change) and uncontrollable variables (ones that you can not alter). Example: I developed a list of variables that influence my golf score (or may influence my golf score) and classified them as ones I can control and ones I can't:

weather: uncontrollable riding a cart or walking: controllable type of clubs used: controllable type of ball used: controllable difficulty of the course: uncontrollable number of players assigned to play with me: uncontrollable ability level of other players: uncontrollable type of shoes: controllable type of beverage: controllable time of day: controllable amount of time I warm-up at the driving range: controllable

3. Choosing a Factor to Investigate You need to choose variables and controls that are within the limits of your experimental framework (can give valuable data with the time, supplies, and cost you have allotted). There are several different kinds of experiments you can design. A common experiment is one factor at a time where the engineer varies one factor and holds all the rest constant. (However, the problem with this is that it does not investigate the fact that factors may be inter-related. In future labs, we will design full factorial experiments to investigate the influence of multiple factors at once.) When choosing a factor to investigate, you need to consider a number of things. The first is that you want to choose a factor that previous experience or engineering knowledge tells you is significant. You need to determine how the influence of the factor will be quantified (what metrology and statistics you will use) to ensure that the influence is measurable. One-way ANOVA (analysis of variance) will investigate the influence of the factor by performing the experiment at different levels (such high, medium, and low) and statistically comparing the data. The value of the levels needs to be controllable. The levels must be chosen to have a significant, quantifiable, and measurably different effect on the end result. You need to choose a range of levels that is broad enough to investigate the full impact of the variable.

ChE/MatE 166 Advanced Thin Films San Jose State University

Lab 1 Handout - 3

Example: I believe that the amount of time I warm up at the driving range will have a significant impact on my game. My hypothesis is that if I do not warm up at all or only for a brief time (less than 15 minutes), I will be stiff and my score will be poor. However, if I warm up too much (over 40 minutes), I will be tired and my game score will also suffer. I need to choose levels of warming up to test this hypothesis that are significantly different enough. The levels I will test are warming up for 0, 10, 30, and 50 minutes. The levels are different enough to be controllable (I can accurately time 10 minutes versus 30 minutes. Investigating the influence of 10 minutes versus 12 minutes would not be controllable. This is because when a warm up starts and ends is vague enough that a 2-minute difference is not significant.) Note that in this experiment, I am choosing a broad range of levels to investigate a maximum- minimum type of influence: not warming up enough and warming up too much.

4. Control of Other Variables In this experiment, you are only trying to investigate the influence of one factor. The other controllable variables should be set to values that you have a tight control over and are within normal range for a typical process. Example: I want to wear my normal shoes and use my normal ball and clubs. Ideally, these would all be done at the same time of day and under the same weather conditions.

5. Randomization & Replication Once you run your experiments, you are going to make conclusions on the influence the factors had. (In the golf example, I would compare golf scores to see if how the factors caused the scores to vary.) To do this, you must also factor out the fact that the data comes from two (or more) different experiments. In other words, you want to be able to say the variation is due to those variables not just the fact that there is statistical variation from run to run. You also need to guarantee that some other, uncontrollable variable wasn’t influencing your results. To do this, your experiments need to contain both replication and randomization. Replication involves repeating the same run more than once such as playing a golf game using the same warm up regiment twice. You could compare the scores of the replicates to determine the run-to-run variation. The more replicates you have, the more statistical confidence you will have in your data. However, there is a limit (based on time, money, and supplies) of how many runs you can do. Example: I have determined that my resources (time and money) allow me to play eight rounds of golf. Therefore, I can have 2 replicates at every data point. It is also important to randomize the order in which you run the experiments. This will average out the influence of any other factors that may contributing that you can't control. Example: Some factors that may influence my golf game but I can't control are the weather and my assigned golf partners. Also, my score may improve just because I am playing more games. To average out these factors, I perform the eight games with varying levels in a random order.

ChE/MatE 166 Advanced Thin Films San Jose State University

Lab 1 Handout - 4

6. Monitoring the Experiment & Quantifying Results For the golf game example, it is very easy to quantify the results. I total up my golf score for the entire game. When I am collecting my data, I want to be careful to note as many other factors as possible (weather, partners, physical condition) in case I need to refer to that information when discussing my data.

7. Analyzing Results Analysis of variance (ANOVA) is a statistical technique that can be used to quantify the significant differences between levels. An analysis of variance is done by comparing the variance within a level against the variance across the whole population. The sum of squares is used (this is the sum of comparing each replicates value with the average value). The square term is used to factor out whether the sample is above or below the mean. That is, if the square wasn’t taken variations below the mean may cancel variations above the mean to make it seem as if there is no variation.

ANOVA Figure 1 shows the plot of three samples of data taken from three populations. It cannot be determined whether the means are statistically the same or not by plotting the data. ANOVA needs to be done to ascertain whether the means are same or not. Figure 2 details the steps needed to determine if the means of the three samples are statistically different. Table 1 gives the factors calculated in ANOVA. Table 2 shows the raw data labeled as it is used in an ANOVA calculation. Is there any difference in mean? +---------+---------+---------+---------+ 9.0 10.0 = Sample from Level (treatment) 1

1x 2x 3x = Sample from Level (treatment) 2

= Sample from Level (treatment) 3 Figure 1: Data from three samples where, 1x = Mean of sample 1; 2x = Mean of sample ;

3x = Mean of sample 3.

ChE/MatE 166 Advanced Thin Films San Jose State University

Lab 1 Handout - 5

Population 1 Population 2 Population 3 … Sample 1 Sample 2 Sample 3 ( )nxxxx ,...,, 321 ( )nxxxx ,...,, 321 ( )nxxxx ,...,, 321 _

Obtain each mean and standard deviation (x, s) Perform F Test Confidence intervals Interpreting result Figure 2: Steps needed to determine if the means of the three samples are statistically different. Table 1: Summary of ANOVA calculation.

Hypothesis nH µµµµµ ===== ....: 43210

1H : At least one µ is different from the others

Where, iµ = the average of population i F test

E

Treatments

MSMS

F =0 (See Tables 2 and 3 for the detailed calculation.)

Where, TreatmentsMS = the sum squares due to differences in the treatment means

EMS = the overall variation due to random error Confidence Interval on thi treatment

≤≤− −• iE

aNi nMS

ty µα ,2 nMS

ty EaNi −• + ,2α

Where, •iy = the average in treatment i

=N total observations , =a number of treatments (levels) n = number of observations in a treatment

Reject 0H aNaFF −−> ,1,0 α

Where d.f = ( )aNa −− ,1 Table 2: Raw data of ANOVA. Treatment

(level) Data Total Averages

1 11y 12y … ny1 •1y •1y

2 21y 22y … ny2 •2y •2y

M M M … M M M a 1ay 2ay … any •ay •ay ••y ••y

ChE/MatE 166 Advanced Thin Films San Jose State University

Lab 1 Handout - 6

ANOVA requires mathematical manipulation of all the values in Table 2. Table 3 details the calculations needed in ANOVA. Table 3: The analysis of variance table for single- factor. Source of Variation

Sum of Square Degree of

Freedom

Mean Square 0F

Within treatment =TreatmentsSS ( )

2

1∑

=••• −

a

ii yyn

1−a TreatmentsMS =

1−aSSTreatments E

Treatments

MSMS

F =0

Error (within treatments)

TreatmentsTE SSSSSS −= aN − EMS =

aNSSE

−

Total ( )

2

1 1∑∑

= =••−=

a

i

n

jijT yySS

1−N

The sum of squares for levels is the sum squares due to differences in the treatment means

=TreatmentsSS ( )2

1∑

=••• −

a

ii yyn

Where, •iy is the sum of all the values in Level (treatment) i: ∑=

• =n

jiji yy

1

•iy is the average of all the values in Level (treatment) i: ny

y ii

•• =

••y is the average of all the samples (all levels) : Ny

y •••• =

n is the number of replicates in that level a is the number of levels: Nna =⋅ The sum of squares for the total population is given as:

( )2

1 1∑∑

= =••−=

a

i

n

jijT yySS

where SStotal is the sum of square for the entire sample population N is the total number of samples

The sum of squares errors is the overall variation due to random error

TreatmentsTE SSSSSS −= An F-test is used to compare the sum of squares of the levels with the sum of squares of the random errors.

ChE/MatE 166 Advanced Thin Films San Jose State University

Lab 1 Handout - 7

E

Treatments

MSMS

F =0

A criteria is used that says that if the calculated Fo is less than a critical value than the levels are not statistically different. A confidence level needs to be set, typically an α=0.05 confidence level is chosen. This signifies that the criterion is (1- α) or 0.95 accurate. If Fo is greater than Fcritical = aNaF −− ,1,α (Rejection Region), the levels are significantly different. The Fcritical value can be found using a chart that is specific to the confidence level chosen. (See Lab 2 Handout, Table 3 for Fcritical for α=0.05.) To use the table you need the degrees of freedom of MSlevel and MSerror that are a-1 and N-a respectively.

Example An experiment was run to investigate the influence of DC bias voltage on the amount silicon dioxide etched from a wafer in a plasma etch process. Three different levels of DC bias were being studied and four replicates were run in random order, resulting in the data in Table 4. Table 4: Amount of silicon dioxide etched as a function of DC bias voltage in a plasma etcher.

DC Bias (Volts)

1

2

3

4

Total

Average

s398 283.5 236 231.5 228 979 244.75 485 329 330 336 384.5 1379.5 344.875 571 474 477.5 470 474.5 1896 474

4254.5 354.54 Hypothesis

3210 : µµµ ==H

1H : At least one µ is different from the others Sum of Squares for Levels

=TreatmentsSS ( )2

1∑

=••• −

a

ii yyn

= ( ) ( ) ( )[ ]222 54.35447454.35487.34454.35475.2444 −+−+− = 105672 Degree of freedom = 1−a = 3 – 1= 2 Sum of Squares for the Total Population

TotalSS = ( )2

1 1∑∑

= =••−=

a

i

n

jijT yySS

= ( ) ( ) ( ) ( )[ ]2222 54.3545.47454.3545.23154.35423654.3545.238 −++−+−+− L

Amount Etched (in Angstroms)

ChE/MatE 166 Advanced Thin Films San Jose State University

Lab 1 Handout - 8

= 109857 Degree of freedom= 1−N = 12 – 1 = 11 Sum of Squares Errors

=ErrorSS TreatmentsTotal SSSS − = 109857-105672 = 4185 Degree of Freedom = aN − = 12 - 3 = 9 F-Statistic

=TreatmentsMS1−a

SSTreatments = 528362

105672=

EMS =aN

SSE

−=

94185 = 465

E

Treatments

MSMS

F =0 = =465

52836 113.63

Where, critialF aNaF −−= ,1,α 9,2,05.0F = 4.26 (determined from Lab 2, Table 3). The summary of the ANOVA calculation for the example is given in Table 5. Table 5: Summary of the ANOVA calculation for the plasma etcher example.

Source of Variation

Sum of Square

Degree of Freedom

Mean Square 0F

Treatments (Levels)

105672 2 52836 113.63

Error 4185 9 465 Total 109857 11 Because the 26.463.1130 >=F , we reject the null hypothesis ( 0H ). We can conclude that the mean of the oxide thickness etched at the different DC bias voltages are different. Confidence Interval With 95 % confidence, the true mean of each treatment is within:

aNt −,2α =9,025.0t 2.262

≤≤− −• iE

aNi nMS

ty µα ,2 nMS

ty EaNi −• + ,2α

4465262.275.244

4465262.275.244 11 +≤≤−= µµ

= 220.36 Angstroms 14.2691 ≤≤ µ Angstroms

ChE/MatE 166 Advanced Thin Films San Jose State University

Lab 1 Handout - 9

4465262.287.344

4465262.287.344 22 +≤≤−= µµ

= 320.48 Angstroms 26.3692 ≤≤ µ Angstroms

4465262.2474

4465262.2474 33 +≤≤−= µµ

=449.61 Angstroms 39.4983 ≤≤ µ Angstroms Results Using Minitab From Minitab results (Table 6), we can conclude that we reject the null hypothesis because the P-value is smaller than 0.05 and 71.563.1130 >=F . In other words, we accept the alternative hypothesis that means the mean of the oxide thickness etched are different, and the level of DC bias voltage affects the mean of the oxide etched. Table 6: Results of ANOVA analysis of plasma etcher example using Minitab. Analysis of Variance Source DF SS MS F P Factor 2 105672 52836 113.63 0.000 Error 9 4185 465 Total 11 109857 Individual 95% CIs For Mean Based on Pooled StDev Level N Mean StDev ---+---------+---------+---------+--- 398 4 244.75 26.04 (--*--) 485 4 344.87 26.60 (--*--) 571 4 474.00 3.08 (--*--) ---+---------+---------+---------+-- 240 320 400 480

Experimental Plan Complete the one-way ANOVA worksheet to develop a detailed designed experiment to investigate the influence of one variable on your assigned process. The designed experiment needs to be carried out and analyzed in two weeks so project management is a very important part of the design process. It may be necessary to divide the workload up between team members.

Laboratory Report Your report should include the following: -Experiment objective -A brief discussion about the theory of each process -One-way ANOVA including controlled and uncontrolled variables, level chosen and why, procedures, -A detailed statistical analysis Average value for each level ANOVA analysis showing at table 3 -Plot of your data showing variation within level and trend between levels

ChE/MatE 166 Advanced Thin Films San Jose State University

Lab 1 Handout - 10

-Discussion of influence of your variable on the process Is it statistically significant? Is it what your hypothesis predicted? Why or why not? -Discussion of possible future experiments to further improve the process -Summary/ conclusion

Acknowledgments The lab was created by Prof. G. Young and Prof. S. Gleixner of the Chemical and Materials Engineering department and Irene Susanti Wibowo as part of her M.S. project in Industrial Systems and Engineering.

Lab Evaluation This portion of the lab will not be graded. I would greatly appreciate it if you did take the time to answer the following questions in regards to your overall satisfaction with the lab. Answers to these questions will be collected anonymously at the same time you turn in your lab report. Please take the time to turn in the attached ANONYMOUS survey. Even if you’ve got no comments please turn in the blank form so I know that you at least read the survey questions.

ChE/MatE 166 Advanced Thin Films San Jose State University

Lab 1 Handout - 11

ANONYMOUS LAB SURVEY: NO NAME PLEASE 1.What was the most useful learning experience that this lab provided you? 2.Do you feel the one-way ANOVA instructions provided you with new information that will be helpful in designing future experiments? 3. Do you feel the step by step process of designing an experiment provided you with new information that will be helpful in designing future experiments? 4.What would you suggest to improve how the lab requires you to explore the concepts of designing a single factor experiment?