A Framework for Media Comparison Analysis

A Framework for Media Comparison AnalysisBryan R. Warnick

Nicholas C. Burbules (?)

University of Illinois at Urbana Champaign

Media comparison research has been a hotly disputed area of educational research. This research aims at finding significant differences in educational outcomes between students who are taught using different types of media. Thus, one group of students will usually be taught using one media, another group will be taught with a different media, and the results of subsequent tests will be compared. This research hopes to find, if there is an educational difference between media, which media are more effective in the process of teaching and learning. At first glance, this research would seem to be an important helping in navigating classrooms through the wealth of new media that have recently become available to educators.

Media-comparison studies, however, have been accused of being both useless and conceptually incoherent. Media comparison studies are said to be useless because the majority of such research finds no statistically significant difference between learning groups. The No-Significant Difference website (http://teleeducation.nb.ca/nosignificantdifference/) lists 114 studies dating from 1920 that show no significant different between technologies, compared to only 39 that do. According to Thomas Russell, one of the sites organizers, some doctoral candidates in Adult and Continuing Education Programs are being discouraged from pursuing such research, in that it amounts to beating a dead horse. And rightly so, he argues: The No Significant Difference Phenomenon listing will no longer be complied when the futility of conducting such studies is recognized and their production ceases.

Russell also advances a theory about why studies commonly find no significant difference between media. He writes:

Technology is not neutral, despite the fact that study after study has concluded that using it in the classroom neither improves nor diminishes instruction for the masses. The truth lies in the fact, often acknowledged but ignored, that students are not alike. Individual differences in learning styles dictate that technology will facilitate learning for some, but will probably inhibit learning for others, while the remainder experience no significant difference. Therefore, when lumping all the students together into a fictional "mass," those who benefit from the technology are balanced by a like number who suffer; when combined with the no-significant-difference majority, the conglomerate yields the widely reported "no significant difference" results.

Russell is probably correct that this conglomerate effect has been playing a part in much media comparison research. However, this result may occur with any line of research into teaching and learning, not just the research into media differences. Indeed, individual differences are obscured in almost any statistically based study, but this does not mean that it is impossible to demonstrate broader trends. Thus, the conglomerate effect, as a possibility shared with other programs in educational research, does not seem to be a compelling reason to stop media comparison studies. Russells argument, however, seems to be that, given the existing evidence, the conglomerate effect has been shown to be the necessary result of media comparison research. It seems unwise to make such strong statements, though, even if they are valid summaries about the state of research. New media are constantly emerging, and what was true of the past may not hold for the future.

Perhaps the most prominent critic of media comparison research has been Richard E. Clark. In his numerous attacks on media comparison research, he writes that media are mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition. He also claims any necessary teaching method could be designed into a variety of media presentations. To the question of why some students have found a significant difference between educational media, he argues that these studies have confounded media with method. That is, the studies that have found a difference are really only finding a difference between two methods of instruction and not between the media themselves. Thus, such studies have failed to hold method constant making it inappropriate to conclude that media are the important independent variable.

In support of his argument that media will never influence learning, he presents a medical analogy. Educational media are compared to the various ways in which medicine can be delivered to the body (e.g., pills, injections, suppositories, etc.). Clark argues that the manner of delivery makes no difference to the healing properties of the medicine: All of these different media are often capable of delivering a necessary active chemical ingredient with necessary levels of efficacy, but with more or less equal effects on our physical symptoms. The chemical component of the medicine he then compares to the method of instruction. Methods of instruction, like the physicians medicine, can be distributed in many ways that will be equally effective.

This debate highlights important issues that arise in researching educational technology. The remainder of this paper will discuss some of conceptual confusions that seem have arisen in this debate. There are at least three levels on which conceptual confusion can occur as media are compared. It can occur (1) when there is an unclear notion of what it means to make a comparison, (2) when there is a simplistic view of how technologies can be compared, and (3) when there are confusions relating to educational media comparison specifically. The goal of this paper is to attempt to untangle some of these confusions, so that media comparison studies may become more profitable.

WHAT DOES IT MEAN TO MAKE A COMPARISON?

Nicholas Burbules, in an article entitled, A Grammar of Difference: Some Ways of Rethinking Difference and Diversity as Educational Topics, undertakes a conceptual analysis of difference, a topic that has much to do with the concept of comparison since finding differences is accomplished through comparison. Burbules makes several points about the necessary assumptions that need to be made in order to say X is different from Y, and which thus undergird the process of comparing. A comparisons of X and Y must be done within a framework that makes the comparison possible. It would be difficult to compare, for example, the color orange with the number three -- there must be some common ground that will make the comparison interesting. Thus, any comparative framework will necessarily imply similarity along with any difference (to fruitfully compare apples with oranges we may assume a common category of food, and focus on taste, nourishment, and texture). Moreover, assertions about what is, and what is not different, are in a constant state of flux as the categories making up the comparative framework change. What we find different between X and Y at time T1, may turn out to be irrelevant at T2, or even turn into a similarity not because X and Y have changed (although that may happen), but because the conceptual framework we are using has changed. In sum, to make a comparison involves a making various and mutable assumptions about similarity. Similarities will necessarily be assumed in the process of comparing and finding difference.

What else is involved in a comparison? A comparison typically involves a point or aim that gives the comparer motivation to make the comparison. There are at least two types of aims that can give a comparison a purpose. First, comparisons can be evaluative. In an evaluative comparison, we want to find out which of two or more things is better suited for a given purpose, P. An example of an evaluative comparison would be comparing two teams to see discover which one is superior. Second, there are interpretive comparisons. In an interpretative comparison, we do not wish to find out which is better for a given purpose; instead, we simply want to increase understanding by holding up one thing in light of another. An example of an interpretive comparison would be comparing the development of Leer with Hamlet. While it is possible to do interpretative comparisons in research on educational technology, much media comparison research is often of the evaluative type of comparison. This will type of comparison, then, will be our focus here.

An aim will include with it a comparative standard. This comparative standard will stipulate the standards by which success at achieving the aim will be known. A standard will spell out (or operationalize if you will) what we mean by superior. That is, it will stipulate the measure according to which superior will be judged. When making comparisons in pedagogical techniques, the common standard is, of course, an objective test, administered after an intervention. In these cases, the objective test gives the standard by which to differentiate effectiveness of the intervention. The higher test score is the standard for choosing which educational intervention is superior.

A comparison must also stipulate an aspect of comparison. That is, in order to compare two things, we need to ask, Which aspect of the things are to be compared, among the infinite possibilities? For example, in comparing two teams to find out which is superior, I may compare many things: hair colors, shoe sizes, scoring averages, heights, weights, birthdays, past records, favorite foods, etc. To make a comparison successful, I need to narrow down this list of possibilities to a list of relevant features. Only some of these aspects of comparison will be relevant to the comparison under any given aim.

ASSUMED SIMILARITIES IN FRAMEWORKS OF COMPARISON

Using these three parts of a comparison -- aim, standard, and aspect along with the idea that for any comparison to succeed certain similarities must be assumed, lets look more closely at the media-comparison studies. What similarities tend to be assumed in this field of research? First, media studies usually assume a common purpose between two technologies only with reference to a common purpose can two media be easily compared. For any given comparison, two media must be evaluated in terms of a similar goal.

In assessing whether purposes have been achieved, media comparison studies usually also assume a similar standard of evaluation between the two media under consideration. It would be difficult to compare two educational media, for example, if different tests were given to those who used a different media. Such an act would, of course, confound the variables, thus making a media comparison problematic. Both groups must be given the same test. Media comparison studies must operate under a similar standard of evaluation.

Media comparison studies generally look at the common ability of two different media to develop skills, increase knowledge, or otherwise promote learning. The experience of two media may be different in many ways one may increase eyestrain, another may come with a soporific humming noise, another may be simply new and exciting. Media comparison studies must focus on a narrow aspect of the media, and generally chooses to focus on the efficacy with which media promotes an intended given learning outcome. Media comparison studies must focus on a similar aspect between two technologies.

To the extent that media comparison studies assume similarities in order to make a comparison, they will also suffer from various problems. Many people would be quick to point out that the standard of comparison, objective test scores, may often miss what differences there may be between two media. There has been some debate, for example, about the differences between actual cadaver dissection in medical school and anatomical simulations. Empirical research has reported the usual no significant difference result between simulated dissection and real dissection when anatomy knowledge is tested by objective tests. Yet there remain differences between these two experiences, of course. At the very least, one situation comes with the smell of formaldehyde, the other doesnt. But there are other differences more closely tied to education: in one situation there is the knowledge you are working with a dead human body, in the other you isnt. Perhaps the best way to describe this difference would be one of mood. There is a different mood between cutting into a human body and using an anatomical simulation. This mood may or may not make an important difference to education, but it would be a difficult thing to reduce to an objective test. Assuming a framework of similarity when it comes to the standard of objective tests will be problematic when it comes to finding differences like alternation is mood.

To further underscore this point, consider Clarks medical analogy: medicine can be delivered to the body in various ways, thorough pills, injections, suppositories, etc. The manner of delivery, Clark argues, makes no difference to the healing properties of the medicine. The medium of delivery fails to affect the medicinal effect of the drug. We may grant this point, but that does not mean that media of delivery does not matter to healing. Medical instruments also come with mood. Needles may invoke a prohibitive mood in the patient; he or she may refuse treatment. If the patient refrains from the medicine because of this mood, then that can be said to be a factor affecting the healing process. If healing is comparable to learning, and if media influence mood, then the medical analogy would support of the point that, contrary to Clark, media matters to learning. Learning cannot be divorced from things like the mood in which learning takes place.

Problems also arise when we assume a common purpose for media technology. Technology always changes the ends of human activity. Technical ends do not simply remain fixed as means change. Using a new technology opens up new ends, it does not just increase the speed or efficacy involved in reaching the old ends; rather, using new technologies influences ones ends. In a sense, then, existing means have helped to construct existing ends. This fact often muddies technology comparisons. It would make no sense to evaluate the difference between automobile-infused cultures with that of a culture that walks everywhere under the framework of ends constructed in the walking society. That is, it would make no sense to compare the time it takes to walk to the nearby drug store with that it takes to drive there, and then declare triumphantly that a significant difference of 3 minutes had been found. This comparison would miss everything important about how automobiles change things. The fact that automobiles exist changes the ends the walker does not have the end of just going to the drugstore anymore. To take an educational example, consider using a long division calculator program to teach long division, and then comparing this to teaching long division through paper and pencil. This is problematic, because the fact that a long-division calculator program exists changes the end of the activity. The point of math education may then no longer be learning algorithms, but may shift to learning how to use the application. Thus, it is not always helpful to compare different technologies in relation to a common end.

Finally, assumptions about the similar aspects of comparison are also problematic. The assumed aspect of media comparison studies is usually the ability to promote an intended learning outcome. Such studies may miss the aspects that surround the medias ability to promote an intended learning outcome. The narrow focus may leave out social, cultural, and physiological differences that may exist between the media. Comparing the ability of a book and a computer to simply promote an intended learning outcome may leave out the fact that students learn about their surrounding cultural life by using its tools. Using computers in schools sends the message that it is good to use computer it serves as a mark of societal validation (the same holds true for books). Thus, using a media may promote future economic consumption of that media. If this is true, this phenomenon would certainly be a type of learning, but it would be left out in studies focused on the aspect of specific, intended learning outcomes.

COMPARING TECHNOLOGIES: PROBLEMS AND POSSIBILITIES

Suppose we wanted to compare the speed of traveling by horse with the speed of traveling by car. This seems to be an easy experiment: designate a beginning and end point, and time how long it takes to reach the end point using the two travel techniques. Indeed, it seems we dont even have to do the experiment traveling by car is obviously much faster than traveling by horse. But consider if I was mounted on a horse, riding around the cattle car of a moving train. In this context, it seems that I could both truly say I was traveling by horse (in the context of a moving car) and that I was traveling faster than a car. This thought experiment, though, need not be so silly consider that the car has broken down. Then traveling by horse is faster than traveling by car (in the context of a broken-down car). The use of a technology cannot be divorced from the context in which the technology finds itself. The physics of a tool underdetermines its context of use.

What does this mean? At first glance, it would seem to make any general statements comparing two technologies impossible. Since contexts are always variable across two or more situations, it makes no sense to make any general statement across these contexts. To compare technologies is to also compare contexts of use, and contexts of use cannot be stipulated in a generalized statement. At the same time, it also seems extreme to say that you cant make any general statements about the comparative speed of a horse versus the comparative speed of car. Why does it seem misguided? The reason is that general comparative statements make built in assumptions about normal uses. When we way that a car is faster than a horse, we make a statement assuming a normal context of use one does not usually travel by horse in the context of a moving cattle-car. This is understood when the comparative statement is made.

So, in order to compare two technologies, we need to determine what a normal context of use is. Admittedly, this can sometimes be a difficult problem.

While it may be easy to see what something like scissors are normally use for, the case become murky with some technologies like computers that seem to have an unending variety of functions. There do seem to be several characteristics of technologies, though, that offer some hope in stipulating a normal context of use. These characteristics show that stipulating a normal context of use is at least theoretically possible.

First, an important characteristic to remember about technologies is that they are not neutral. They are not neutral in many senses. Household appliances may be built specifically for women, for example, and thus betray a lack of political neutrality. This comes from the fact that technologies can suggest a context of use. By having a single keyboard, computers suggest a context of solitary use, rather than group work (although groups can, of course, awkwardly use a single computer keyboard). Thus, although technologies may still be used awkwardly in a wide variety of contexts, their designs often suggest one context over another.

Moreover, even the awkward contexts of use are not indefinitely variable. Some contexts of tool use are impossible, for example, because of the physics of a tool. One could not, no matter how inventive one was, use a fork made entirely of plastic to cut through a diamond. Thus, plastic forks will never by found in the context of being used for diamond cutting. Furthermore, something may be technically possible, but never arise because it presents contradictions in practice. For example, it may be possible to use motor oil to build a bathtub, assuming temperatures were low enough to carve a bathtub out of frozen motor oil. However, in such a context to make the physics work, no one would want to use this bathtub to actually take a bath it would be too cold.

If we were to take an example of this type of limitation on normal uses from education, we could turn to an example by Clark. As he attempts to show that one task can be performed by several technologies, he writes:

Many writers seem to suggest that these methods are somehow intrinsic to a given medium. My argument is that the usual uses of a medium do not limit the content it is capable of presenting. Computers can present realistic, real-time documentary information [like a television], and television can present semantically dense simulations [like a computer].

While it may be true that educational methods can be replicated by TV and computers, it is not true that a book could ever present real-time documentary information the delays of the publishing process forbid it. Thus, not all methods can be replicated by in all media, although any one method may be replicated by multiple media. While educational media might not have any one intrinsic method, there may exist certain methods that any one media intrinsically cannot have. Technologies cannot be used in an infinite variety of contexts for an infinite variety of purposes.

So, with these two factors limiting context use in mind, how are we to find normal contexts of use? First, it is important that normal contexts of use cannot be found by consulting the intentions of a designer. Often technologies are used in contexts (and for purposes) that are vastly different than what was intended on being designed. Instead, to find normal uses, the best thing to do is look at how people actually use the technology. Sometimes, there will be a fact of the matter with regard to a technologys context of use. From my observations around campus today, I found that bicycles are more often used in the purpose of traveling to class than as a weapon; I also found that single seat bicycles were much more common than multiple seats, etc. If I were comparing bicycles with another technology, such observations would serve as my assumed context of normal use.

Having said that, there should be added to caveats. First, if a fact of the matter is indeed found concerning a normal context of use, it is important to note that this may change abruptly. It may have been true 10 years ago that the normal use of classroom computers was for drill-and-practice exercises. Although this still may be a widely shared context of use, it can no longer be assumed that this is the normal context of use. Second, with regard to information technologies, it is doubtful that any one relatively stable normal context of use can ever be stipulated. If a thing is broken down into data and reconstituted into virtually, then the thing breaks free from any limitations placed upon it by physics. If the idea of a palm-tree is digitized and placed in a virtual world, it can be planted and flourish on an iceberg. The thing can be refigured and reshaped to fit almost any context.

FURTHER ISSUES IN MEDIA COMPARISION

One of the major points raised by critics of media comparison research is that economic differences between two media should be separated from differences in learning. Clark writes, for example, Media may have important influence on the cost or speed of learning but only the use of adequate instructional methods will influence learning. The point of such assertions is to combat the irrationality that sometimes may enter when decisions are made about educational technology. Often, costly information technologies are thought to be absolutely necessary, even though they have yet to demonstrate any difference in learning outcomes. Thus, money and effort is wasted to secure high-tech media, when other less-costly methods could easily fit the existing need. This point is well taken.

As a conceptual point, however, the distinction between economic differences and learning differences fails to hold up well to close analysis. It is clearly the case that making an economic difference is making a difference in learning. If a media makes a particular method financially possible, then this is an important educational difference, even if that method could theoretically been achieved in other ways. It would always be theoretically possible to visit Japan and talk with Japanese people face to face. But this method of learning about Japanese customs would be not be possible for most children under normal economic conditions. Conversely, with information technology it would be possible. Economics make all the learning difference in the world.

Another major point of the media comparison critics is that media should not be confounded with method. They argue that a study is never just comparing two media, but is always comparing the media and method. Therefore, method should remain constant in order to isolate the variable of media.

But is there a real conceptual distinction between media and method? Media and method are both ways of doing things, and of accomplishing goals. They may both arise in context of the question, How are you going to do that? To this question, I may reply, Im going to do that on my computer, or I might say, Im going do that by drill-and-practice. Or, we may reply to the question, How are you going to do that? with a response that includes both method and media: Im going to do drill and practice on my computer. Thus, a media or a method may be offered in response to the question, How are you going to do that?

One further doubt about a sharp separation between media and method is raised when it is realized we ask ourselves what we mean by media. Media are thought of as vehicles for delivery of something, usually some sort of communication. Indeed, we tend to think of media as methods of delivery. We can ask what is the method for having students do drill-and-practice? And reply that the method is using the computer. Media are one class of methods; they are methods that are used in the delivery and transportation of something. Thus, media are one sort of method, just as a truck is one sort of motorized vehicle. It would be a mistake to make a sharp distinction between a truck and a motorized vehicle. Any one teaching and learning procedure will actually consist of multiple methods methods of delivery (media), methods of attention-holding and motivating, methods of evaluation, etc.

CONCLUSIONS

This paper has raised several points that may be of help to those trying to compare various media.

To compare any two things, one must find a comparative framework built on similarity. The similarity often assumed for media comparison studies is problematic in assuming similarities among the purposes, aspects, and standards of comparison of different media. Purposes change as new media is introduced, objective tests may be impotent to find differences in things like mood between media, and the ability to promote intended learning is too narrow an aspect.

Comparing two technologies is also to compare contexts of use. When attempting to make general comparisons about media, the context of use must be stipulated. Although it may be difficult to find normal uses of technology, certain characteristics of technologies make it a theoretical possibility. To find normal uses one must turn to observations of how technologies are actually used, and not rely to the intentions of designers. Analysis of media outside of how they are actually used in a classroom (apart from the watchful instructions of designers) may be misleading.

Economic facilitation cannot be divorced from learning outcomes. To make a method economically feasible is to affect learning.

Media cannot be divorced from method media are methods of delivery.

It is hoped that these points will be of help in developing a more useful framework for media comparison analysis.