Showing Concern A special creativity extension to

Constructive, Inquiry-Based, Multimedia Learning in Computer Science Education (Grant Number EIA-0087977)

1.0 Introduction

In our original proposal to the NSF CRCD Program, we asserted that �it is crucial that students perceive that their instructors are genuinely concerned about them as individuals.� There is significant support for this assertion in various bodies of both educational and social/psychological literature. For example, Eugene Edgar confirms this in the following excerpt from his article Developing Civility in the Classroom Through Democratic Education:

Nell Noddings has coined the term �teacher as all caring� [Noddings]. Basically this means that the student believes the teacher cares for them MORE than the content of instruction. Students who hold this belief will treat the teacher with reciprocal care. Part, perhaps the most important part, of the teacher's role is to model and teach caring for others. This is not to indicate that the teaching of skills and knowledge is unimportant because a teacher who really cares about students will take the act of content teaching very seriously. Rather it means that when push comes to shove the teacher will place the needs of the individual above content. Judith Kleinfeld [Kleinfeld] also describes this aspect of teacher personality as the warm demander; someone who the student believes knows and cares about them as a human being and demands high standards of work and behavior. [Edgar]

Similarly, Kathleen Cotton highlights the fact that �results have shown that schools where students are involved in programs designed to increase empathy and create �caring communities� have higher scores than comparison schools on measures of higher-order reading comprehension� [Cotton]. Likewise, Tarajean Yazzie points out that effective teachers are �informal, are caring and warm, give up authority, and have and show respect for the students� [Yazzie]. Clearly empathetic, caring behavior enhances learning. Empathetic or compassionate behavior and attitudes have been shown to improve doctor-patient relationships and patient satisfaction in the medical field as well [Roter et al.].

At the recent NSF ECEG 2002 PI Workshop, we heard anecdotal evidence suggesting that our assertion is also true for computer science and engineering undergraduates. During the student panel, the women undergraduates emphasized the importance of a sense of community and of faculty members who shared that they cared about them as a role in attracting and retaining them in engineering. In addition, after the PI for this extension brought up the importance of caring during an open session, other faculty members approached him to affirm this factor in their own experience and to express interest in investigating this issue in the broader context of engineering education.

From an educational standpoint, researchers divide outcomes into three domains: cognition (intellectual operations); affect (attitudes, feelings, values); and psychomotor (body movement). The affective domain of learning [Krathwohl, Bloom and Masia] deals with feelings, attitudes, and values rather than intellectual operations (keeping in mind that feelings, values, and attitudes

2

all have a cognitive component). Simply put, learning involves emotions. The affective domain deals less with the ability to perform certain operations and more with the student�s willingness to do them. Without meeting the most basic levels of the affective domain (willingness to pay attention and respond), it is difficult to see how much in the way of intellectual development can take place. Robert Mager�s [Mager] minimal affective objective exemplifies this: when students leave a class they should like the subject matter no less than when they came to that class.

Based on our own experience as teachers these issues have come into focus. As a result, we believe there are a number of important research questions to be addressed:

• Does showing concern have the same impact on learning at the undergraduate level in

computer science (and engineering) disciplines that it has at the K-12 level? Specifically, does it affect performance on course assignments? Does it affect retention of students?

• Is showing concern a significant factor especially for women and minorities? • How do professors known to be empathetic communicate their concern? Many

instructors care about teaching and learning but are less successful at communicating it. • How do we evaluate how well a teacher expresses his or her concern? How do we

evaluate the impact of showing concern on actual learning, in undergraduate computer science education?

• Are their principles, techniques or heuristics that can be articulated and disseminated to other faculty members, graduate and undergraduate teaching assistants? How should this best be done: by written reports, web sites, workshops, course material or mentoring?

These are all research and development questions about human teaching and learning. They

are related to important research questions for the development of instructional technology. For example, can we create a collaborative learning environment that facilitates communication and thereby helps teachers show they care about their students?

Our original proposal noted that since �it is not possible for multimedia persona to meet the complete needs of human learners in this respect, we further propose to develop an interface in which persona are seamlessly connected, via networking technologies, to real, human instructors. Students will be able to reach beyond the professor, teaching assistant and librarian personae to communicate directly with a live person � an expert that the persona represents.�

Our vision is to create a learning environment in which rich multimedia content is tied to tools for inquiry-based learning and collaboration with genuine, caring instructors. At critical times, such as when a student is stuck and about to get frustrated with a programming task, or when a student is unsure how to investigate a research problem, students need to know they can get help from teachers that genuinely care about them.

In our current research we are creating and evaluating a framework for learning through multimedia, which we call CIMEL (Constructive, collaborative Inquiry-based Multimedia E-learning). We now propose to extend this project to investigate how to make a learning environment that helps teachers communicate concern. A number of important questions now come into focus:

• We have evidence from the field of psychology that eye contact, tone of voice, and body

language (e.g., gestures) all play a role in communicating concern. In CIMEL, personae of professors, teaching assistants and librarians interact with those of students to discuss

3

material and work on interactive exercises. Does the way personae look or the tone of their narration affect the way students perceive concern? If so, can they be improved and would such changes improve learning?

• Would it be effective if the collaborative environment facilitated awareness? For example, a teacher could be alerted that a student is trying to establish contact as well as to the nature of their question. A learner could be alerted as to the availability of a teacher to answer a question, or that other students are talking with a teacher about a problem that may be of interest. A learner might also be informed that a teacher has already discussed a similar problem with another student, archived in a library of �show me� tutorials. In essence the question is �Can awareness promote a sense of concern?�

• Introductory level students have misconceptions about what computer science practitioners actually do (i.e., they don�t just write code or email). Can video interviews of computer scientists at practice help clear away some of these misconceptions? Can these interviews also incorporate principles and techniques of showing concern?

These are examples of instructional technology research questions that we propose to explore

in this special extension to our CRCD grant. 2.0 High Risk, High Payoff

Special creativity extensions must be both high risk and have high potential payoffs. We believe our proposal clearly has both. Here are a few of the high risks:

• Are there a set of principles, techniques and heuristics for communicating concern that a community of teachers can agree on?

• Does the application of such principles, techniques and heuristics result in improved student performance at the undergraduate level? Would it increase the number of women and under-represented minorities entering computer science and engineering?

• Can these principles, techniques and heuristics be effectively taught and successfully learned?

• Can principles and techniques of showing concern be incorporated into an enhanced version of CIMEL personae, collaborative tools, etc.?

• Can we measure the effect of these enhanced tools on learning and learners? Here are some high potential payoffs:

• Significantly improve teaching by both faculty and teaching assistants. • Attract more students to computer science and engineering, especially women and under-

represented minorities. • Create a multimedia learning environment that engenders concern and awareness

between teachers and students and results in improved student attitudes towards and performance in computer science and engineering.

• Put a face on computer science and engineering that cares.

4

3.0 Approach

First, we will create a team of computer science and engineering professors known for their effectiveness in communicating their concern for students. Two possible candidates identified themselves at the NSF ECEG PI Workshop (Doug Christensen at the University of Utah and Don Elger at the University of Idaho). We will also seek to identify women and underrepresented minority teachers who possess the same qualities. We will create and conduct surveys for students to confirm that these teachers are indeed successful in this regard, why students believe this is so, and how concern impacts their performance in courses and their interest in the discipline. We will have educational experts observe some of these teachers in action. We will conduct focus groups with expert teachers and students to understand their teaching principles and techniques. For example, what impact does the teacher�s tone of voice or expressiveness of gestures have? What impact do a teacher�s activities outside of class have? We will also investigate appropriate metrics for measuring the impact of caring. Our team therefore includes experts in psychology, social relations, and education, to help us to address these issues.

Once we have identified principles and techniques for showing concern with which our team of experts agrees, then we will proceed on two majors development tracks: plan the training of teachers to use these principles and techniques in courses and evaluate their effectiveness; and use these principles and techniques to redesign the CIMEL collaborative tools, user interface and multimedia content.

Training of teachers at two or three test schools will first entail the development of curricular material based on the results of our preliminary research. We will develop and offer a course in the Computer Science and Engineering Department at Lehigh entitled �Teaching Computer Science� for graduate students who are (or wish to be) Teaching Assistants and for undergraduate Apprentice Teachers. We will invite faculty from our department as well as students and faculty from our local partner schools to participate (Cedar Crest College, a women�s college, and Northampton Community College, a two-year institution). Project team members with expertise in education, psychology and social relations will design this course, and will team-teach it with the PI the first time it is offered. We expect that the course will include at least some elements of empathy training given to medical students [Kramer et al.]. As a follow-up in the spring semester, we will conduct surveys and focus groups among both teachers giving instruction and their students in order to evaluate whether these principles and techniques successfully change the way students perceive these teachers and their disciplines.

Redesigning CIMEL software will entail development in several areas: user interface design (e.g., the appearance and tone of voice of personae); overcoming misconceptions about computer science with practitioner videos; and awareness in collaborative tools. We will also evaluate both the attitudes and performance of students to ascertain the impact of incorporating these principles, techniques and heuristics of caring into the CIMEL system. 3.1 User interface design

Examples of user interface issues that may pertain to showing concern include the following:

• Over the past summer, REUs and high school students working on the CIMEL project agreed that our current caricatures are less effective than actual digital photos of faces. Would personae with a greater variety of expressions or animation be more effective?

5

• Would it be effective if, when a learner clicks on a persona, the standardized personae graphics in the multimedia morph into graphics of actual teacher, offering to interact with the student electronically?

Following the investigation of principles and methods for showing concern, we will review,

revise and evaluate the user interface to make the CIMEL learning environment more effective. 3.2 Overcoming misconceptions about computer science

A recent Washington Post article, �College Students Turning Away from Bits and Bytes� (August 27, 2002, E1), described a trend of declining enrollment in computer science, a trend that began with women in the mid-90s [Camp] [Cohoon]. As the Post article suggests, part of the problem has to do with misconceptions that beginning students have about computer science, which are unfortunately reinforced by the way introductory courses are typically structured. In connection with the CIMEL project, co-PI Blank and his colleagues have written a second draft of a new book for CS0 or CS1, The Universal Machine: Introducing Computer Science with Multimedia. In the first chapter, we discuss several common misconceptions about computer science, including such ideas as �computer science is about nerds staring at computer screens�, �computer science is just a branch of mathematics�, �computer science is only about computer hardware�, etc. The text discusses each of these misconceptions and why they aren�t true. Multimedia could help, with video interviews of practitioners of computer science at work. In the introduction, different practitioners can discuss what they do in ways that reverse misconceptions. We will, as part of these interviews, actively incorporate the affective techniques that we compile as a result of the first phase of our proposed research (see Project Plan for a timeline of the project). For example, a chapter on computer architecture might include an interview with an engineer talking about how he designs new chips in the context of a community of researchers at his company. Alternatively, a chapter on user interfaces might include an interview with a designer talking about how she comes up with a design that engages a particular audience through the use of techniques that we have identified as effective in communicating concern. 3.3 Adding awareness to collaborative tools

An excellent opportunity to demonstrate concern is when a learner gets stuck on a concept or homework exercise and needs help. Often, learners (especially in a first year course) do not look for help at all, but if they do, a lack of availability limits its effectiveness. Sending off an email at any hour facilitates asking a question when it comes to mind. However, learners usually have no idea as to when the email might be read, and may feel neglected when a question goes unanswered when a fast reply is expected. As part of the CIMEL system, we have developed collaborative tools that address this problem by providing a chat room that identifies who is on-line and incorporates rich, remote-control interaction [Wang et al.]. However, there are still problems to be addressed, such as when a teacher may be running the collaborative tool, and be on-line, but may have stepped out of the office momentarily or be busy talking with other learners in the office or on-line. A learner who is not aware how busy a teacher is may have unrealistic expectations for getting a response.

6

As helpful as it might be, teachers cannot be available at all hours. When a learner cannot reach a teacher directly, he or she looks to see if the teacher has cared enough to anticipate the problem facing the learner and provided some advice, or provided answers of questions asked by other learners to all in the class. The collaborative tools we have developed make this more likely by making it easier to edit and save an electronic conversation and SHOW ME demonstration in a �multimedia frequently asked question� (mFAQ) archive that learners can access when no one is accessible in person.

We plan to develop and implement an interface and network system for providing awareness for teachers and learners, develop additional tools to ease the editing and incorporation of help sessions that include chat text and SHOW ME demonstrations in the mFAQ to provide relevant and focused help to other learners, and to evaluate the effectiveness of the awareness interface and mFAQ availability in providing learners with a sense of being cared for.

For our awareness interface, we plan to design a virtual space like the area around an effective instructor�s office door, where learners can clearly tell when an instructor is available, unavailable, or busy with other learners, and tell about how long it may take to get an answer. Providing awareness to users of computer supported cooperative work (CSCW) is an active research field [Beaudouin-Lafon and Karsenty; Dourish and Bellotti; Palfreyman and Rodden]. Our work focuses on two aspects of the problem, specific to course-related interactions. A combination of camera-based sensors can automatically determine if an instructor, TA, etc. is actually in the office and potentially available. An instructor will also get ways to explicitly indicate availability. A �waiting room� interface, using a office and conference table metaphor, will show instructors and learners currently in the office, who is waiting, and what topics are being discussed or will be discussed. With this interface, a learner can join an existing group that has the same question he or she has, and the instructor can choose to address certain waiting learners if the topic can be quickly answered. The interface will also give the instructor an awareness of how many are waiting outside his �office� in order to facilitate the flow of his or her interaction with learners. Such an �awareness widget� is similar to CSCW tools for collaborative document preparation [Gutwin et al.; Stefik et al.].

The multimedia FAQ (mFAQ) facility provides a useful alternative to contacting an instructor directly, since the same question often comes up for many learners. Such a facility is only as useful if it is used. We will investigate ways to streamline the steps necessary to create an mFAQ item and add it to the course mFAQ. We will provide a tool to generate automatically a series of web pages representing the mFAQ items listed in a hierarchy by course and topic. We will redesign the SHOW ME tool to record the application display, annotations, and chat messages in separate media tracks, and develop tools to edit annotations and chat messages (to correct errors, modify annotation positions, clarify statements, or remove private data such as learners names). We will enhance the SHOW ME playback tool to allow annotations that were shown directly on the application interface to be nudged or moved to the side to avoid occluding the interface, or moved closer to the object to which explanatory text refers.

In our investigations of awareness, we will study the correlation of �awareness� in collaborative tools and the availability of course specific mFAQ items with the caring factor. We will also design tutorials that help teachers use collaborative tools to promote showing concern. We note, however, that even though we develop techniques that employ instructional technology to communicate caring, they will not work unless the instructors doing the teaching really do care! �

7

4.0 Evaluation Component

Evaluation is an integral component of our proposed research. Issues that our evaluation component must address include, but are not limited to, the following:

• Identify faculty that explicitly identify themselves as caring (supported by student surveys), and faculty who do not, and somehow compare the students� performance. How much did the student like the class? Does this include the caring factor? If so, what impact did this factor have on student performance in the class? Do the techniques scale to women/minorities?

• If we develop methods, can they be learned? How can we evaluate that they been learned? Do the students have a better attitude towards the material? Are they more likely to ask for help? Are they less likely to drop the course? Are women or members of under-represented groups reporting more interest or enthusiasm for computer science and engineering?

• Does a collaborative environment facilitate the caring factor? Does a user interface with personae that morph into real people help convey the idea of availability and concern?

• Grad students typically have more contact than faculty with students in entry-level courses (labs, recitations). Can principles and methods of caring make them more effective? (Are they less set in their ways than older faculty with established habits with the result that they can learn to show concern more easily?)

Our evaluation plan involves two components. The first is a formative evaluation, which is

ongoing throughout the project. The second component is a summative evaluation, which documents the extent to which the project meets its goals.

The formative evaluation will provide a description of the context within which the project operates, the project participants and their roles, and the activities carried out to meet project objectives. Along with this historical accounting, the formative evaluation will provide an important feedback loop to help the Principal Investigators judge the usefulness of the various activities and tools stemming from the project. This feedback from the project�s clients (teachers and students) will allow the project staff to modify subsequent practices and tools to better serve these groups.

This aspect of the evaluation will involve qualitative methods, such as conducting focus groups, observing training workshops and course lectures, and attending project strategy sessions. It will also include feedback surveys of teachers and students, and possibly, interviews with selected individuals. The evaluator will develop the surveys with guidance from the Principal Investigators and Project Consultants familiar with the literature on empathy, the impact of the affective domain in teaching, principles of connecting, and gender issues connected with teacher effectiveness and evaluation.

The summative evaluation will assess the extent to which the goals of the project have been met. There are four major goals: to identify principles and techniques for showing concern; to train teachers to use these principles and techniques; to use these principles and techniques to redesign the CIMEL collaborative tools, user interface, and multimedia content; and to disseminate the results of the evaluation and provide training materials to the larger computer science and engineering community.

8

Create training materials and a collaborative learning environment that help teachers show concern to college students, to enhance learning and attract more students to computer science and engineering, especially women and minorities.

Identify principles and techniques for showing concern

Train teachers to use principles and

techniques for showing concern

Use principles and techniques for

showing concern to redesign the

CIMEL collaborative tools, user interface, and

multimedia content

Vision Goals Program Components

• Review the literature regarding teachers� showing concern in the classroom

• Survey teachers who identify themselves as concerned

• Survey students to identify other concerned teachers

• Have teachers and students describe characteristics of concerned teachers

• Develop training materials • Conduct workshop at

selected test sites • Teach course on showing

concern • Solicit feedback from

teachers implementing the techniques

• Evaluate effect of perceived concern on student learning

• Get student feedback on how current interface can convey concern

• Provide digital photos of faces with a variety of expressions

• Develop methods to increase awareness of the availability of help

• Provide video interviews of practitioners to reverse misconceptions about computer science

Disseminate results of

evaluation and provide training materials to the larger computer

science and engineering communities

• Hold a training workshop at a major conference

• Disseminate results to GK-12 teachers

• Publish results at national/ international /disciplinary/ regional meetings

• Publish results in pertinent journals

• Make information available to relevant and frequently- visited web sites

Evaluation Mechanism o Develop and pilot test

survey instruments for teachers/students

o Formulate and document procedures for collecting the data

o Analyze data and report findings

o Conduct focus groups and surveys of students

o Get/analyze feedback on workshops and course

o Follow-up survey to elicit feedback from teachers

o Develop instrument for student perceptions of learning

o Pretest and posttest of course content prior to and after implementing techniques of showing concern

o Develop survey for student impressions of interface

o Administer survey before and after redesign

o Pretest/posttest survey on attitudes toward computer science

o Prepare a final report on evaluation findings

o Document methods used to disseminate methods

o Estimate numbers of teachers having access to results

Figure 1: Project Vision, Goals, Program and Evaluation

9

To assess the first goal, the evaluator will develop instruments for teachers and students to enumerate the behaviors and qualities associated with teachers labeled as concerned. Along with confirming that students recognize the qualities described in the literature, the surveys may offer additional characteristics of concerned teachers.

The second goal involves training teachers to use the principles and techniques. The teachers who participated in the course on showing concern will be contacted periodically in the following semester to identify the principles and techniques employed and estimate the frequency with which they were used. They will be asked which techniques they find helpful and not helpful and to relate any difficulties in implementing the principles and techniques. Based on this feedback, the training materials may be revised.

To determine the effect on students of working with teachers who are able to communicate concern, the evaluation will employ a two-pronged approach. The first involves a subjective judgment by the student regarding the amount learned in classes taught by teachers who took the course on showing concern. The second approach will use a more objective measure of learning. Just before the semester preceding the training in showing concern, teachers will be asked to create a short exam covering the content of the course. This same exam will be given to the students in the beginning (pretest) and end of the semester (posttest), in both the semester preceding the workshop and the one following it. The first-semester students will be used as a control for the second-semester students, and comparisons of the pretest will determine if there are significant differences in the initial abilities of the students. Then, comparisons of the posttests for the two groups of students will determine whether learning had increased. Since this is a quasi-experimental design, other rival hypotheses will be explored to see if other factors might account for a differential in the posttest scores. Since it cannot be determined at this time that teachers conduct two classes with the same content in both the spring and fall semesters, the evaluation will have to rely on the subjective ratings of the students as well.

The third goal would be evaluated through on-line surveys that elicit student feedback on the interface, and then follow-up surveys on the revisions. Since one of the goals is to attract students to computer science by informing them of the activities involved, a pre and posttest survey would also include attitudes toward computer science.

Finally, the evaluator would document the methods through which the principles and techniques were disseminated to the larger computer science and engineering community and estimate the numbers of teachers having access to the information.

Figure 1 (previous page) summarizes the evaluation component in the larger context of the vision, goals, program components, and evaluation mechanisms for the project. 5.0 Project Timeline

Year 1, first six months: identify and describe principles, techniques, and methods for effectively communicating concern.

• At the September 2003 ECEG PI Workshop or shortly afterward, we will hold a brainstorming workshop to begin developing our first draft of principles and techniques of caring.

• We will then seek to confirm that teachers can indeed be indeed successful in this regard, why students believe this is so, and how concern impacts their performance in courses and their interest in the discipline. We will conduct interviews, focus groups and surveys

10

with teachers and students at various institutions, including minority and women-serving institutions. Based on our CIMEL project research, we have identified partners at Morgan State University, a historically black college, Cedar Crest College, a women�s college, and Northampton Community College, a two-year institution.

• We will write a report describing results of preliminary investigations, recommending principles and techniques for showing concern. We will share this report with teachers (both faculty and graduate) at several schools, soliciting feedback.

• We will redesign the CIMEL user interface and collaborative tools in response to findings from the report.

Year 1, second six months: design training materials, learning environment and evaluation plan.

• We will revise the recommendations in the report in response to feedback. • We will research, design and plan implementation and evaluation of these teaching

principles and techniques at two or three test sites. This will entail the development of a course to teach the principles, techniques, and heuristics of showing concern, as well as surveys and other evaluation materials.

• We will research, design and implement the CIMEL user interface and content in ways that incorporate these principles and techniques (e.g., use of personae and video interviews).

• We will run pilot tests of the collaborative tools and mFAQ to ensure that the interface is clear and usable, and develop a tutorial for teachers to make the most use of CIMEL tools.

• We will plan evaluation of the enhanced CIMEL and at least two different sites.

Year 2, first six months: Formative evaluation of training materials and learning environment.

• We will offer the course on showing concern to faculty and graduate student teaching assistants at two or more different schools, including one school teaching primarily women and/or minorities. We will also conduct evaluation of the teaching techniques.

• We will use and evaluate the enhanced CIMEL system in introductory courses at two or more different schools.

• We will conduct formative evaluation, for both teaching methods and instructional technology in the CIMEL system, using pre-tests and post-tests, observation, and surveys, and tracking data, in order to determine the effectiveness of the new techniques at showing concern in overcoming common misconceptions about computer science.

Year 2, second six months: Summative evaluation and dissemination.

• We will report the results of our use and formative evaluation to our team of teachers, as well as to colleagues at conferences, such as SIGCSE, ITiCSE and ASEE.

• Using results of formative evaluation, we will design and implement refinements and additions to our training materials and CIMEL system.

• We will plan and conduct summative evaluation of the training materials and the enhanced CIMEL system.

• We will offer to conduct workshops on showing concern in panel presentations or workshops at SIGCSE, ITiCSE, ASEE, and/or the Fall 2005 NSF ECEG conference.

11

6.0 Project Team PI William M. Pottenger, Ph.D., is P.C. Rossin Assistant Professor of Computer Science and Engineering at Lehigh University. Dr. Pottenger's research interests extend to a variety of topics in the fields of data mining and parallel and distributed systems. His research in emerging-trends detection builds on his research program in textual data mining funded in part by the NSF CRCD project [Kontostathis et al.; Roy et al.; Pottenger and Yang]. Dr. Pottenger will oversee the investigation of principles and methods for showing concern, planning of workshops, coordination of development and teaching of the course on showing concern, coordination of evaluation, and overall management of the project extension. Co-PI Glenn D. Blank, Ph.D., is the Principal Investigator of the current CRCD project and the lead author The Universal Computer and lead designer for the associated multimedia. He is an Associate Professor of Computer Science at Lehigh University. He has a wide-ranging background in natural language processing [Blank] and cognitive science. Dr. Blank developed most of the textbook and multimedia of The Universal Machine, supervising the work of a joint author, multimedia art director, illustrator, cartoonist, researchers, three graduate and eight undergraduate students, in coordination with professional multimedia developers sub-contracted by the publishers Richard D. Irwin and WCB/McGraw-Hill. Blank then developed and produced the completed second version of the Universal Machine. Blank is now PI of the CIMEL project, responsible for multimedia development, evaluation and overall project coordination. Co-PI G. Drew Kessler, Ph.D., is an Assistant Professor in the Computer Science and Engineering Department of Lehigh University. He will oversee the development and implementation of the collaborative tools and mFAQ generation. His dissertation was on frameworks and support libraries for extensible, dynamic, and distributed virtual environment systems. In addition to the activities of the NSF CRCD project, his current research activities involve software libraries and interaction techniques to support single user and collaborative virtual environments, multimodal and dynamic graphical applications, and networking protocols and distributed system algorithms for distributed, interactive applications. Dr. M. Jean Russo, a research scientist and scholar at the Center for Social Research at Lehigh University, will oversee and carry out the evaluation. Dr. Russo has conducted numerous evaluations funded by federal agencies, including the National Science Foundation, the Department of Labor, and the Department of Education. Dr. Russo has experience with all aspects of evaluation, including evaluation design, questionnaire development and testing, data collection and management, data analysis, and preparation of evaluation reports. Along with various evaluation activities, she has served for the past ten years as a consultant on the Family Study conducted at Lehigh University by R. C. Herrenkohl and E. C. Herrenkohl. In this capacity, she performed data analysis and co-authored several journal articles.

Dr. James Feldman has over 25 years of clinical and supervisory experience in the mental health field. He has provided psychotherapeutic care to individuals, groups and families. He has also provided clinical supervision and training to graduate social work students, psychology interns, third and forth year medical students and hospital based nursing staff. Dr. Feldman has been a professional educator for 17 years in the Graduate School of Social Work at Marywood University. The majority of his teaching has focused on the skills required for sound practice with various populations and cultures, including empathy training. He has also assisted in the design, implementation and data analysis of two national surveys focused on parenting issues and concerns.

12

Dr. H. Lynn Columba, Associate Professor in Education and Human Services in Lehigh University�s Graduate College of Education, brings broad expertise in STEM education to the project. Dr. Columba is also experienced in performing assessment in an educational setting, and has been Director of the Five-Year Combined Degree Program offered by the College of Education for over seven years. This program enables students to earn both a bachelor�s degree and a master�s degree in five years, preparing them for a variety of careers in education.

As consultants on the project, Drs. Columba, Feldman and Russo will aid the PIs in the planning and execution of brainstorming workshops to discover how college professors and TAs can use affective techniques to show concern and empathy to learners. They will also aid in the design, curriculum development and delivery of the planned course on showing concern. The Computer Science and Engineering (CSE) Department has committed to provide release time for one course per academic year for the three PIs during the two-year duration of this project. In addition, the CSE department will allocate one Information Technology scholar per year as a Research Assistant to work on the project; the proposal requests two other research assistants, for a total of three, one associated with each of the three PIs on the proposal. 7.0 Dissemination Plan By the end of the current CRCD project we should have a lot of multimedia, a working set of collaborative tools, and emerging trends detection tools for research-oriented inquiry-based learning exercises. The PI will discuss our work and plans at the upcoming Deans� Summit, in January 2003. At the end of year one, we will hold a workshop to perform training of teachers (faculty, graduate and undergraduate), in order to disseminate the principles and methods of showing concern, as well the curriculum plan for the course to be taught in the first half of the second year of the project. In addition, we will also instruct workshop attendees in the use of the CIMEL learning environment in introductory computer science courses. We plan to carry this workshop to several campuses in the region surrounding Lehigh University. The teachers that enroll in the course will report back to the research and development team with results for formative evaluation. By the end of the second year, we will organize a workshop at a major conference (SIGCSE, ASEE or ITiCSE), and possibly at the 2005 NSF ECEG PI Workshop (with the Program Managers� blessing �), to disseminate results. We also expect to disseminate the results of this study at a GK-12 teachers� conference to make it available at the GK-12 level. As we have already begun to do with the CRCD grant, we will also publish our results at national meetings such as SIGCSE [Blank et al. 2002a] and IEEE Frontiers in Education, international meetings such as ITiCSE [Blank et al. 2001] and ASEE [Blank et al. 2002], disciplinary meetings such as ECOOP [Blank et al. 2002b], and regional meetings such as Computing Consortium for Small Colleges [Blank et al. 2002c]. In addition to posters and presentations, we will follow the model of the BlueJ community by distributing sample CD-ROMs of our work at conferences. The CIMEL web site lets visitors try out the prototype and peruse relevant documentation. We will ask other relevant and frequently visited web sites involved in computer science education, such as SIGCSE (www.acm.org/sigcse), the Computer Science Teaching Center (www.cstc.org), BlueJ (www.bluej.org), etc., to establish links to our site. In addition, the PI is preparing a concept letter for the establishment of a web-based clearinghouse for all NSF-funded ECEG grant recipients. The site will highlight various projects, one each month, throughout the three-year period of the current ECEG initiative, and will of course include results of this proposed special creativity extension should it be funded.

13

8.0 Results from current CRCD grant: Constructive, Inquiry-Based, Multimedia Learning in Computer Science Education (Grant Number EIA-0087977 for $556,000)

The CIMEL project on Constructive, Inquiry-Based, Multimedia Learning in Computer Science Education began in October 2000. The PIs have built and evaluated a prototype in an introductory graduate level course, Object-Oriented Software Engineering (OOSE), and two upper level undergraduate courses, Software Engineering (SE) and Programming Languages (PL). Our preliminary results are promising [Blank et al. 2001; Blank et al. 2002; Blank et al. 2002a; Blank et al. 2002b; Blank et al. 2002c; Roy et al.; Kontostathis et al.; Wang et al.]. Our papers and other documents, evaluation materials and a prototype are all available at www.cse.lehigh.edu/~cimel. In this section, we briefly describe results and ongoing work on the multimedia framework, emerging trend detection tool, collaborative tools, and evaluation results.

8.1 Multimedia Figure 2 (following page) illustrates several features of the CIMEL multimedia framework:

• The TRACK LIST on left displays the content of a lesson as a sequence of screens. The menu uses check marks to show progress and highlights the current screen in red.

• Multimedia personae model a community of learners and instructors. The personae include a professor, teaching assistant (shown here), a reference librarian, and two students. In addition to graphical images, they speak in audio and/or text boxes. These personae model students and teachers studying material together, working through interactive and constructive exercises, and suggest exploratory research on relevant topics using online information.

• The COLLABORATE tools facilitate network-based interaction with other students, instructors, teaching assistants and librarians. Live links, remote-controlled SHOW ME sessions and a multimedia FAQ (mFAQ) of recorded SHOW ME sessions will encourage students to get help.

• The EXPLORE button facilitates inquiry-based learning, via directed queries on the web and an emerging-trends text mining and visualization tool. An emerging trend is a topic area for which one can trace the growth of interest and utility over time. (The prototype includes a multimedia tutorial setting up an emerging-trends exercise.)

• The FIND button will bring up a search tool, a glossary, and possibly index and search tools for the textbook.

• The PREFERENCES icon presents a panel of options letting the user adapt the environment according to his or her personal learning style, including turning text boxes or audio on/off, toggling auto-advance or wait for next page, setting the timing rate where there is no audio narration, etc. A user may change these settings at any time during a session.

• A JUST THE FACTS mode lets users switch to viewing non-interactive content (text and graphics) presented in HTML pages. From there, one can switch back to rich media mode via hyperlinks anchored to the corresponding Flash page.

While our approach is to present enough didactic material in the multimedia that it can be

a standalone learning experience, interactivity is frequent and rich in CIMEL content. Personae provide feedback to all responses, which are in turn logged to a web-based tracking

14

system for evaluation purposes. Constructive exercises challenge a learner to build solutions to problems by dragging and dropping pieces of structures into place, incrementally.

Figure 2: Screen Capture from CIMEL Prototype

8.2 Inquiry-based learning and emerging trend detection One of the goals of the CIMEL framework is to offer students ways to go beyond the knowledge presented in the course work, by exploring current research trends. Providing students with opportunities to explore the research literature related to a course gives them a better appreciation for the fundamentals presented in the courses, as well as a perspective of where technology is headed. Learning how to do research is also a valuable skill for lifelong learning. Inquiry-based exercises facilitate learning by doing research. For example, after studying ADT for collections, a screen asks to the student to investigate the design of similar ADTs in the most recent JDK. The following screen then asks a follow-up question designed to find out what the student has learned from the inquiry-based exercise. An emerging-trends tool provides more advanced inquiry-based exercises. Through the detection of incipient emerging trends the students sees the role that current topics play in course related research areas. An emerging trend is a topic area for which one can trace the growth of interest and utility over time. An example of such a trend is XML, a technology that emerged in the mid 1990s. The necessity for automated methods for detecting emerging trends has grown with the exponentially increasing availability of digitized information. Trend detections depends, however, not only on the data collected but also on domain expertise of the person searching for trends. As a result, trend detection is based on human-expert analysis of sources (e.g., patent, trade, and technical literature) combined with bibliometric and text mining techniques that employ both semi- and fully automatic methods [Kontostathis et al.].

15

As part of the CIMEL project, we developed an inquiry-based learning unit that teaches students how to do detect emerging trends in computer science research [Roy et al.]. Using multimedia, it explains the utility of emerging-trends detection with an interactive tutorial and assignment. Currently emerging-trends detection is a semi-automatic technique that we developed to improve the precision of previous methods. Our methodology detects emerging trends in a research area through the exploration of conference and workshop online sites for candidate emerging trends. These candidate emerging trends are then verified and additional candidate trends are discovered using a web-mining algorithm [Roy]. Trends are verified using semi-automated techniques that search a domain-specific repository of research literature [Gevry]. 8.3 Collaborative networking Figure 3 depicts the architecture of collaborative networking in CIMEL. In order to support high bandwidth information exchanges between participants, the framework for our collaborative tools uses a peer-to-peer communication scheme instead of the traditional client-server model. The server in our architecture only maintains global state information of participants, such as availability. The architecture includes collaborative tools for instant messaging and chat among two or more participants, and for remote SHOW ME demonstration of applications on the computer desktop. Communication sessions that occur through these tools can be recorded and added to a searchable multimedia frequently asked questions (mFAQ) database.

The SHOW ME collaborative tool provides, on the computer display of one person, a view of the graphical windows of an application (or entire screen, if specified) on another person�s display. Mouse pointers are shown on both displays and outline, filled area, arrow, and text annotations can be made directly on the application�s interface. SHOW ME help sessions can be saved as a video clip and added, by the instructor, to the mFAQ.

Figure 3: Software architecture for CIMEL collaborative learning

16

8.4 Evaluation of the CIMEL prototype With CRCD support, we have developed and evaluated new materials for upper level courses in computer science, including a graduate level course on Object-Oriented Software Engineering (OOSE) and upper level undergraduate courses in Software Engineering (SE) and Programming Languages (PL). We developed units on inheritance and dynamic binding and another on Abstract Data Types (ADTs). The latter formalizes the meaning of classes in connection with object-oriented design. In the past, upper level students have found it difficult to master this material from lecture and textbooks alone. Our premise was that multimedia would help students understand the material better, objectively, and also improve their performance on an assignment, designing actual ADTs to solve a problem. In the fall of 2001, we conducted a study on abstract data types (ADTs) in the OOSE course. Mean scores on objective multiple-choice tests improved significantly, suggesting that the multimedia does indeed contribute to objective learning of this content [Blank et al. 2002]. However, the results for task learning were less clear than the results for objective knowledge. These mixed results may have been due in part to the design of the experiment and in part to the design of the multimedia. Learning from our experience, we improved both. In spring 2002, we conducted a study in the SE course, using a 2x2 design, dividing a class of about 72 upper level undergraduates randomly into four groups: one fourth got neither the multimedia nor lecture, a second got just the multimedia, a third got just a lecture, and the fourth got both the multimedia and lecture. All four groups got the same homework problem description (see www.cse.lehigh.edu/~cimel/eval/beta/ADTassignment.htm). After completing the assignment, students took an online post-test of twenty multiple-choice questions presented in random order. The results, presented in [Blank et al. 2002b], are promising. The students getting multimedia score higher, on both the assignment, F(1,57)=12.25, p<.01, and the post-test, F(1,58)=15.39, p<.001. On the other hand, the lecture did not have a significant effect on either assignment grades, F(1,57)=1.11 or the post-test, F(1,58)=0.55, and there was no interaction between groups, for either the assignment, F(1,57)=0.01, or the post-test, F(1,58)=0.43. Thus the new multimedia has a significant effect on learning, both in terms of the objective knowledge (the post-test) and task knowledge (the assignment). Good as these results are, closer examination of the task results indicated that further improvement of the multimedia was possible. With or without the multimedia, most students missed an important component of the assignment�reusing a given ADT (List) to construct the semantics of a new ADT (Bowl). This observation led to a redesign of the multimedia to make this point clearer: rearranging materials on ADT for collections as a separate chapter and adding another exercise that explicitly gets learners to solve problems from existing ADTs. Further improvements have also been made in response to feedback to student comments on a usability survey. These modifications have led to improved performance in a study conducted in the OOSE course in the fall of 2002. Thus, evaluation is helping to drive our development process. An experiment was also conducted to test our methodology for detecting emerging trends in inquiry-based exercises. Twenty-one students in a graduate level OOSE class participated in this experimental evaluation to discover emerging trends.

The class was divided into two groups, groups A and B, each group having an approximately equal number of students. Students from both groups A and B were expected to have attended the lectures of the class. They were also expected to have introductory knowledge in the main

17

topic area before participating in this experiment. This was necessary as at different steps of the experiment they needed to apply their domain knowledge to justify their choices of emerging versus non-emerging trends. Also, all the students had access to their textbooks, reference books and handouts given in the class. Both groups A and B attempted an exercise that involved identification of three emerging trends in OOSE. In addition, group B was provided with the methodology. Group B was also provided with a practical case study that demonstrated the process of detecting emerging trends as outlined in the methodology. After completing the task, students in group A were given the methodology and case study and required to resubmit their solutions using the methodology.

Using a lower tail t-test we found, with a confidence level of 99%, that the mean precision of sample 2 (followed the methodology including resubmissions from Group A) was significantly greater than the mean precision of sample 1 (without the methodology). Also, with a confidence level of 95%, mean precision of sample 2 (followed the methodology excluding resubmissions from Group A) was significantly greater than sample 1 (without the methodology). We thus concluded that students who used the methodology performed significantly better on the task of Emerging Trend Detection [Roy et al.]. A similar experiment was conducted using the multimedia tutorial in an undergraduate Programming Languages class. As with the first experiment, students in the experimental group achieved significantly greater precision in detecting emerging trends when using the methodology implemented in the multimedia tutorial [Roy]. 9.0 Expected significance and impact

We believe that discovering and disseminating principles and techniques for showing concern will improve the teaching of computer science, as well as other engineering and science disciplines. Improving teaching and putting a human face on technological fields will help to attract students who currently may be intimidated, feel excluded or have misconceptions, especially women and minorities. Collaborative tools that enhance awareness will help students who get stuck early or often. Incorporation of diverse and caring personae and video interviewees of diverse practitioners at work, plus input on design, development and evaluation from students and faculty at Northampton Community College, Cedar Crest College and Morgan State University, will help assure that our materials appeal to a broad diversity of students and attract more women and underrepresented minorities into computer science. 10.0 Budget outline

If funded, this special creativity extension would begin in October of 2003. Our budget request for this extension is as follows: stipend and tuition for two graduate student research assistants for the two year duration of the project, and summer salary support for one month for each of the two summers for each of the three PIs. As noted earlier, the CSE department has committed to provide release time for one course per academic year for the three PIs during the two year duration of the project. In addition, the CSE department will cost-share one research assistant to work on the project; we thus requests two other research assistants, for a total of three, one associated with each of the three PIs. We also request $5,000 per year in travel funds, $3,000 per year for materials and supplies, and $45,000 to support the consultants (including the evaluator) on the project.

18

11.0 References

[Beaudouin-Lafon and Karsenty] Michel Beaudouin-Lafon and Alain Karsenty, �Transparency and awareness in a real-time groupware system,� Proceedings of the Fifth Annual ACM Symposium on User interface Software and Technology, Monterey, CA, November 1992, p.171-180.

[Blank] A finite and real-time processor for natural language syntax, Communications of the ACM 32(10), 1174-1189 (October 1989).

[Blank et al. 2001] G. D. Blank, W. M. Pottenger, G. D. Kessler, S. Roy, H. Jaffe and M. Herr. CIMEL: Constructive, collaborative Inquiry-based Multimedia E-Learning. The Sixth Annual Conference on Innovation and Technology in Computer Science Education, Canterbury, UK, June 2001.

[Blank et al. 2002] G. D. Blank, W. M. Pottenger, G. D. Kessler, S. Roy, D. R. Gevry, J. J. Heigl, S. A. Sahasrabudhe and Q. Wang. Design and Evaluation of Multimedia to Teach Java and Object-Oriented Software Engineering, Proceedings of American Society of Engineering Education, Montreal, June 2002.

[Blank et al. 2002a] Glenn David Blank, William M. Pottenger, G. Drew Kessler, Edwin J. Kay. Teaching Java and Abstract Data Type Semantics with Multimedia. Presented at the 33rd ACM Technical Symposium on Computer Science Education (SIGCSE), Northern Kentucky/Cincinnati, February 2002.

[Blank et al. 2002b] Glenn D. Blank, Edwin J. Kay, William M. Pottenger Jeffrey J. Heigl, Soma Roy and Shreeram A. Sahasrabudhe. Teaching Abstract Data Type Semantics with Multimedia. Workshop on Teaching Object-Oriented Programming, ECOOP, June 2002.

[Blank et al. 2002c] G. D. Blank, S. Sahasrabudhe, and J. Heigl. Adapting Multimedia For Diverse Student Learning Styles. 2002 Eastern Conference of the Consortium for Computing in Small Colleges, Bloomsburg University, PA, Oct 18-19, 2002.

[Camp] Tracy Camp. The Incredible Shrinking Pipeline Unlikely to Reverse. ACM�s Committee on Women in Computing. Available online at www.mines.edu/fs_home/tcamp/new-study/new-study.html.

[Cohoon] J. McGrath Cohoon. Women in CS and Biology. In Proceedings of the 33rd SIGCSE Technical Symposium on Computer Science Education, Northern Kentucky, February 2002, 82-86.

[Cotton] Kathleen Cotton. Developing Empathy in Children and Youth. School Improvement Series. Series VII, 1993-1994, Close-Up #13.

[Dourish and Bellotti] P. Dourish, and V. Bellotti, �Awareness and Coordination in shared workspaces,� Proceeding of the ACM 1992 Conference on Computer Supported Cooperative Work, Toronto, Ontario, December 1992, pp. 107-114.

[Edgar] Eugene Edgar. Developing Civility in the Classroom Through Democratic Education. 1998. Retrieved from the web on 10/13/2002: tiger.coe.missouri.edu/~vrcbd/civility.pdf

[Gevry] D. Gevry, Detection of Emerging Trends: Automation of Domain Expert Practices. M.S. Thesis, Department of Computer Science and Engineering at Lehigh University, 2002.

[Gutwin et al.] Carl Gutwin, Mark Roseman, and Saul Greenberg. �A usability study of awareness widgets in a shared workspace groupware system.� Proceedings of the ACM 1996 conference on Computer supported cooperative work, Cambridge, MA, November 1996, pp. 258-267.

19

[Kleinfeld] Kleinfeld, J. 1975. Effective teachers of Eskimo and Indian students. School Review, 83, 301-344.

[Kontostathis et al.] April Kontostathis, Leon Galitsky, William M. Pottenger, Soma Roy and Daniel J. Phelps. A Survey of Emerging Trend Detection in Textual Data Mining. In A Comprehensive Survey of Text Mining, Michael Berry, Ed., Springer-Verlag, 2003. (Forthcoming)

[Kramer et al.] Kramer, D., Ber, R., and Moore, M. 1989. Increasing empathy among medical students. Medical Education, 23, 168-173.

[Krathwohl, Bloom and Masia] Krathwohl, D. R., Bloom, B. S., and Masia, B. B. 1964. Taxonomy of educational objectives: Handbook II: Affective domain. New York: David McKay.

[Mager] Mager, R. F. 1984. Developing attitude toward learning (2nd ed.). Belmont, CA: David S. Lake.

[Noddings] Noddings, N. 1984. Caring: A feminine approach to ethics and moral education. Berkeley, CA: University of California Press.

[Palfreyman and Rodden]Kevin Palfreyman and Tom Rodden, �A protocol for user awareness on the World Wide Web,� Proceedings of the ACM 1996 Conference on Computer Supported Cooperative Work, Cambridge, MA, November 1996, pp. 130-139.

[Pottenger and Yang] William M. Pottenger and Ting-hao Yang. Detecting Emerging Concepts in Textual Data Mining. Computational Information Retrieval, Michael Berry, Ed., SIAM, Philadelphia, PA, August 2001.

[Roter et al.] Roter, D., Rosenbaum, J., de Negri, B., Renaud, D., DiPrete-Brown, L., and Hernandez, O. 1998. The effects of a continuing medical education programme in interpersonal communication skills on doctor practice and patient satisfaction in Trinidad and Tobago. Medical Education, 32(2), 181-189.

[Roy] Soma Roy, A Multimedia Interface for Emerging Trend Detection in Inquiry-based Learning. M.S. Thesis, Department of Computer Science and Engineering at Lehigh University, 2002.

[Roy et al.] Soma Roy, David Gevry, William Pottenger. Methodologies for Trend Detection in Textual Data Mining. Proceedings of the Textmine '02 Workshop, Second SIAM International Conference on Data Mining, April, 2002.

[Stefik et al.] M. Stefik, D. Bobrow, G. Foster, S. Lanning and D. Tatar, '�WYSIWIS revised: Early experiences with multiuser interfaces.� ACM Trans Office Information Systems, 5(2), April 1987, 147-167.

[Wang et al.] Qiang Wang, G.D. Kessler, G.D. Blank, William M. Pottenger. A TA Demonstration Tool. ACM 2002 Conference on Computer Supported Cooperative Work. November 2002, New Orleans, Louisiana, USA.

[Yazzie] Tarajean Yazzie. Culturally Appropriate Curriculum: A Research-Based Rationale. In Next Steps: Research and Practice to Advance Indian Education, Karen Gayton Swisher and John W. Tippeconnic III, Ed., 1999, soft cover, ISBN 1-880785-21-8, Educational Resources Information Center (ERIC).

20

William M. Pottenger Computer Science and Engineering Department at Lehigh University

19 Memorial Drive West, Bethlehem, PA 18015 610-758-3454 610-758-4096 (fax)

www.cse.lehigh.edu/~billp EducationPh.D. in Computer Science, May 1997 University of Illinois at Urbana-Champaign Champaign, Illinois M.S. in Computer Science, December 1994 University of Illinois at Urbana-Champaign Champaign, Illinois

B.S. in Computer Science, May 1989 Math, Science Secondary Teaching Credential University of Alaska, Fairbanks B.A. in Religion Studies, May 1980 Lehigh University Bethlehem, Pennsylvania

Current Position P.C. Rossin Assistant Professor, Computer Science and Engineering Department, Lehigh University Research & Work Experience

Computer Science and Engineering Department, Lehigh University National Center for Supercomputing Applications, University of Illinois at Urbana-Champaign

PI for multiple NSF, DoD, and industrial funded research initiatives. Several years research experience in the data mining and computational knowledge management field working on the development and implementation of HDDI , Hierarchical Distributed Dynamic Indexing, infrastructure for organization, categorization, mining and management of distributed repositories scaling across collection size and subject domain. Application areas include fields as diverse as Trend Detection, Patent Intelligence, Optimization, and Finance. Research integrated closely with educational outreach in computer and information science: organized The Technologies of Scalable Semantics seminar series in computer science, supervised undergrad/grad independent study and thesis research, and currently supervising graduate students and teaching computer science at Lehigh University. Research and educational outreach conducted at Lehigh University in Bethlehem, Pennsylvania and the National Center for Supercomputing Applications at the University of Illinois at Urbana-Champaign [1,2,3,4,5,6,7,8,9].

Department of Computer Science Center for Supercomputing Research and Development (CSRD)

University of Illinois at Urbana-Champaign Research, development and implementation of algorithms to automatically parallelize computer programs. Recent accomplishments include the introduction of a powerful and general model for characterizing parallelism in sparse and irregular Fortran, hybrid C/Fortran, and C++ codes [10]. C++ implementation of techniques completed as part of Polaris restructurer. Citations P.C. Rossin Assistant Professorship (2001-2003); United States Air Force Certificate of Appreciation (2001); Who�s Who Among America�s Teachers, 6th Edition (2000); Phi Beta Delta Honor Society; UAF Chancellor's List; Phi Kappa Phi Honor Society; Outstanding Student of the Year in Computer Science Publications 1. G. D. Blank, E. J. Kay, W. M. Pottenger, J. J. Heigl, S. Roy and S. A. Sahasrabudhe. Teaching Abstract Data

Type Semantics with Multimedia. European Conference on Object-Oriented Programming, Workshop on Teaching Object-Oriented Programming, June 2002.

2. Soma Roy, David Gevry and William M. Pottenger. Methodologies for Trend Detection in Textual Data Mining. Proceedings of the Textmine '02 Workshop, Second SIAM International Conference on Data Mining, April 2002

3. William M. Pottenger and Ting-hao Yang. Detecting Emerging Concepts in Textual Data Mining. In Computational Information Retrieval, Michael Berry, Ed., SIAM, Philadelphia, PA, August 2001.

4. Jirada Kuntraruk and William M. Pottenger. Massively Parallel Distributed Feature Extraction in Textual Data Mining Using HDDITM. In the Proceedings of The Tenth IEEE International Symposium on High Performance Distributed Computing (HPDC-10). San Francisco, CA, August 2001.

21

5. William M. Pottenger, Yong-Bin Kim and Daryl D. Meling. HDDITM: Hierarchical Distributed Dynamic Indexing. In Data Mining for Scientific and Engineering Applications, Robert Grossman, Chandrika Kamath, Vipin Kumar and Raju Namburu, Eds., Kluwer Academic Publishers, July 2001.

6. William M. Pottenger, Miranda R. Callahan, and Michael A. Padgett. Distributed Information Management. Annual Review of Information Science and Technology (ARIST Volume 35, 2001).

7. Glenn David Blank, William M Pottenger, G. Drew Kessler, Martin Herr, Harriet Jaffe and Soma Roy. CIMEL: Constructive, Collaborative Inquiry-based Multimedia E-Learning. Presented at The 6th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE), Canterbury, UK, June 2001.

8. Fabien D. Bouskila and William M. Pottenger. The Role of Semantic Locality in Hierarchical Distributed Dynamic Indexing. Proceedings of the 2000 International Conference on Artificial Intelligence (IC-AI 2000), Las Vegas, Nevada, June 2000.

9. L. S. Auvil, W. H. Hsu, William M. Pottenger, D. Tcheng, and M. Welge. Self-Organizing Systems for Knowledge Discovery in Databases. In the Proceedings of the International Joint Conference on Neural Networks (IJCNN-99), Washington, DC, July 1999.

10. William M. Pottenger. The Role of Associativity and Commutativity in the Detection and Transformation of Loop-Level Parallelism. Proceedings of the 12th ACM International Conference on Supercomputing, Melbourne, Australia. July 1998.

Recent Training Activities I have supervised dozens of undergraduate students in research projects over the last five years. I have

specifically published with R. Bader, N. Miller, D. Grim and J. Krause. In 1999, I supervised research for five graduate students: V. Pinto, S. Yih, Y. Chen, F. Bouskila and Y. Kim. In 2000, I supervised research for 11 graduate students: Z. Xiong, L. Zhou, M. Jang, Y. Jin, T. Yang, Y. Kim, V. Pinto, D. Gevry, A. Kontostathis, J. Kuntraruk and T. Wu. In 2001, I supervised research for 12 graduate students: L. Galitsky, J. Kuntraruk, A. Kontostathis, T. Wu, D. Gevry, S. Roy, Q. Wang, K. Peri, S. Mukkamalla, J. Eynon, H. Huo and Y. Kim. In 2002 (to date), I have supervised research for 12 graduate students: L. Galitsky, J. Kuntraruk, A. Kontostathis, T. Wu, D. Gevry, S. Roy, J. Eynon, S. Li, N. Chandrashekar, D. Deasey, R. Viswanathan and B. Wu.

While obtaining my secondary teaching certificate in Math and Science at the University of Alaska I taught continuing education courses and provided technical support for the College of Education. I also designed the local area network for a new private school in Champaign, Illinois. In addition, I worked for several years as the primary networking designer and consultant for the High School SuperQuest program. As part of this program I designed and implemented several compute server installations, wide area Internet connections and local area networks at high schools from California to Kansas to North Carolina using a variety of connection technologies. I also aided in the design of a district-wide network for the Champaign School District that combined local and wide area networks using both wired and wireless technologies. Collaborators, Advisors, and Students (* ���� graduated)

i. I. Alber (Boeing); L. S. Auvil (NCSA); R. Bader (Motorola); B. Beggs (Boeing); J. Bernardin (Data Synapse); G. Blank (Lehigh); J. Buonocore (Goldman Sachs); M. Callahan (NCSA); H. Chen (U of AZ); Y. Chung (UIUC); M. Farber (Boeing); M. Herr (Lehigh); T. Huang (UIUC); W. H. Hsu (Kansas State); H Jaffe (Lehigh); B. Kalathil (Lockheed Martin); A. Kao (Boeing); D. Kessler (Lehigh); S. Lathrop (NCSA); P. Lee (Data Synapse); D. Meling (NCSA); M. Padgett (UIUC); R. Panton (USAF); D. Phelps (Kodak); T. Prudhomme (NCSA); G. Roberts (Lockheed Martin); B. R. Schatz (UIUC); M. Skreiner (NCSA); D. Tcheng (NCSA); T. Thompkins (NCSA); T. Yang (Microsoft); M. Welge (NCSA); Z. Xiong (UIUC); N. Yeager (NCSA)

ii. David A. Padua (PhD Thesis advisor, UIUC); Rudolf Eigenmann (MS Thesis advisor, Purdue) iii. L. Galitsky (Lehigh PhD); A. Kontostathis (Lehigh PhD); J. Kuntraruk (Lehigh PhD); T. Wu (Lehigh

PhD); S. Li (Lehigh PhD); J. Eynon (Lehigh MS)*; D. Gevry (Lehigh MS)*; S. Roy (Lehigh MS)*; Y. Kim (UIUC MS)*; Y. Jin (UIUC MS)*; T. Yang (UIUC MS)*; F. Bouskila (UIUC MS)*; L. Zhou (UIUC MS)*; V. Pinto (NCSA RA); Y. Chen (NCSA RA); M. Jang (NCSA RA); S. Yih (NCSA RA); N. Yeager (NCSA FTE); M. Callahan (NCSA FTE); Y. Cho (NCSA FTE)

Courses Taught Computer Architecture I: Computer Organization and Design (three times with an average of about 150 students each time); Programming Languages (three times with an average of about 45 students each time); Data Mining (two times with an average of about 15 students each time); Advanced Topics in Textual Data Mining (two times with an average of about 10 students each time).

22

Glenn David Blank

Address: CSE Department, 19 Packard Lab, Lehigh University, Bethlehem, PA 18015 E-mail: glenn.blank@lehigh.edu Web page: www.cse.lehigh.edu/~glennb Academic qualifications: Ph.D., Cognitive Science, University of Wisconsin-Madison, 1984 M.S., Computer Science, University of Wisconsin-Madison, 1983 M.A., English, University of Michigan, 1975 B.A., English, Pennsylvania State University, 1974 Professional Experience: Associate Professor of Computer Science, Lehigh University, 1991 - present. Assistant Professor of Computer Science, Lehigh University, 1984 -1991. Teaching Assistant, Computer Science Department, University of Wisconsin, 1983 - 1984. Teaching Assistant, English Department, University of Wisconsin, 1977 - 1980. Computer Programmer, Veterans Administration, Philadelphia, 1975-1977. Education/Outreach Activities: Lead author for The Universal Machine: A Multimedia Introduction to Computer Science, 1998 Lead author for The Universal Computer: Introducing Computer Science with Multimedia, in preparation, with Robert F. Barnes and Edwin J. Kay, Lehigh University Faculty advisor for student chapter of Association for Computing Machinery at Lehigh University, 1990-1999. Student Research Supervision (last 3 years): Undergraduate: David Goldfeder, David Greenley, Greg Hild, Darryl Jones Masters: Jeffrey Heigl, Adam Kinnear, Andrew Mall, Sasisdhar Mukkamalla, Shreeram Sahasrabudhe, David Servas PhD: Amnon H. Eden, Hua Hwang, Alan Kwiatkowski Courses taught (past 3 years): Introducing to Computing (for majors and non-majors), Multimedia Design and Development, Object-Oriented Software Engineering Current and Past Collaborators: Arthur Kuntz (University of Wisconsin-Madison), Deborah Dahl (Unisys), Robert F. Barnes, William M. Pottenger, Edwin J. Kay, G. Drew Kessler and Henry Odi (Lehigh University), Morgan Jennings (Metropolitan State College of Denver) Membership in Professional Societies: Association for Computing Machinery

23

Relevant Publications: Adapting Multimedia For Diverse Student Learning Styles, with S. Sahasrabudhe, and J. Heigl. 2002 Eastern Conference of the Consortium for Computing in Small Colleges, Bloomsburg University, PA, Oct 18-19, 2002. Design and Evaluation of Multimedia to Teach Java and Object-Oriented Software Engineering, with W. M. Pottenger, G. D. Kessler, S. Roy, D. R. Gevry, J. J. Heigl, S. A. Sahasrabudhe and Q. Wang. NSF showcase, Proceedings of American Society of Engineering Education, Montreal, June 2002. Teaching Abstract Data Type Semantics with Multimedia, with E. J. Kay, W. M. Pottenger, J. J. Heigl, S. Roy and S. A. Sahasrabudhe. European Conference on Object-Oriented Programming, Workshop on Teaching Object-Oriented Programming, June 2002. CIMEL: Constructive, collaborative Inquiry-based Multimedia E-Learning, with W. M. Pottenger, G. D. Kessler, S. Roy, H. Jaffe and M. Herr.. The Sixth Annual Conference on Innovation and Technology in Computer Science Education, Canterbury, UK, June 2001. Experiences with The Universal Machine: A Multimedia Introduction to Computing, ACEE Ed-Media, Montreal, CA, June, 2000. The Universal Machine: A Multimedia Introduction to Computing, with Robert F. Barnes, WCB-McGraw-Hill, 1998. Semantic interpretation in linear time, with Kirk Mousley, Paul A. Kogut and Edwin J. Kay, Applied Intelligence, 5:4 (October 1995). Towards robust, real time processing of spoken language, with Kirk Mousley and Deborah Dahl, Proceedings of Avignon '94, Paris, France, June 1-3, 1994. PRIOPS: A real-time production system architecture for programming and learning in embedded systems, with Dale Parson, International Journal Of Pattern Recognition and Artificial Intelligence 4 (3), 1990, 509-526. Also in Advances in Artificial Intelligence: Applications and Theory, ed. J. C. Bezdek, World Scientific Publishing Series in Computer Science, vol. 27, 1990, pp. 205-222. Automatic vs. controlled processing: an architecture for real-time production systems, with Dale Parson, International Journal of Expert Systems Research and Development, 2 (3/4), 1989, 393-417. A finite and real-time processor for natural language syntax, Communications of the ACM 32(10), 1174-1189 (October 1989).

24

G. Drew Kessler Assistant Professor

Computer Science and Engineering Lehigh University

Degrees Georgia Institute of Technology, Atlanta, Georgia Ph.D. Computer Science Sept., 1997 M.S. Computer Science June, 1995 University of Virginia, Charlottesville, Virginia B.S. Computer Science, High Distinction, Minor in Electrical Engineering May, 1992 Service to Lehigh University Appointed to CSE department Sept. 1, 1999. Other Professional Experience University of Pennsylvania, Philadelphia, Pennsylvania, Lecturer 1997-1999 Georgia Institute of Technology, Atlanta, Georgia, Research Assistant 1992-1997 Internship at Naval Research Lab, Washington, D.C., Summer Intern Summer, 1994 University of Virginia, Charlottesville, Virginia, Research Assistant 1990-92 Principle Publications in Last 5 Years

• Kessler, G. Drew, �A Flexible Framework for the Development of Distributed, Multi-User Virtual Environment Applications.� Doctoral Thesis, Georgia Institute of Tech-nology. Professor Larry F. Hodges, advisor. Sept., 1997.

Refereed Journal Publications • Kessler, G. Drew, Doug A. Bowman, and Larry F. Hodges. �The Simple Virtual

Environment Library, an Extensible Framework for Building VE Applications.� PRESENCE, 9 (2), Apr. 2000.

• Hodges, Larry F., Benjamin A. Watson, G. Drew Kessler, Barbara O. Rothbaum, and Dan Opdyke. �Virtually Conquering Fear of Flying.� IEEE Computer Graphics and Applications, 16 (6), Nov. 1996.

• Rothbaum, Barbara O., Larry Hodges, Benjamin A. Watson, G. Drew Kessler, and Dan Opdyke. �Virtual Reality Exposure Therapy in the Treatment of Fear of Flying: a Case Report.� Behavioral Research & Therapy, 34 (5/6), 1996.

• Kessler, G. Drew, Larry F. Hodges and Neff Walker. �Evaluation of the CyberGlove whole hand input device.� ACM Transactions on Human Computer Interaction, vol. 2, no. 4, Dec. 1995.

25

Refereed Conference Proceedings • Kessler, G. Drew, �A Framework for Interactors in Immersive Virtual

Environments.� IEEE Virtual Reality �99, Houston, TX, Mar. 1999. • Kessler, G. Drew, Larry F. Hodges, and Mustaque Ahamad. �RAVEL, a Support Sys-

tem for the Development of Distributed, Multi-user VE Applications.� IEEE Virtual Reality Annual International Symposium �98, Atlanta, GA, Mar. 1998.

• Kessler, G. Drew and Larry F. Hodges. �A Network Communication Protocol for Distributed Virtual Environment Systems.� IEEE Virtual Reality Annual International Symposium �96, Santa Clara, CA, Mar. 1996.

• Hodges, Larry F., Barbara O. Rothbaum, Benjamin A. Watson, G. Drew Kessler, and Dan Opdyke. �A Virtual Airplane for Fear of Flying Therapy.� IEEE Virtual Reality Annual International Symposium �96, Santa Clara, CA, Mar. 1996.

Book Chapter • Kessler, G. Drew, �Virtual Environments Modeling.� In The Handbook of Virtual

Environments Technology, ed. Kay M. Stanney. Invited contribution. In press. Membership in Scientific and Professional Societies

• ACM (ACM SIGGRAPH) • IEEE (IEEE Computer Society)

Institutional and Professional Service in Past 5 Years

• Program Committee, IEEE Virtual Reality, 2001. • Tutorials Co-Chair, Program Committee, IEEE Virtual Reality, 2000. • Video co-chair and program committee member, IEEE Virtual Reality, 1999. • Video co-chair and program committee member, IEEE Virtual Reality Annual Interna-

tional Symposium, 1998. • Advisory board member, The Handbook of Virtual Environments Technology. • Reviewer, IEEE VR, 1999-2001, IEEE VRAIS, 1997-1998. • Reviewer, ACM Computing Surveys, 1999. • Reviewer, ACM Symposium on User Interface Software and Technology, 1998. • Reviewer, IEEE Transactions on Systems, Man, and Cybernetics, 1996. • Lehigh University EECS Department Faculty Search Committee, Curriculum Committee,

Spring 2000

26

Résumé

James H. Feldman, Ph.D. 1678 Chestertown Road, Allentown, Pennsylvania 18104

610-391-9772 Education BA, 1970, Long Island University, C.W. Post College MSW, 1977, Fordham University Graduate School of Social Services Ph.D., 1994, Human Services/ Human Development, Walden University Post Graduate Training September 1977 - December 1978, New York Center for Psychoanalytic Training October 1980 - Child Abuse Training Program, Children�s Aid Society of New York January 1981 - Clinical Hypnosis, University of Pennsylvania, School of Medicine September 1982 - May 1986 - Certificate in Advanced Theory Development,

University of Pennsylvania, Graduate School of Social Work Professional Experience April 1985 to Present

KidsPeace, The National Center for Kids Overcoming Crisis, Orefield, PA National Director of Public Education, (1990)

• Directed the development, implementation, and data analysis of local and national treatment and prevention research for use by health care professionals, parents and educators in the health/mental health care fields

• Provided clinical oversight and management to the KidsPeace National Psychiatric Hospital. Assessment of inpatient programs and clinical supervision. Worked closely with psychiatry, social work and nursing

• Author clinical material for publication in national media including: USA Today, The Psychiatric Times, The Washington Post and other outlets

• Provide clinical training to local, national and international professionals on a variety of clinical issues at workshops and conference

• Collaboration with renowned professionals in the health care field including Dr. Alvin Poussaint of Harvard Medical School, the late Dr. Lee Salk and the late Dr. Jonas Salk, polio vaccine pioneer

• Serve as the organizations clinical spokesperson to the media, appearing on national and international television networks including: NBC, CBS, ABC, CNN and the BBC. Numerous print interviews in publications including: The New York Times, Business Week, The Philadelphia Inquirer, The Washington Post, Baby Talk Magazine, The Psychiatric Times, and USA Today among others. Radio interviews on all major networks across the United States.

27

KidsPeace - April 1985 to October 1990 - Director of Outpatient Psychiatric Services Management of all aspects of both public and private outpatient psychiatric facilities serving several counties. Responsibilities included: psychotherapeutic care to adults, children and families, preparation of budgets, strategic planning, clinical supervision of professional and support staff and interfacing with county, state and federal officials. Also responsible for hiring and managing all clinical staff for the organization, totaling over 40 professionals in the fields of psychiatry, social work and psychology. Teaching Experience � Associate Professor Marywood University Graduate School of Social Work, Scranton and Allentown PA campuses - September 1986 to present Graduate level courses taught include: Family Therapy, Psychopathology, Macro Practice, Child Maltreatment, Ethnicity, Crisis Intervention/Short Term Treatment, Supervision and Administration of Human/Social Service Organizations, Advanced Theory and Practice with Individuals, Groups and Families Allentown Osteopathic Medical Center, Allentown, PA - June 1982 to April 1985 Provided clinical training to psychiatric nursing staff and third and fourth year medical students. Private Practice � 1981-1998 Private clinical practice working with adults, families, adolescents and children in the Allentown, Pennsylvania area. Referrals primarily from physicians and schools Publications Contributor. (2001). In Brooks, J. Don�t Let Death Ruin Your Life. New York: Knopf Feldman, J. (1996). Do Today�s Children Worry Too Much? Learning. March/April (pp.54-56). Greensboro, North Carolina: The Education Center, Inc. Feldman, J. (1994). Adult Victims of Child Maltreatment. Dissertation Abstracts. Interviewed and quoted in numerous articles in several publications including: USA Today. (2001) ; YM Magazine. (2001) ; U-Turn Magazine. (2002). Professional Presentations/Consultation Numerous presentations and workshops to a variety of professional organizations, nationally and internationally on clinical, management and human/social service issues

• The International Conference on Youth Violence, Israel, 1999, 2002 • The International Conference on Adolescent Healthcare, Royal College of Physicians,

London, England, 2001 • The United States Department of Justice, 2001 • The Canadian Institute of Child Health, 2002

• Television news magazine 20/20, 2001 • State and national organizations such as the National PTA, Girl Scouts of the USA and

the National Association of Social Workers

28

LYNN COLUMBA College of Education

Lehigh University Bethlehem, PA 18015

(610) 758-3237 A. Professional and Academic Essentials EDUCATION Ed.D. University of Louisville, Louisville, Kentucky, 1989 M.Ed. University of Louisville, Louisville, Kentucky, 1977 B.A. Morehead State University, Morehead, Kentucky, 1972 CERTIFICATIONS Elementary Teacher (1-8) Reading Specialist (K-12) Administration--Elementary Principal Supervision Endorsement PROFESSIONAL POSITIONS 1996-2000 Coordinator, Teacher Education Program, Lehigh University, Department

of Education and Human Services, Bethlehem, PA 1995-Present Associate Professor, Lehigh University, Department of

Education and Human Services, Bethlehem, PA 1989-1995 Assistant Professor, Lehigh University, Department of Education and

Human Services, Bethlehem, PA 1986-1989 Clinical Instructor, University of Louisville, Department of Early and

Middle Childhood Education, Louisville, KY 1986-Present Educational Consultant, School Districts in Pennsylvania, New Jersey, and

Maryland; International Schools in Guatemala, Thailand, and Philippines; Textbook Publishers

1972-1986 Elementary Teacher, Jefferson County Public Schools, King Elementary School, Louisville, KY

B. Five Pubications Most Relevant to Proposed Project

Spade, J. Z., Columba, L., & Vanfossen, B. E. (1997). Tracking in mathematics and science: Courses and course selection procedures. Sociology of Education, 70(4), 108-127.

Columba, L., & Dolgos, K. A. (1996). Integrating mathematics and science through bubble exploration. 1996 Pennsylvania Council of Teachers of Mathematics Yearbook, Integrated Mathematics and Integrating Mathematics, 1-6.

Columba, L., & Dolgos, K. A. (1996). Building bridges through fraction language. (Reprinted from the New England Mathematics Journal). Virginia Mathematics Teacher, 23(1), 15-17.

Columba, L., & Dolgos, K. A. (1995). Exploring with Geometric Solids. 1995 Pennsylvania Council of Teachers of Mathematics Yearbook, Teaching Mathematics with a Focus on Learning, 39-44.

Columba, L., & Dolgos, K. A. (1995). Building bridges through fraction language. New England Mathematics Journal, 28(1), 48-55.

29

C. Five Other Significant Publications

Columba, L., & Dolgos, K. A. (1995). Portfolio assessment in mathematics. Reading Improvement, 32(3), 174-176.

Columba, L., & Dolgos, K. A. (1994). Going the distance: Interpreting data. 1994 Pennsylvania Council of Teachers of Mathematics Yearbook, Mathematics for All Students, 90-98.

Dolgos, K. A., & Columba, L. (1993). Alternative assessment: The mathematics portfolio. 1993 Pennsylvania Council of Teachers of Mathematics Yearbook, Alternative Assessment in Mathematics, 23-28.

Columba, L., & Dolgos, K. A. (1993). Professional development for teachers of mathematics. Education, 113(4), 656-660.

Columba, L., & Dolgos, K. A. (1993). Tips for beginners: Daily quiz sheet. Mathematics Teacher, 86(5), 378-379.

D. Presentations Relevant to the Proposed Project INTERNATIONAL AND NATIONAL

Friendliness with Fractions. Extended Session. National Council of Teachers of Mathematics, Central Regional Meeting and School Science and Mathematics Annual Meeting, October, 1998, Louisville, KY.

Implementing the National Council of Teachers of Mathematics' Standards. A week of workshops at the International School of Manila, Philippines, October, 1996.

Mathematics Assessment. What? How? A week of workshops at the International School of Bangkok, Thailand, March, 1996.

Exploring Content Subject Literature in Literacy Methods Courses, Integrating Content Instruction in the College Classroom. A panel with G. Smith and P. Richards, Symposium at the Annual of the Association of Supervision and Curriculum Development, March, 1995, New Orleans, LA.

Teaching Reading in the Content Areas. A panel with G. Smith, J. Simmons, S. McKee, & B. Grugel, International Reading Association Annual Meeting, Anaheim, CA, May, 1995.

Action in Fractions - Hands-on Thinking Tools. National Council of Teachers of Mathematics, Annual Meeting, Indianapolis, IN, April, 1994.

Multiple Perspectives of Computer-Assisted Instruction in the Elementary Classroom. Co-presented with G. Smith, INFOCOMM and Association of Educational Communications and Technology, Annual meeting, New Orleans, LA, January, 1993.

How Computers Change the Learning Environment. Co-presented with G. Smith, Association of Supervision and Curriculum Development, Annual Meeting, Washington, D.C., March, 1993.

The Action in Fractions - Hands-on Thinking Tools. NCTM Canadian Regional Conference, Montreal, Quebec, August, 1992.

E. Graduate Students Advised Over the Past Five Years Ballard, Connie. Coyle, Carolyn Hanlon, Sr. Martha. Hartman, Kathryn Kistler, Barbara Barno, Richard Kramer, Pamela A. Kolodziej, Nancy Bray, Jane Dech, Kathryn Grigoruk, Melissa Cueny, Mary Ellen Iskold, Lubov Fields, Allen

30

Dr. M. Jean Russo Research Scientist Phone: 610-758-3803 Center for Social Research Fax: 610-758-6350 Lehigh University Email: mjr6@lehigh.edu Price Hall, 681 Taylor Street Bethlehem, Pennsylvania 18015 Education Ph.D. in Applied Social Research Lehigh University, 1992. M.A. in Social Relations Lehigh University, 1988. B.A. in Social Work Moravian College, 1984. Professional Experience As a Research Scientist and Scholar at the Center for Social Research at Lehigh University, Dr. M. Jean Russo has been involved in numerous evaluation and research projects. Presently, she is conducting evaluations of academic programs supported by the National Science Foundation, the U. S. Department of Labor, and the U. S. Department of Education. Dr. Russo has experience with all aspects of the evaluation, including evaluation design, questionnaire development and testing, data collection and management, data analysis, and preparation of evaluation reports. Along with various evaluation activities, she has served for the past ten years as a consultant on the Family Study conducted at Lehigh University by R. C. Herrenkohl and E. C. Herrenkohl. In this capacity, she performed data analysis and co-authored several journal articles.

Dr. Russo is currently serving as evaluator for Lehigh University�s Global Manufacturing Logistics Doctoral Program sponsored by the NSF Integrative Graduate Education and Research Training (IGERT) Program. In the past, she was evaluator for three NSF Industry/University Cooperative Research Centers (I/UCRCs) located at Lehigh University. Other evaluation and research activities were supported by various public and private organizations. Among them are: the William and Flora Hewlett Foundation Educational Program, and the U. S. Department of Health and Human Services Administration on Aging, the U.S. Department of Health and Human Services National Center of Child Abuse and Neglect, the National Institute of Mental Health, the National Institute of Standards and Technology, the Office of Naval Research, the U. S. Department of Commerce.

Dr. Russo served as editor and contributing author on a two-volume series that included selections of the important works of Donald T. Campbell, a preeminent scholar in the social science community. In the past, she taught graduate courses in Social Research Methods for the College of Education at Lehigh University and undergraduate courses at Cedar Crest College. She is a member of the American Evaluation Association and the Eastern Evaluation Association.

31

With Others (2001). Social Measurement. Thousand Oaks, CA: Sage Publications. (2001). Abusive Early Child Rearing and Early Childhood Aggression. Child Maltreatment, 6,

3-16. (2000). Benchmarking your R&D: Results from IRI/CIMS Annual R&D Survey for FY '98.

Research�Technology Management, 43,1. (1999). Social Experimentation. Thousand Oaks, CA: Sage Publications. (1999). Benchmarking your R&D: Results from IRI/CIMS Annual R&D Survey for FY '97.

Research�Technology Management, 43,1. (1998). Benchmarking your R&D: Results from IRI/CIMS Annual R&D Survey for FY '96 and

Some Interpretations. Research�Technology Management, 41,1. (1998). Using the IRI/CIMS R&D Database. Research�Technology Management, 41,

November-December, July-August, May-June. (1998). The impact of R&D on firm innovation performance: A comparative analysis between

Brazil and the United States. Presented at the 7th International Conference on Management of Technology, Orlando, Florida, February 16-20.

(1998). The relationship between early maltreatment and teenage parenthood. Journal of

Adolescence, 21, 3. (1997). Meet your competition: Results from the IRI/CIMS Annual R&D Survey for FY1995.

Research�Technology Management, 40,1. (1996). Meet your competition: Results form the 1994 IRI/CIMS Annual R&D Survey.

Research�Technology Management, 39,1. (1996). Why partner? Analysis of white papers submitted to the Advanced Technology Program.

U. S. Department of Commerce, Office of Technology Policy. (1991). Factors affecting the transfer of technology from industry/university cooperatives to

sponsoring companies. Journal of Technology Transfer, 15. (1990). Tracking the transfer of information from I/UCRCs to sponsoring companies. Final report to the National Science Foundation.