Microsoft Word - Renewal_Review_Protocol_Gen · Web viewDo not renew and phase-down support....

Engineering Research Centers ProgramDivision of Engineering Education and Centers

National Science Foundation

Third and Sixth Year Renewal Review Protocoland

Review Criteriafor

Engineering Research Centers (Gen-2)

National Science FoundationDivision of Engineering Education and Centers

4201 Wilson Boulevard, Suite 585Arlington, VA 22230

Telephone: (703)-292-8381

January 2015

ENGINEERING RESEARCH CENTERS PROGRAM THIRD AND SIXTH YEAR RENEWAL REVIEW PROCESS

I. GOALS AND KEY FEATURES OF THE ENGINEERING RESEARCH CENTERS PROGRAM

The goal of the Engineering Research Centers Program is to educate a globally competitive and diverse engineering workforce in an integrated, interdisciplinary research environment where academe and industry join in partnership to advance fundamental engineering knowledge and engineered systems.

All ERCs share the following key features: Long-term, strategic vision for a transforming or next-generation engineered system with the potential

to transform or significantly strengthen industry, the service sector, or infrastructure and society while increasing the involvement of women and underrepresented racial and ethnic minorities in the engineering and scientific workforce;

Strategic plan to deploy a diverse and cross-disciplinary team of faculty and students to realize the vision;

Cross-disciplinary research program to integrate fundamental and enabling technology research with proof-of-concept test beds designed to test theory in functioning systems;

Education program to team a diverse cadre of undergraduate and graduate students in cross-disciplinary, systems focused research and integrate research findings into curricular materials for students and practitioners;

Outreach in research and education to involve diverse cadres of college and pre-college students and their faculty and teachers in the ERC to stimulate interest in engineering research, motivate students to study engineering, and infuse engineering concepts into the pre-college classroom; and

Partnership with industry and other practitioners to formulate, evolve, and strengthen the ERC and speed technology transfer.

ERCs require the following resources: Integrated institutional configuration that is appropriate to achieve the ERC’s goals; A Director, Deputy Director, and other members of a leadership team to guide and manage the center; A core, cross-disciplinary diverse team of faculty committed to integrating their skills to achieve the

ERC's vision and goals; Leadership, faculty, and student teams that are diverse in gender, race, and ethnicity; Diversity strategic plan with goals, intended actions, milestones, outcomes, and impacts that are

exceeding national averages in engineering; A management system to organize and deploy the center’s resources to achieve its goals and secure

external advice on strategic directions and project selection and assessment; Experimental, computational, and other equipment, facilities, and laboratory space; headquarters space

to foster interdisciplinary collaboration and support center-level activities; Academic cash cost sharing plus cash and in-kind support from industry and other sources to leverage

substantially NSF's support and sustain it after NSF support ceases; and Institutional commitment to facilitate and foster the culture and diversity of the ERC.

II. LIFE-CYCLE OF AN ERC UNDER NSF SUPPORT

An Engineering Research Center begins operation with a five-year award under a cooperative agreement with the NSF. The agreement has the potential to extend NSF support to a maximum of 10 years. After that point, the ERC is expected to be self-sustaining. The progress and future plans of each ERC are assessed

annually through merit review by outside experts, with the first renewal review in the third year of operation. A successful third-year renewal review extends the cooperative agreement for three more years to year eight, with a second renewal review in the sixth year. A successful sixth-year renewal review extends the cooperative agreement for another two years to complete the maximum term of NSF support. A period of phased-down support is provided to an ERC that is not renewed. If an ERC is performing so poorly that it has lost its focus and character as an ERC, an annual review may result in NSF phasing out support to the ERC over one or two years.

III. THIRD AND SIXTH-YEAR RENEWAL REVIEW PROCESS

A team of peers of the ERC and NSF staff conducts the third and sixth-year renewal reviews. Prior to the visit, the members of the site visit team review the center’s renewal proposal using the ERC Program’s renewal review criteria. The review criteria are qualitative statements intended to describe expected characteristics of the ERC based on the required key features described above. They can be found in Section IX, ERC Performance Review Criteria, below. The review is guided by an overall analysis of the major strengths (S), weaknesses (W), opportunities (O) for new directions, and any serious weaknesses that threaten (T) the success of the center if they are not or cannot be addressed, a SWOT analysis.

The renewal recommendation options are as follows: Renew as proposed; Renew pending approval of a strategy to address a significant weakness or threat that must be

corrected before the ERC can be recommended for renewal by the site visit team; or Do not renew and phase-down support.

The site visit report contains an overall analysis of the technical merit and broader impacts of the ERC, the review team’s SWOT analysis, an assessment of the quality of the progress and plans in each ERC key feature, and a recommendation regarding renewal. The report is prepared on site immediately after the completion of review.

If the recommendation is to renew pending approval of an addendum to the renewal proposal, the ERC’s leadership will be given a brief period to prepare the addendum. Upon completion, NSF will forward the addendum to the site visit review team members for review and determination of the final recommendation. A site visit review team's recommendation is advisory to the ERC Program Director (PD) who uses it as a basis for a recommendation to the ERC Program Leader. Any recommendation for renewal requires the approval of the Director of the Division of Engineering Education and Centers and the Assistant Director for Engineering. If the budget is above certain thresholds, approval by the Director of NSF and the National Science Board also may be required.

IV. THE SITE-VISIT REVIEW TEAM

The members of the site visit review team are selected by the ERC's Program Director, with the approval of Program and Division management. The ERC Director may submit names of potential reviewers to the ERC's PD. The site visit review team will include members with the required scientific and technical expertise plus experience with cross-disciplinary research, engineering education, industrial research and development, technology transfer, and research management. Site visitor review team members are drawn from academe, industry, and government; are diverse in race, gender, and ethnicity; and most team members will have participated in a review of the center in prior years. NSF conflict-of-interest guidelines govern the reviewer selection. The ERC's Program Director leads the site visit, with the assistance of the other NSF staff. References in this document to the site visit team refer to the site visit review team plus the

NSF staff.

V. REVIEW MATERIALS The site visit team members are provided with the following materials: the ERC Program's FY 2013 Third and Sixth-year Renewal Protocol and Review Criteria (this document); for third-year renewals, the ERC's renewal proposal summarizing its progress and plans for the next five years (in electronic or hard copy format) and for sixth-year renewal, the ERC’s renewal proposal summarizing its progress and plans for the next four years (in electronic or hard copy format); and the annotated site visit reports from the prior two site visits. Information drawn from the site visit briefings and discussions supplements this information to provide the basis for the renewal recommendation.

VI. SITE VISIT

The duration of the site visit review is three days. On the night before the visit, the site visitors are briefed on the ERC program’s goals and its renewal review process by the NSF Program staff. The first two days of the visit are focused on presentations by the ERC team about its progress and plans. The agenda provides time for dialog between the site visit team and the leadership team of the ERC, its faculty, students, industrial partners, and university administrators. The agenda also includes executive sessions during which site visit review team members discuss their findings. At the end of the second day, the site visit review team presents any issues needing further clarification to the ERC leadership team. The site visit team meets the next morning with key members of the ERC's leadership team to discuss these issues. The site visit review team then carries out an analysis of the quality of the ERC, determines its recommendation, and writes its report. The site visit report is the primary input to NSF on the quality of the ERC's accomplishments and proposed plans and it contains the site visit review team's recommendation for renewal or termination. The report is the result of extensive discussion among site visit review team members and is a consensus document.

VII. SITE VISIT PRESENTATIONS

The presentations and discussions during the site visit allow the site visitors to calibrate the accomplishments and plans of the ERC against related efforts and the ERC review criteria. The presentations should convey an overall command of the field upon which the ERC is focused and provide information pertinent to the review criteria. The ERCs have been instructed to provide ample time for questions so that the site visitors can probe the intellectual foundations of the research, education, outreach, and technology transfer, and infrastructure accomplishments and plans in order to resolve any ambiguities. Presentations should be prepared to move quickly past problem definition and illustrate the motivation for and rigor of the approaches taken with specific examples. In the ideal case, many of these illustrations will be insights, mechanisms, discoveries, or accomplishments that are unique to the ERC and are recognized as leading the field.

The strategic planning and research presentations should be highly selective and present only key information supporting the critical path detailed in the current research plan. They should specify progress-to-date, current thinking, and plans with regard to:

The fundamental scientific insights motivating the ERC and their foundations in the work of researchers in and outside of the ERC;

The enabling and systems technology opportunities presented by these insights in relation to current practice and the state of the art;

The key scientific and/or technical barriers to realizing these technologies and the rationale for

believing that no critical barriers remain unidentified; The specific approaches being taken to overcome the barriers and criteria for determining success in

each; The role of the test beds in integrating the research and exploring and proving enabling and systems

level technologies; The major potential limitations of the proposed approaches (show stoppers); and potential alternative

approaches that may be used if the preferred approaches prove unsuccessful; Accomplishments and their impacts so far; Key resources within the core faculty of the ERC, supplemented on an as-needed basis by outreach to

faculty at other institutions.

The education and outreach briefings should explain how the specific elements of the research program are foci for pre-college, university, and practitioner-level courses and course materials, and any ERC-fostered degree programs. They should illustrate the Center’s implementation approach using actual examples.

The industrial/practitioner collaboration/technology transfer briefings should provide the rationale for the composition of the membership, the progress in membership to date, the modes of industrial involvement. It will provide information on the selection of technology targets, given the state of the art and the ERC's research, and the impact of ERC-developed technology on industry and practice achieved so far.

The infrastructure briefings should include information on the leadership, management, and resources of the ERC and on plans for self-sufficiency once NSF support ceases. The ERC will present its strategy to increase the gender, racial, and ethnic diversity of the ERC’s leadership team, faculty, students at all levels, actions taken, milestones, results to date benchmarked against national engineering-wide averages, and plans for the future.

VIII. ERC ANNUAL AND RENEWAL REVIEW JUDGMENTS VIS-A-VIS THE CHALLENGES INHERENT IN THE ERC, ITS STAGE OF DEVELOPMENT, AND THE REVIEW CRITERIA

The systems vision, the strategic plan, and the research program form the intellectual structure from which all other activities of the ERC derive. Therefore, a well-conceived and effectively conducted strategic plan and research program are a prerequisite for a successful ERC. An effective research program will be complemented by equally effective mechanisms for building and sustaining industrial/practitioner collaboration, transitioning key results into industrially relevant technologies and building innovative education and outreach programs based on the ERC’s research. All elements should be presented in the context of evolving the academic culture of research and education to embrace a cross-disciplinary and team-based approach that is mindful of industrial practice and focused on outcomes and impacts. The ERC’s leadership team, management systems, and other elements of its infrastructure are designed and operated to provide the support systems needed for an effective ERC. The ERC team is diverse in gender, race, and ethnicity and represents a multicultural mix of participants from the U.S. and abroad.

Engineering Research Centers are required to embrace the cultural, gender, racial, and ethnic diversity of the U.S. in the composition of their leadership, faculty, and student teams and their graduates in order to assure that all talented people in our country can pursue and receive engineering degrees and be engaged in engineering research and education. NSF expects the faculty and staff of all ERCs and the administrations of all institutions receiving NSF funding to share this commitment and to devote the time and effort required to ensure that the diversity of the center’s leadership team, faculty, and students at all levels serves as a model for diversity within each institution and for the nation as a whole. This expectation is made with

the understanding by NSF that ERCs do not have the authority to hire faculty, accept students, or grant degrees. While a center’s diversity strategic plan cannot by fiat include quantitative targets, each ERC is expected to benchmark its progress against national averages for the involvement of women, underrepresented racial minorities (African Americans, Pacific Islanders, Native Americans), underrepresented ethnic minorities (Hispanic Americans), and persons with disabilities in engineering as faculty, as undergraduate, masters, and doctoral students, and post-doctorate students. Since no set of formal requirements can ensure that a desired level of dedication to achieving diversity is engendered, the success depends in large part on the spirit in which the diversity strategy is that implemented by the center and its collaborating departments and university-level schools.

Final judgments on the quality of the overall achievements of an ERC will depend upon the stage of development of the ERC, the perceived difficulty of integrating the disciplines involved in a particular ERC, the degree of sophistication of the targeted industrial/practitioner community in transforming the research results into technological innovations, the relative need for educational innovations in the field, and the degree of difficulty in attracting and retaining women and underrepresented minorities to the field upon which the ERC is focused.

Judgments will be drawn about the degree to which the ERC meets these criteria depending upon the age of the center. Reviewers of a start-up ERC in its first year understand that the ERC has been functioning for only about eight months; and therefore, do not expect a fully structured ERC nor should they expect significant outcomes and impacts. Rather they use the criteria to determine how well the team is positioned to begin to fulfill its vision. Thus ERCs in their first year are putting the required ERC features in place for research, education, industrial collaboration, diversity, leadership, management, and an infrastructure including headquarters space to support the ERC. The membership agreement and intellectual property policy are in place, effectively structuring a team of industrial partners for active involvement and benefit. Collaborative activities are starting up in research and education with predominantly female and underrepresented minority serving institutions, at least one of the NSF-supported Louis Stokes Alliance for Minority Participation (LSAMP), and with one or more awardee(s) of the NSF-supported Alliance for Graduate Education of the Professoriate (AGEP), and a university serving Native Americans or person with disabilities These collaborations will involve diverse cadres of students and faculty in the ERC’s research and education programs. The diversity strategy is positioned to produce results in the gender, racial, and ethnic diversity of the ERC and its graduates benchmarked against national engineering-wide averages. The leadership team is in place and learning how to implement their vision of an effective ERC and the management and oversight systems will support a strong ERC. Financial support is building from non-NSF sources. The university administrators from the lead and any core partner institutions are supportive with space and policies to facilitate interdisciplinary research, industrial collaboration, and diversity. There is an effective partnership building in research, education, and industrial collaboration across the lead and core partner institutions.

ERCs in their second and third years have the ERC features in place and are transforming their style of operation from individual and single discipline efforts to interdisciplinary and interdependent team efforts. They are learning strategic planning and how to operate with a shared vision. The systems issues are generally in the formulation stage but must be driving the ERC’s strategic plan and research program. Industrial collaboration is beginning to become effective; there may be some technology transferred to industry. The education program is starting to provide a new culture for its students based on the ERC key features and building on the research outcomes and integrating them into the curriculum. The education outreach programs are under way at the undergraduate and pre-college levels. At least one systems level course is planned or in place. The collaborations with female and underrepresented minority serving

institutions, at least one of the NSF-supported LSAMPs, and with one or more awardee(s) of the NSF-supported AGEPs, tribal colleges, etc. involve a diverse cadre of faculty and students in the ERC’s research and education programs. An infrastructure of space and equipment supports the ERC and the headquarters adequately house the ERC and contribute to collaboration. A sound diversity plan has been developed, which is beginning to show impacts on the gender, racial, and ethnic diversity of the leadership, faculty, student teams at all levels, and ERC graduates that are benchmarked against national engineering-wide averages. The team of faculty and students is beginning to work in synergy. The leadership team is becoming effective in implementing their vision of an effective ERC and the management and oversight systems are sound. Financial support is building from non-NSF sources. The university administrators from the lead and any core partner institutions have provided space for the ERC and are supportive with policies to facilitate interdisciplinary research, industrial collaboration, and diversity. There is an effective partnership in research, education, and industrial collaboration across the lead and core partner institutions.

ERCs that have passed their third year renewal review enter a period where the groundwork that was laid out in the first three years begins to pay off. There is an effective strategic planning process that focuses and integrates resources to achieve systems goals through an integrated cross-disciplinary research program that is delivering advances in knowledge and technology enabled by test beds. The ERC's research is impacting the curriculum at the course level and some ERCs may be developing new degree programs or options. At least one systems level course is in place. Educational outreach programs are showing an impact in attracting K-12 students to study engineering, and undergraduate students to graduate work. Engineering concepts derived from the ERC’s research are in use in the pre-college classroom because of involvement of K-12 teachers in the ERC. The collaborations with female and underrepresented minority serving institutions, at least one of the NSF-supported LSAMPs, and with one or more awardee(s) of the NSF-supported AGEPs, tribal colleges, etc. involve a diverse group of faculty and students in the ERC’s research and education programs. Effective and highly collaborative relationships with industry/practitioners are under way in research and education and the ERC's research is having a strong impact on these partners. The infrastructure of equipment and space is in place and productive and the headquarters space is conducive to interdisciplinary collaboration. A sound diversity plan has been developed, which is yielding impacts on the gender, racial, and ethnic diversity of the leadership, faculty, and student teams at all levels, and ERC graduates that are benchmarked against national engineering-wide averages. The team is becoming more interdependent and productive due to collaboration. The leadership team is effective, management and oversight systems are in place and effective; the financial support is strong. Starting in year 5, there is a plan for financial self-sufficiency after ERC program support ceases. The university administrators from the lead and any core partner institutions have provided space for the ERC and are supportive with policies to facilitate interdisciplinary research, industrial collaboration, and diversity. There is a strong partnership in research, education, and industrial collaboration among the lead and core partner institutions. In summary, there is full implementation of all the ERC key features that continues to strengthen through the sixth-year renewal review.

After a successful sixth year renewal, the ERC is restructuring its strategic plan to open new avenues of research to stay ahead of the field and finding systems-level research paying off through test beds and technology focused projects. Curricular impacts are broad and pervasive in the lead and core partner university involved and beyond. Educational outreach programs are extensive and showing a significant impact on attracting K-12 students to engineering, on the pre-college classroom through involvement of teachers in the ERC, and on attracting undergraduates to graduate work. The collaborations with underrepresented female and minority serving institutions, at least one of the NSF-supported LSAMPs, and with one or more awardee(s) of the NSF-supported AGEPs, tribal colleges, etc. involve a diverse group of faculty and students in the ERC’s research and education programs. Industrial collaboration should be

broadening and intensifying as the ERC prepares for self-sufficiency. This may necessarily increase the number of directed projects as the ERC nears graduation. A sound diversity plan has been developed, which is yielding impacts on the gender, racial, and ethnic diversity of the leadership, faculty, student teams at all levels, and ERC graduates that are benchmarked national engineering-wide averages. The infrastructure supports the ERC effectively in terms of leadership, team, management systems, and financial support. The team is cohesive and productive due to interdisciplinary collaboration in research and education. The lead and any core partner universities are supportive with policies and space to facilitate interdisciplinary and industrial collaboration and help the ERC achieve its diversity goals. Well-developed business plans are in place for self-sufficiency after ERC Program support ceases. The partnership across the universities involved continues to be strong.

IX. ERC PERFORMANCE REVIEW CRITERIA

GEN-2 PERFORMANCE CRITERIA

High Quality Vision & Value Added (Years 1-3)

Low Quality Vision and Value Added (Years 1-3)

Systems Motivation: Strong systems vision motivates the ERC, early systems requirements understood

Systems Motivation: Little understanding of engineered systems

Transformational: Vision has potential to transform or significantly impact industry/practitioners, the workforce, and society

Transformational: Losing sight of the promise of the vision and its potential impact

Leading-edge: Vision positions the ERC to lead in the field

Leading-edge: ERC lags the state of the art or is already eclipsed by competitors

High Quality Research: Research output is high quality, some deriving from cross-disciplinary collaboration, publications based on ERC research in process

High Quality Research: Research output is low quality; or if high quality, it resembles the output of a collection of single investigator projects

Industrial Relevance: Some high quality research advances moving into use, most likely to be useful in a few years

Industrial Relevance: Low probability of impact of the research on industry and practice or, if output is moving into industry, it is low quality or low impact

Educational Impact: ERC research impacting courses and, if part of the strategic plan, impacting curriculum development

Educational Impact: Little or no impact on courses or curriculum

High Quality Strategic Plan (Years 1-3) Low Quality Strategic Plan (Years 1-3)

Systems: Systems concepts and technology goals drive and integrate all levels of research

Systems: The strategic research plan is not motivated by systems concepts and technology goals

Research Integration: Research organized into well-integrated thrusts designed to achieve the vision and redirected as necessary

Research Integration: Thrust have little relationship to each other or the vision; ineffective thrusts have not been redirected or terminated

Barriers: Strategic plan focuses on significant barriers and challenges that position the research to lead the field and

Barriers: Barriers and challenges are not significant and will not result in contributions that will lead the field

advance the state of the art

Test-beds: Test beds or test bed plans provide a significant opportunity to integrate the research thrusts to explore and prove enabling and systems level technologies

Test-beds: No evidence of test beds in plans or test beds appear to be demonstrations, isolated from research thrusts or the strategic plan

Cross-disciplinary: The team is appropriately cross-disciplinary with necessary disciplines and sub-disciplines

Cross-disciplinary: The team is not appropriately cross-disciplinary, necessary disciplines or sub-disciplines are missing

High Quality Research Program (Thrust Level) (Years 1-3)

Low Quality Research Program (Thrust Level) (Years 1-3)

Contribution: Thrust and its projects designed to contribute to the goals and vision of the ERC

Contribution: Thrust has little relevance to the goals and vision of the ERC

Interdependence: Projects are appropriately cross-disciplinary and becoming integrated, growing interdependence of projects within the thrust, appropriate interdependence among thrusts beginning

Interdependence: Thrust resembles a collection of single investigator projects working in isolation

Methodology: Significant research barriers/challenges being addressed through high quality research methods

Methodology: Research barriers/challenges are not significant and research methods are not advancing the state of the art

Project Selection: Projects are appropriate to fulfill thrust goals; decisions based on external input when needed, and sufficiently funded; weak or inappropriate projects are terminated

Project Selection: Projects are not appropriate to fulfill thrust goals; decisions are not based on external input when needed, and are not sufficiently funded; weak or inappropriate projects are not terminated

Team Dynamics: Thrust team is becoming cohesive; opportunities for cross-institutional collaboration being pursued

Team Dynamics: Faculty no cohesive and/or opportunities for cross-institutional collaboration not pursued

Significant Results: Positioned to or beginning to deliver results that are unique in the field, high quality publications in process, some results due to cross-disciplinary collaboration and insight

Significant Results: Results do not appear to be unique in the field, could have been achieved by a collection of individual projects

Technology Transfer: Research and technology advances beginning to transfer to industry/practitioners

Technology Transfer: Little interest on the part of industry/practitioners in the outcome of the research

High Quality Education and Educational Outreach (Years 1-3)

Low Quality Education and Educational Outreach (Years 1-3)

Culture: Cross-disciplinary, cross-institutional education culture is developing

Culture: Little or no cross-disciplinary, cross-institutional, and/or team-based interaction on the part of faculty and students

Teaming: Undergraduate and graduate students are starting to work in teams; significant commitment to involvement of undergraduates in research (ratio of graduate to undergraduate students of approaching 2:1)

Teaming: Minimal involvement of undergraduates in research

Curricula: Research results being incorporated in existing courses, and there are specific plans for programs/options

Curricula: Few if any research results are being integrated into courses for students and/or practitioners, little or no activity related to developing any proposed degree programs/options

Assessment: Strong plans in place to assess and disseminate new education programs and curricular materials

Assessment: Evaluation/assessment plans poor or they do not exist; personnel involved lack appropriate background for the task

Systems Training: Specific plans in place for formal student training in systems concepts

Systems Training: Students have little or no awareness of systems issues, no formal training planned

Industry Interaction: Students have ample opportunities to work with industry. practitioners in internships or on sponsored projects

Industry Interaction: Few of any students work with practitioners at the ERC or in industry

Outreach: Collaborations with female/minority serving partners are effective in increasing ERC's diversity in ERC's education programs

Outreach: No or ineffective diversity partnerships in education, no commitment to diversity

REU: ERC provides funding to support Research Experiences for Undergraduates. Program provides non-ERC students with an ERC research experience and focuses on underrepresented groups

REU: Precollege outreach programs are nonexistent, inappropriate or disconnected from the ERC's research and education programs, or no emphasis on diversity

K-12 Outreach and RET: Pre-college outreach includes a Research Experiences for Teachers program and involves K-12 students with an emphasis on increasing the participation of underrepresented groups in engineering

K-12 Outreach and RET: Education activities in multi-institutional ERCs are not coordinated and are not impacting all the core partner institutions

Cross-institutional Educational Opportunities: A partnership in education among the lead and core partner institutions involves all core partners and impacts their ERC students

Cross-institutional Educational Opportunities: Education activities between ERC partners are not coordinated and are not impacting all the core partner institutions

High Quality Industrial/Practitioner Collaboration & Technology Transfer (Years 1-3)

Low Quality Industrial/Practitioner Collaboration & Technology Transfer (Years 1-3)

Membership Agreement: Center-wide membership agreement structures the industry collaboration program with clear statements of fees, benefits, and intellectual property policies

Membership Agreement: Membership agreement not in place; Center IP policies deter industry membership or technology transfer

Membership: Growing or stable group of members across sectors and throughout the supply chain appropriate for the ERC's vision, key players have joined by the third year or are in the process of joining

Membership: Membership promise of proposal not fulfilled, many of those committed or promising to commit did not sign up, significant numbers of firms/agencies leaving, and/or major sectors are missing

IAB: Industrial Advisory Board (IAB) active and effective and SWOT process yielding cogent advice to the ERC

IAB: IAB rarely meets, SWOT process not in place or outcome ignored

Industry Integration: Industrial collaboration beginning to achieve a cooperative partnership that is integrated into the ERC's planning, research, and education activities

Industry Integration: Industry involved only on a project-by-project basis, no collective, collaborative partnership

Membership Fees: Membership fees provide discretionary funds for the ERC, commensurate with typical investments in academic R&D for the sectors represented by the firms involved

Membership Fees: Low level of membership cash support for discretionary fund

Technology Transfer: Knowledge and technology transfer is starting to impact industry/practitioners

Technology Transfer: Little knowledge or technology transfer has occurred, the center has had little impact on industry/practitioner

High Quality Infrastructure (Years 1-3) Low Quality Infrastructure (Years 1-3)

High Quality Configuration & Leadership Effort (Years 1-3)

Low Quality Configuration & Leadership Effort (Years 1-3)

Institutional Configuration: Appropriately integrated institutional configuration among lead, core partner, and outreach institutions, partnership beginning to emerge

Institutional Configuration: Individual center institutions operating mostly independently of each other

Center Director/Deputy Director: Effective Center Director and Deputy Director, able to implement vision and provide leadership.

Center Director/Deputy Director: Center Director and/or Deputy Director have not translated vision into operation, leadership skills of one or both are not up to the task

Leadership Team: Other members of the leadership team (research thrusts, education, industrial collaboration, and administration) are becoming cohesive and effective in planning and implementing the research, education, industrial collaboration, and administrative aspects of the ERC

Leadership Team: Some or all of these leaders are not effective and there are no plans to replace them

Research Team: High quality research team shares the vision, appropriate mix of expertise (faculty, practitioners, and students) to fulfill vision and systems goals

Research Team: Research team does not share the vision, is low quality, or does not have the mix of expertise necessary to fulfill the vision and systems goals

Student Leadership Council: Student Leadership Council (SLC) in place, starting to effectively lead student programs, SWOT process starting to impact ERC, ERC leaders receptive to the SLC's recommendations for improvement, SLC has adequate resources to achieve its goals

Student Leadership Council: Student Leadership Council is not effective, doesn't use SWOT analysis

High Quality Diversity Effort (Years 1-3) Low Quality Diversity Effort (Years 1-3)

Diversity Strategic Plan: Strong strategic plan for diversity in place benchmarked against national engineering averages over the award period, results beginning to demonstrate effectiveness of plan. Strong partnership for diversity with partner deans and department chairs.

Diversity Strategic Plan: No strategic plan for diversity in place; no evidence of commitment to diversity. No partnership for diversity with administrators. Results demonstrate ineffective plan or effort.

Minority Serving Institution Interactions: In research and education, ERC involves at least one minority or female serving institution , a Louis Stokes Alliance for Minority Participation (LSAMP), and has at least one other connection with an Alliance for Graduation Education of the Professoriate (AGEP), or at institutions that involved Native Americans, etc.

Minority Serving Institution Interactions: No effort or unsuccessful efforts to involve institutions / NSF diversity awardees that serve women and other underrepresented groups

Leadership Diversity: Team of leaders is diverse in gender, race, and ethnicity

Leadership Diversity: Little or no commitment to diversity at the leadership level

Women Faculty Involvement: A significant number of women faculty involved, active recruitment continues

Women Faculty Involvement: None or a small number of women faculty involved since center inception even though candidates available

Underrepresented Minority Faculty Involvement: A significant number of underrepresented minority faculty involved, active recruitment continues

Underrepresented Minority Faculty Involvement: None or a small number of underrepresented minority faculty involved even though candidates available

Underrepresented Students: Graduate and undergraduate women and underrepresented minority students broadly involved in center activities

Underrepresented Students: Few or no women and underrepresented minority students appear to be involved in center activities

Involvement of Persons with Disabilities: Efforts are underway to increase the involvement of persons with disabilities at all levels and provide them with appropriate support/access to carry out their work

Involvement of Persons with Disabilities: Little understanding of how to attract and recruit persons with disabilities to the ERC or if they are there, they have poor support and access to carry out their work

High Quality Management Effort (Years 1-3) Low Quality Management Effort (Years 1-3)

Management Systems: Effective management systems, goals are set and met or revised, effective use of performance indicators to track and improve performance

Management Systems: Management systems weak, poor goal setting and delivery, management ignores indicators of poor performance

Use of Financial Resources: Effective use of financial resources to achieve the ERC's goals, thrust and institution level budgets are appropriate for their roles in the ERC, timely allocation of funds; after year one, any residuals should be below 20% of NSF support

Use of Financial Resources: Allocation of resources not commensurate with achieving the ERC's goals, long delays in allocation of funds; after year one, any annual residuals are significantly greater than 20% of NSF

Outside Input: Effective incorporation of outside input in planning, project review, and assessment after year one, any annual residuals should be below 20% of NSF support

Outside Input: Planning and project review are conducted mostly or exclusively within the ERC and minimal outside input or outside input is ignored

High Quality Resources & University Commitment (Years 1-3)

Low Quality Resources and University Commitment (Years 1-3)

Equipment/Facilities: High quality experimental and enabling equipment/facilities; test beds under development according to strategic research plan

Equipment/Facilities: Experimental and/or enabling equipment/facilities lack critical components, are not state-of the art, or test bed development not proceeding in accordance with strategic research plan

Communications Capability: Headquarters and communications network facilitate interaction among students, faculty, industry/users and participating institutions

Communications Capability: Headquarters and communications network are ineffective or don't serve to serve to integrate the faculty and students

University Administration: Effective partnership with university administration facilitates the success of the Center through policies that encourage its cross-disciplinary configuration, diversity, and the partnership with industry; deans and department heads committed to success

University Administration: University administration does not facilitate the cross-disciplinary configuration, diversity, or industrial partnership of the Center, deans and department heads not addressing identified weaknesses of Center

Funds: Funds provided by industry/users, university, and other non-NSF sources are commensurate with their ability to contribute and benefit

Funds: Support levels for most or all sectors are below what would be expected

High Quality Systems Vision & Value Added

(Years 4-6)

Low Quality Systems Vision & Value Added

(Years 4-6)

Systems Motivation: Strong transformational systems vision is fully operational as a motivator for the ERC, systems requirements understood, vision is evolving as appropriate

Systems Motivation: Systems vision does not motivate the ERC or it has been achieved already; no compelling challenges remaining

Transformational: Vision is transforming or significantly impacting industry/practitioners, the workforce, and society

Transformational: Losing sight of the promise of the vision and its

Leading-edge: Center is recognized as one of the leaders in the field because of its cross-disciplinary, systems level vision and significant output

Leading-edge: Center is behind leaders in the field and center contributions are rarely recognized by the field as significant potential impact

High Quality Research: Research output is high quality and largely derived from cross-disciplinary collaboration, extensive cross-disciplinary publications in important journals


Industrial Relevance: ERC is producing broad-based and unique impact on technology (inventions, licenses, technology in use in industry or other arenas)

Industrial Relevance: ERC has largely failed to impact technology and practice

Educational Impact: Significant course and curriculum impacts derived from the ERC's research

Educational Impact: Curricular impact is minimal or could have been achieved without the ERC

High Quality Strategic Research Plan (Years 4-6) Low Quality Strategic Research Plan (Years 4-6)

Systems: Systems and technology goals have matured and integrated all levels of research, and will continue to evolve appropriately

Systems: Systems and technology goals have not matured or evolved as necessary; research levels are not integrated

Research Integration: Research effectively organized into well integrated thrusts that contribute to the vision, results being used within and across thrusts

Research Integration: Thrusts have little relationship to each other and the vision; ineffective thrusts have not been redirected or terminated

Barriers: Strategic plan focuses on remaining significant barriers and challenges, many initial barriers have been overcome, research leads the field and advances the state of the art

Barriers: Little progress toward overcoming barriers and challenges, identified barriers and challenges are not significant or relevant to the vision, research is lagging the field

Test-beds: Test beds effectively integrate the research to explore and prove enabling and systems level technologies

Test-beds: No test beds underway or are not integrated with the research thrusts

Cross-disciplinary: The team is appropriately cross-disciplinary with strong interdependence between disparate disciplines and sub-disciplines

Cross-disciplinary: Team is not sufficiently cross disciplinary, necessary disciplines or sub-disciplines are missing, no interdependence evident


Low Quality Research Program (Thrust Level) (Years 4-6)

Contribution: Thrust and its projects contribute significantly to the goals and vision of the ERC


Interdependence: Projects are appropriately cross-disciplinary and integrated, interdependence of projects within the thrust, robust interdependence among thrusts

Interdependence: Thrust resembles a collection of single investigator projects, most or all projects are isolated from one another, thrust is isolated from the others

Methodology: Prior and current significant research barriers/challenges effectively addressed through high quality research methods

Methodology: Research barriers/challenges are not significant or have not been effectively addressed; or research methods are not advancing the state of the art



Project Management: Research projects managed over time through different students and dissertations to achieve thrust and ERC deliverables

Project Management: Thrusts projects result in a collection of individual dissertations with little or no integration or synergy over time

Team Dynamics: Thrust team is cohesive, opportunities for cross-institutional collaboration effectively implemented

Team Dynamics: Faculty not cohesive and/or opportunities for cross-institutional collaboration not pursued

Significant Results: Delivering results that are unique in the field, high quality publications in journals important for the field, many results are due to cross-disciplinary collaboration and insight

Significant Results: Results do not appear to be unique in the field, could have been achieved by a collection of individual projects, minimal publications resulting from cross-disciplinary collaboration

Technology Transfer: Results significantly impacting industry/practitioners (patents, licenses, technology transferred to industry and practice)

Technology Transfer: Results are not advancing technology/processes/procedures for industry/practitioners

High Quality Education/Educational Outreach (Years 4-6)

Low Quality Education/Educational Outreach

(Years 4-6)

Culture: Cross-disciplinary, cross-institutional education culture in place

Culture: Little or no cross-disciplinary, cross-institutional, and/or team-based interaction on the part of faculty and students

Teaming: Undergraduate and graduate students work in teams; ratio of graduate to undergraduate students of approximately 2:1


Curricula: Research results continue to be incorporated in existing courses, and some new courses or degree programs/options are being implemented

Curricula: Few if any research results are being integrated into courses for students and/or practitioners, little or no activity related to implementing new courses or degree programs/options

Assessment: New education programs and curricular materials are being assessed and disseminated, including summative and formative evaluations

Assessment: New education programs and curricular materials have not been assessed; personnel involved lack appropriate background for the task

Systems Training: Students have opportunities for formal training in systems integration

Systems Training: Students have little or no awareness of systems issues, no formal training in place

Industry Interaction: Many students have had opportunities to work with industry/practitioners through internships or sponsored projects, several hired by member firms

Industry Interaction: Few if any students have worked or had opportunities to work with industry/practitioners

Outreach for Diversity: Collaborations with female/minority serving partners have resulted in progressively increasing diversity in ERC's education programs over time

Outreach for Diversity: No or ineffective collaborations with female/minority serving partners or no trends over time that increase diversity in ERC's education programs


REU: ERC failed to provide an REU program or the students involved in its REU program are not diverse

K-12 Outreach and RET: Pre-college outreach includes an effective Ret program and effectively involves K-12 students with an emphasis on increasing the participation of underrepresented groups in engineering

K-12 Outreach and RET: Pre-college outreach or RET programs are nonexistent, inappropriate or disconnected from the ERC's research and education programs, or there is no emphasis on increasing the participation of

underrepresented groups in engineering



High Quality Industrial/Practitioner Collaboration and Technology Transfer (Years 4-6)

Low Quality Industrial/Practitioner Collaboration and Technology Transfer (Years 4-6)

Membership Agreement: Center-wide membership agreement structures the industry collaboration program with clear statements of fees, benefits, and intellectual property policies that promote technology transfer


Membership: Growing or stable group of members across sectors and throughout the supply chain appropriate for the ERC's vision. Key players are members.

Membership: Membership promise of proposal not fulfilled, many of those committed or promising to commit did not sign up, significant numbers of firms/agency are leaving, and/or major sectors are missing

IAB: Industrial Advisory Board (IAB) active and effective; SWOT process yielding cogent advice to the ERC


Industry Integration: Industrial collaboration has become a cooperative partnership that is integrated into the ERC's planning, research, and education activities

Industry Integration: Industry involved only on a project-by-project basis, no collective, collaborative partnership

Membership Fees: Membership fees provide discretionary funds for the ERC and commensurate with typical investments in academic R&D for the sectors represented by the firms involved


Technology Transfer: Knowledge and technology transfer is impacting industry/practitioners




Low Quality Configuration & Leadership Effort Years (Years 4-6)

Institutional Configuration: Appropriately integrated institutional configuration among lead, core partner, and outreach institutions, partnership strong


Center Director/Deputy Director: Highly effective Center Director and Deputy Director, have implemented vision and are providing capable leadership for the ERC and the university

Center Director/Deputy Director: Center Director and/or Deputy Director have not translated vision into operation, leadership skills of one or both are not up to the task

Leadership Team: Other members of the leadership team Leadership Team: Some or all of these leaders are not

(research thrusts, education, industrial collaboration, SLC, and administration) are cohesive and effective in planning and implementing the research, education, industrial collaboration, and administrative aspects of the ERC

effective and there are no plans to replace them

Research Team: High quality integrated research team shares the vision, appropriate mix of expertise (faculty, practitioners, and students) to fulfill vision and systems goals

Research Team: Research team does not share the vision, is low quality, or does not have the mix of expertise necessary to fulfill the vision and systems goals, or operating independently

Student Leadership Council: Student Leadership Council (SLC) in place, is effectively leading student programs, SWOT process is impacting ERC, ERC leaders are receptive to the SLC's recommendations for improvement, SLC has adequate resources to achieve its goals

Student Leadership Council: Student Leadership Council is not effective, doesn't use SWOT analysis, ERC leaders not receptive to SLC recommendations

High Quality Diversity Effort (Years 4-6) Low Quality Diversity Effort (Years 4-6)

Diversity Strategic Plan: Strong strategic plan for diversity in place benchmarked against national engineering averages over the award period, results demonstrate a strong and effective plan. Strong partnership for diversity with partner deans and department chairs.

Diversity Strategic Plan: Ineffective strategic plan for diversity in place; no evidence of commitment to diversity. No partnership for diversity with administrators. Results demonstrate ineffective plan or effort.

Minority Serving Institution Interactions: In research and education, ERC involves at least one minority or female serving

Minority Serving Institution Interactions: No effort or unsuccessful efforts to involve institutions / NSF diversity awardees that serve women and other underrepresented groups

Leadership Diversity: Team of leaders is diverse in gender, race, and ethnicity institution , a Louis Stokes Alliance for Minority Participation (LSAMP), and at least one other connection with an Alliance for Graduation Education of the Professoriate (AGEP), or at institutions that involved Native Americans, etc.












Management Systems: Management systems weak, poor goal setting and delivery, management ignores indicators of poor performance

Use of Financial Resources: Effective use of financial resources to achieve the ERC's goals, thrust and institution level budgets are appropriate for their roles in the ERC, timely allocation of funds, any annual residuals are below 20% of NSF support

Use of Financial Resources: Allocation of resources not commensurate with achieving the ERC's goals, long delays in allocation of funds, any annual residuals are significantly greater than 20% of NSF support

Outside Input: Effective incorporation of outside input in planning, project review, and assessment


Post-graduation: By year 5, realistic and sound initial plan for financial self-sufficiency when NSF support ceases

Post-graduation: Weak plan for financial self-sufficiency when NSF support ceases



Equipment/Facilities: High quality experimental and enabling equipment/facilities; test beds operating according to strategic plan

Equipment/Facilities: Experimental and/or enabling equipment/facilities lack critical components, are not state-of the art, or test bed development is not proceeding according to plan

Communications Capability: Headquarters and communications network facilitate interaction among students, faculty, industry/users and participating institutions

Communications Capability: Headquarters and communications network are effectively non-existent

University Administration: Effective partnership with university administration facilitates the success of the Center through policies that encourage its cross-disciplinary configuration, its diversity, and its partnership with industry; deans and department heads committed to success

University Administration: University administration does not facilitate the cross-disciplinary configuration, diversity, or industrial partnership of the Center, deans and department heads not involved

Funds: Investment made by industry/users, university, and other non-NSF investors commensurate with their ability to contribute and benefit

Funds: Most or all sectors are below what would be expected

High Quality Strategic Vision and Value Added

(Years 7-10)

Low Quality Strategic Vision and Value Added

(Years 7-10)

Systems Motivation: Systems vision more challenging and evolving to sustain ERC past graduation

Systems Motivation: Systems vision has been lost or the systems vision has been fulfilled already

Transformational: Vision has transformed or significantly impacted industry/practitioners, the workforce and society

Transformational: Vision has resulted in little or no impact on industry/practitioners, the workforce, or society

Leading-edge: Center is recognized as one of the leaders in the field because of its cross-disciplinary, systems level vision and significant output

Leading-edge: Center is behind leaders in the field and center contributions are rarely recognized by the field as significant has been fulfilled already

High Quality Research: Research output is high quality and largely derived from cross-disciplinary collaboration, extensive cross-disciplinary publications in important journals


Industrial Relevance: ERC is producing broad-based and unique impact on technology (inventions, licenses, technology in use in industry or other arenas)

Industrial Relevance: ERC has largely failed to impact technology and practice

Educational Impact: Significant course and curriculum impacts derived from the ERC research

Educational Impact: Curricular impact is minimal and could have been achieved without the ERC

High Quality Strategic Research Plan (Years 7-10) Low Quality Strategic Research Plan (Years 7-10)

Systems: Systems and technology goals are mature and have integrated all levels of research, appropriate evolution continues

Systems: Systems and technology goals have not matured or evolved as necessary; research levels are not integrated.

Research Integration: Research effectively organized into well-integrated thrusts that contribute to the vision, results being used within and across thrusts, integrated thrusts continue to evolve to sustain the ERC past graduation

Research Integration: Thrusts have little relationship to each other and the vision; ineffective thrusts have not been redirected or terminated

Barriers: Strategic plan focuses on remaining significant barriers and challenges, initial barriers have been overcome, research leads the field and continues to advance the state of the art

Barriers: Limited progress toward overcoming barriers and challenges, remaining barriers and challenges are not significant or relevant to the vision, research is lagging the field

Test-beds: Test bed(s) operational, effectively integrating the research to explore and prove enabling and systems level technologies

Test-beds: No test beds operational or are not integrated with the research results

Cross-disciplinary: The team is appropriately cross-disciplinary with strong interdependence between disparate disciplines and sub-disciplines

Cross-disciplinary: Team is not sufficiently cross disciplinary, necessary disciplines or sub-disciplines are missing, no interdependence evident or opportunities have been missed because they did not achieve interdependence


Low Quality Research Program (Thrust Level)

(Years 7-10)

Contribution: Thrust and its projects contribute significantly to the goals and vision of the ERC


Interdependence: Projects are appropriately cross-disciplinary and integrated, interdependence of projects within the thrust, robust interdependence among thrusts

Interdependence: Thrust resembles a collection of single investigator projects, most or all projects are isolated from one another, thrust is isolated from others

Methodology: Prior and current significant research barriers/challenges effectively addressed through high quality research methods

Methodology: Research barriers/challenges are not significant or have not been effectively addressed; or research methods are not advancing the state of the art



Project Management: Research projects managed over time, and through NSF phase down and into self-suffiency, through different students and dissertations to achieve thrust and ERC deliverables

Project Management: Thrust projects result in a collection of individual dissertations with little or no integration or synergy over time, or no planning for NSF phase down or post-NSF support

Team Dynamics: Thrust team is cohesive, opportunities for cross-institutional collaboration effectively implemented

Team Dynamics: Faculty not cohesive and/or opportunities for cross-institutional collaboration not pursued

Significant Results: Delivering results that are unique in the field, high quality publications in journals important for the field, many results are due to cross-disciplinary collaboration and insight

Significant Results: Results do not appear to be unique in the field, could have been achieved by a collection of individual projects, minimal publications resulting from cross-disciplinary collaboration

Technology Transfer: Results significantly impacting industry/practitioners (patents, licenses, technology transferred to industry and practice)

Technology Transfer: Results are not advancing technology/ processes/ procedures for industry/practitioners

High Quality Education/Educational Outreach

(Years 7-10)

Low Quality Education/Educational Outreach

(Years 7-10)

Culture: Cross-disciplinary, cross-institutional education culture flourishing with impacts beyond the ERC

Culture: Little or no cross-disciplinary, cross-institutional, and/ or team-based interaction on the part of faculty and students

Teaming: Undergraduate and graduate students work in teams; ratio of graduate to undergraduate students of approximately 2:1


Curricula: Research results continue to be incorporated in existing courses, and some new courses or degree programs/options are being implemented ERC continues to produce high quality educational output based on its research with a significant impact on the curriculum (impact on courses is required, new degree programs/options are optional) for undergraduate and

Curricula: Few if any research results have been integrated into courses for students and practitioners, little or no activity related to any proposed degree programs/options

graduate students and practitioners

Assessment: New education programs and curricular materials continue to be assessed and disseminated, including summative and formative evaluations

Assessment: New education programs and curricular materials have not been assessed; personnel involved lack appropriate background for the task

Systems Training: Students have opportunities for formal training in systems integration

Systems Training: Students have little or no awareness of systems issues, no formal training in place

Industry Interaction: Students continue to have ample opportunities to work with industry/practitioners through internships or sponsored projects, several hired by member firms

Industry Interaction: Few if any students have worked or had opportunities to work with industry/practitioners

Outreach: Collaborations with female/minority serving partners have resulted in progressively increasing diversity in ERC's education programs over time

Outreach: No or ineffective collaborations with female/minority serving partners or no trends over time that increase diversity in ERC's education programs


REU: ERC failed to provide an REU program or the students involved in its REU program are not diverse

K-12 Outreach and RET: Pre-college outreach includes an effective RET program and effectively involves K-12 students with an emphasis on increasing the participation of underrepresented groups in engineering

K-12 Outreach and RET: Pre-college outreach or RET programs are nonexistent, inappropriate or disconnected from the ERC's research and education programs, or there is no emphasis on increasing the participation of underrepresented groups in engineering



High Quality Industrial/Practitioner Collaboration and Technology Transfer (Years 7-10)

Low Quality Industrial/Practitioner Collaboration and Technology Transfer (Years 7-10)

Membership Agreement: Center-wide membership agreement structures the industry collaboration program with clear statements of fees, benefits, and intellectual property policies that promote technology transfer


Membership: Growing or stable group of members across sectors and throughout the supply chain appropriate for the ERC's vision. Key players are members, members strongly committed to supporting the ERC after graduation

Membership: Membership promise of proposal not fulfilled, many of those committed or promising to commit did not sign up, significant numbers of firms/agency are leaving, and/or major sectors are missing, little commitment to the ERC after graduation

IAB: Industrial Advisory Board (IAB) active and effective; SWOT process yielding cogent advice to the ERC


Industry Integration: Industrial collaboration has become a Industry Integration: Industry involved only on a project-

cooperative partnership that is integrated into the ERC's planning, research, and education activities

by-project basis, no collective, collaborative partnership

Membership Fees: Membership fees provide discretionary funds for the ERC and commensurate with typical investments in academic R&D for the sectors represented by the firms involved


Technology Transfer: Knowledge and technology transfer is impacting industry/practitioners




Low Quality Configuration & Leadership Effort Years (Years 7-10)

Institutional Configuration: Appropriately integrated institutional configuration among lead, core partner, and outreach institutions, partnership strong


Center Director/Deputy Director: Highly effective Center Director and Deputy Director, have implemented vision and are providing capable leadership for the ERC and the university, effectively structuring the ERC for graduation

Center Director/Deputy Director: Center Director and/or Deputy Director have not translated vision into operation, leadership skills of one or both are not up to the task, little or no leadership evident regarding survival of ERC after graduation

Leadership Team: Other members of the leadership team (research thrusts, education, industrial collaboration, SLC, and administration) are cohesive and effective in planning and implementing the research, education, industrial collaboration, and administrative aspects of the ERC

Leadership Team: Some or all of these leaders are not effective and there are no plans to replace them

Research Team: High quality integrated research team shares the vision, appropriate mix of expertise (faculty, practitioners, and students) to fulfill vision and systems goals

Research Team: Research team does not share the vision, is low quality, or does not have the mix of expertise necessary to fulfill the vision and systems goals, or operating independently

Student Leadership Council: Student Leadership Council (SLC) in place, is effectively leading student programs, SWOT process is impacting ERC, ERC leaders are receptive to the SLC's recommendations for improvement, SLC has adequate resources to achieve its goals

Student Leadership Council: Student Leadership Council is not effective, doesn't use SWOT analysis, turnover in leadership not smooth, ERC leaders not receptive to SLC recommendations

High Quality Diversity Effort (Years 7-10) Low Quality Diversity Effort (Years 7-10 )

Diversity Strategic Plan: Strong strategic plan for diversity in place benchmarked national engineering averages, results demonstrate continued effectiveness of plan. Strong partnership for diversity with partner deans and

Diversity Strategic Plan: Ineffective strategic plan for diversity in place; no evidence of commitment to diversity. No partnership for diversity with administrators.

department chairs.

Minority Serving Institution Interactions: In research and education, ERC involves at least one minority or female serving institution , a Louis Stokes Alliance for Minority Participation

Minority Serving Institution Interactions: Ineffective or unsuccessful efforts to involve institutions / NSF diversity awardees that serve women and other underrepresented groups

Leadership Diversity: Team of leaders is diverse in gender, race, and ethnicity (LSAMP), and at least one other connection with an Alliance for Graduation Education of the Professoriate (AGEP), or at institutions that involved Native Americans, etc.












Management Systems: Management systems weak, poor goal setting and delivery, management fails to correct poor performance

Use of Financial Resources: Effective use of financial resources to achieve the ERC's goals, thrust and institution level budgets are appropriate for their roles in the ERC, timely allocation of funds, any annual residuals are below 20% of NSF support

Use of Financial Resources: Allocation of resources not commensurate with achieving the ERC's goals, long delays in allocation of funds, any annual residuals are significantly greater than 20% of NSF support

Outside Input: Effective incorporation of outside input in planning, project review, and assessment


Post-Graduation: Realistic and sound business plan for financial self-sufficiency when NSF support ceases

Post-Graduation: Weak or unrealistic business plan or none in place for financial self-sufficiency when NSF support ceases



Equipment/Facilities: High quality experimental and enabling equipment/facilities; test beds effective

Equipment/Facilities: Experimental and/or enabling equipment/facilities lack critical components, are not state-of the art, or test bed development is not evident

Communications Capability: Headquarters and communications network facilitate interaction among students, faculty, and industry/users and participating institutions

Communications Capability: Headquarters and communications network are effectively non-existent

University Administration: Effective partnership with university administration facilitates the success of the Center through policies that encourage its cross-disciplinary configuration, its diversity, and its partnership with industry; deans and department heads committed to success

University Administration: University administration does not facilitate the cross-disciplinary configuration, diversity, or industrial partnership of the Center, deans and department heads not involved

Funds: Investment made by industry/users, university, and other non-NSF investors commensurate with their ability to contribute and benefit

Funds: Most or all sectors are below what would be expected

Early Testbed

High Quality Testbed Low Quality Testbed

Testbed Requirements and Metrics: The ERC has begun to define requirements for the testbed derived from the vision and systems goals of the ERC

Testbed Requirements and Metrics: Requirements for the testbed are not in line with the systems goals of the ERC

Technology Integration: Testbeds are designed to prove the feasibility of the ERC’s vision and implemented to probe the research by testing the enabling technology, including devices, modules or subsystem components.

Technology Integration:The testbed are not designed to effectively prove the feasibility of the enabling technology, including devices, modules or subsystem components, in a system-like environment.

Function in Research: The testbed is serving as a versatile experimentation site, through which the performance of novel technologies are measured and results are fed back into the research thrusts to stimulate improvements or generate new research directions. ERCs equipped with several testbeds have ensured complementary functionalities and exchange

Function in Research: The testbed is not serving as a versatile experimentation site for novel technologies, and / or the measured performance of modules is not fed back into the research thrusts to stimulate improvements or generate new research directions. Testbeds are duplicating testing functions.

across testbeds.

Guidance: Testbeds requirements and metrics are reviewed on a yearly basis by the ERC team with input from the IAB, SAB, or other appropriate user inputs, i.e. clinicians or local government users, etc.

Guidance: The testbed is not reviewed on a yearly basis by the ERC team with input from the IAB, SAB, and other appropriate user inputs, i.e. clinicians or local government users, etc.

Role in Education: Testbeds are providing students with hands-on experience in “building” technology, integrating devices and components, or testing system-level performance.

Role in Education: Testbeds are not providing students with experience in “building” technology, integrating devices and components, or testing system-level performance.

Assessment: Through the definition of objective, stage-appropriate metrics, successful technologies are being identified and analyzed; the testbed is designed as a tool for comparing and validating the research approach(es).

Assessment: The testbed is not being used as a tool for comparing and validating the research approach(es).

Developing Testbed


Testbed Requirements and Metrics: With a set of performance metrics in place, the ERC has successfully implemented some of its near-term testbed milestones. In response to milestone accomplishments, testbed requirements are being refined to be consistent with the vision and system goals of the ERC. Long term testbed goals continue to push the state-of-the-art.

Testbed Requirements and Metrics: The ERC has not achieved any of its testbed milestones nor refined the requirements for the testbeds, to meet the vision and systems goals of the ERC.

Technology Integration: The testbed is utilized to probe the research by testing the enabling technology at its different levels of maturity, including devices, modules or subsystem components in a system-like environment.

Technology Integration: Testbeds are not utilized to probe the research by testing the enabling technology, including devices and subsystem components in a system-like environment.

Function in Research: The testbeds serve as a versatile experimentation site, through which the performance of component technologies may be measured and/or compared with competing technologies and results are fed back into the research thrusts to stimulate improvements or generate new research directions.

Function in Research: The testbeds are not serving as a versatile experimentation site, through which the performance of component technologies may be measured and/or compared with competing technologies. The collected data is not fed back into the research thrusts to stimulate improvement s or generate new research directions.

Technology Translation: Testbed results are improving confidence in the technology’s performance and reproducibility, highlighting relevant

Technology Translation: Testbed results are not improving confidence in the technology’s performance and reproducibility. The data collected in the testbed is not used to

applications. The testbed data collection is designed to help facilitate potential technology translation opportunities.

highlight relevant applications or facilitate potential technology translation.

Guidance: Testbeds are reviewed on a yearly basis by the ERC team with input from the IAB, SAB, and other appropriate user input, i.e. clinicians or local government users, etc.

Guidance: Testbeds are not reviewed on a yearly basis by the ERC team with input from the IAB, SAB, or other appropriate user inputs, such as clinicians or local government users, etc.

Role in Education: Testbeds are providing students with hands-on experience in “building” technology that result in peer reviewed publications or conference presentations. Hands-on experience includes integrating devices and components, or testing system-level performance.

Role in Education: Testbeds are not providing students with experience in “building” technology, integrating devices and components, or testing system-level performance. Very few opportunities for students to present at conferences are coming out of the testbed research.

Assessment: Through the refining of objective, stage-appropriate metrics, successful technologies are being identified and pursued; the testbed has become a tool for comparing and validating the research approach(es). Accomplishments are benchmarked against the state-of-the-art.

Assessment: The testbed has not become a tool for identifying successful technologies nor comparing and validating the research approach(es). Testbed results are not relevant to the state-of-the-art.

Mature Testbed


Testbed Requirements and Metrics: The ERC has established, clear requirements for the testbeds derived from the systems goals of the ERC and a set of performance metrics has been implemented and refined.

Testbed Requirements and Metrics: The ERC does not have established, clear requirements for the testbeds as derived from the systems goals of the ERC and a set of performance metrics has not been appropriately implemented.

Technology Integration: Testbeds are utilized to probe the research thrusts by testing the enabling technology, including devices and subsystem components, and integrating functionalities in a system-like environment.

Technology Integration: Testbeds are not utilized to probe the research thrusts by testing the enabling technology, including devices and subsystem components, nor integrating functionalities in a system-like environment.

Function in Research: The testbed serves as a versatile experimentation site, through which the performance of novel technologies are measured and/or compared with competing technologies and results are fed back into the research thrusts to stimulate improvements or generate new research directions.

Function in Research: The testbeds are not serving as a versatile experimentation site, through which the performance of component technologies are measured and/or compared with competing technologies. Testbed results are not fed back into the research thrusts to stimulate improvement s or generate new research directions.

Technology Translation: Testbed results are improving confidence in the technology’s performance and reproducibility. They highlight relevant applications and help accelerate technology translation opportunities. Research projects nearing translation to industry are refined and consider the specific market requirements (such as performance, manufacturability or cost).

Technology Translation: Testbed results are not improving confidence in the technology’s performance or reproducibility. Relevant applications or potential technology translations are not being pursued. Research project nearing translation to industry, are not ensuring that the design addresses specific market requirements (such as performance, manufacturability or cost).

Guidance: Testbeds are reviewed on a yearly basis by the ERC team with input from the IAB, SAB, and other appropriate user inputs, such as clinicians or local government users. Those inputs are used to optimize the testbeds functionality.

Guidance: Testbeds are not reviewed on a yearly basis by the ERC team with input from the IAB, SAB, and other appropriate user inputs, such as clinicians or local government users. Those inputs are not exploited to optimize the testbeds functionality.

Role in Education: Testbeds are providing students with hands-on experience in “building” technology that results in conference presentations and publications in refereed journals. Hands-on experience includes integrating devices and components, testing system-level performance, and envisioning market application requirements.

Role in Education: Testbeds are not providing students with experience in “building” technology, integrating devices or components, testing system-level performance, or understanding market application requirements. Very few opportunities for students to present at conferences or publish in refereed journals are coming out of the testbed research.

Assessment: Through the optimization of objective, stage-appropriate metrics, successful technologies are being identified and pursued; the testbeds are a critical tool for comparing and validating the research approach(es). The testbeds is clearly being used to push the technology state-of-the-art.

Assessment: The testbeds are not utilized for comparing the research approach(es), nor validating that successful technologies are being identified and pursued. Testbed lags the state-of-the-art.

Microsoft Word - Renewal_Review_Protocol_Gen · Web viewDo not renew and phase-down support....

Documents

Transcript of Microsoft Word - Renewal_Review_Protocol_Gen · Web viewDo not renew and phase-down support....