National Intelligence University Outcomes and Assessment Toolkit

Tools for development of effective curriculum, assessment and evaluation

Introduction

"An institution of higher education is a community dedicated to students, to the pursuit and dissemination

of knowledge, to the study and clarification of values, and to the advancement of the society it serves.

The Middle States Commission on Higher Education (MSCHE), through accreditation, mandates that its

member institutions meet rigorous and comprehensive standards, which are addressed in the context of

the mission of each institution and within the culture of ethical practices and institutional integrity expected of accredited institutions.1''

"Midd le States accreditation is an expression of confidence in an institution's mission and goals, its

performance, and its resources. An institution is accredited when the educational community has verified

that its goals are achieved through self-regulation and peer review. The extent to which each educational

institution accepts and fulfills the responsibilities inherent in the process of accreditation is a measure of its commitment to striving for achieving excellence in endeavors.2"

The National Intelligence University (NIU) has evolved over the years to redefine its mission and

curriculum, along with changes to its name. Accredited institutions are expected to demonstrate

compliance and conduct their activities in a manner consistent with the standards and requirements

established by MSCHE. These efforts focus on the student learning experience, institutional assessment and assessment of student learning, and continuous self-review and improvement.

This document provides university faculty, administration, and staff with the too ls needed to understand

the standards and assessment processes at NIU that help ensure we meet accreditation standards. The

toolkit is updated regularly with:

• Information needed by both staff and faculty to ensure the quality of education

• New requirements from NIU's accrediting bodies and from the internal knowledge and experience gained during the previous review cycle

• Faculty tools, templates, aides, and other resources required (or strongly recommended) by

the Office of the Provost.

More specifically, this document is a toolkit to help faculty develop quality programs and student learning

outcomes, syllabi that enable student understanding of expectations, and effective tools for student assessment.

Questions about the information in this document can be directed to the Institutional Effectiveness (IE)

Department at:

NIPR: NIU IE Dept@dodiis.mil

JWICS: NIU IE Dept@coe.ic.gov

1 Middle States Commission on Higher Education (MSCHE), "Standards for Accreditation and Requirements of Affiliation", Thirteenth Edition, pg. v. 2 Middle States Commission on Higher Education (MSCHE), "Standards for Accreditation and Requirements of Affiliation", Thirteenth Edition, pg. 1.

ii

Addit iona l information about IE and university assessments can be found on the NIU SharePoint site:

https :(/iconline. coe. ic. gov/sites/NIU MGT /niu 1 provost/lnstutiono/Effective ness/Site Pages/Home. ospx

iii

Table of Contents

Institutional Effectiveness Department .. ............................. ............................ .................................... 1

Policies .... ......... ....... ........ ... ..... .. ....... ....... .... ..................... .... .... ........................ ... ... ................ ... ...... . 2

NIU's Institutional Assessment Plan .... ......... ......... ..... .. ... ... ................. ... .... .................. ..... ....... ... .... ... 5

Assessment Strategies and Tools ............ .......................... ............................ ..................................... 7

Faculty Curriculum Vitae Requirements ................................. ............................................................. 9

Academic Review Roles ........ ..... ...... .... ...... .... .......... .... ........... ...... ... .... ...................... .... ...... ..... ... ..... 10

Development of Learning Outcomes .... ....... .. .......... ..... ..... ... ... ... ......... ...... .... ...... ........ ... ........... .... .... 13

Bloom's Taxonomy .......................................................................................................................... 15

Course Syllabi .................................................................................... ............... ........ ....................... 18

Academic Program Review ... ... ...... ..................... .... ......... ....................... ..... ... .... ............. .. .......... .... . 23

NIU's Roadmap to Evidence .. ..... ............................................. ...... ...... ... .......................................... 24

Appendix A: Assessment Acronyms & Glossary ...................................... ........................................... 25

Appendix B: Direct vs. Indirect Measures of Assessment ..... ...................................... ........................ 27

Appendix C: Student Surveys ..................... .... ........ .... ...... .... ....... ........... .. .... ........... ..... ............. ... ..... 28

Appendix D: Using CoursEval to Access Survey Results ...... ................ ... .... ........ ................................. 32

Appendix E: Evaluation vs. Assessment .. ............................................................ ............................... 34

Appendix F: Rubrics ......... .................. ...... .... ..... ................. ... .... ..................... ........... ... .................... 35

Appendix G: Faculty Peer-to-Peer Review Form ......... ....................... .................... ............... ... .... .... 39

Appendix H: Course Syllabus Template .................... .... .... .. ............................... .............. ... .. ...... ... .... 40

Appendix I: Outcomes Assessment Form ................................................. ........................... .............. 43

Appendix J: Pre-Public Release Review Process ... ..... ........ ................ .... ......... ......... ................... ... ..... 45

iv

v

NIU Institutional Effectiveness Department

The Institutional Effectiveness {IE) Department serves, under the Provost, as a resource for assessment,

data gathering and analysis, program review, and support of accreditation efforts. IE is committed to being

proactive in providing quality services to support the university and decision makers with accurate and

timely information needed to effective ly lead the university.

VISION

Build a university-wide culture of assessment and continuous improvement.

MISSION

Enhance university effectiveness, maintain accreditation, and advise leadership through

institutional evaluation and assessment.

GOALS

l. Coordinate and monitor compliance activities for institutional accreditation.

2. Produce accurate, concise, and timely data and analysis to support institutional

planning, leadership decision-making, and university publ ications.

3. Foster conversion of institutional assessment data into strategic action.

4. Measure achievement of institutional, program, and course learning outcomes.

1

Policies

Defense Intelligence Agency (DIA) is the executive agent for NIU, and therefore NIU falls under governance

of DIA policies, directives and instructions. NIU only develops policies to cover specific areas that are not

covered by DIA issuances.

NIU Policies can be found on the NIU SharePoint site on JWICS at:

https://icon line .coe. ic.gov /sites/NIU MGT /N IU%20Policies/Forms/Allltems.aspx

DIA issuances can be found on JWICS at :

https://dia.coe.ic.gov/sites/lssuances/default .aspx

Some of the most commonly used DIA policies are:

DIA Directives

DIAD 1000.100 DIA Editors Board 30 Jun 2015

DIAD 1010.100 Health Promotion and Wellness Program 24Jul2012

DIAD 1020.100 DIA Equal Employment Opportunity and Diversity 15 May 2014 Program

DIAD 1100.100 Disclosure of DIA Organizationa l and Personnel 04 May 2015 Information

DIAD 1500.lOOA DIA Civilian Fitness Program 23 Dec 2012

DIAD 2200.lOOA Com batting Trafficking in Persons 19 Aug 2014

DIAD 3200.100 Human Subjects Research Plan 24 Feb 2014

DIAD 3305.100 Joint Reserve Intelligence Program 12 May 2014

DIAD 5100.100 National Intelligence University Board of Visitors 17 Mar 2014

DIAD 5200.100 Workplace Violence Prevention and Response 18 Jun 2014

DIAD 5200.200 Investigative and Special Investigative Inquiries 28 Dec 2015

DIAD 5240.200 Intelligence Oversight Program 17 Nov 2013

DIAD 5410.600 DIA Policy and Procedures on Civilian Military and 27 May 2014 Contract Employee Contact with Congress

DIAD 5700.100 Access Identification Badges 2 Jul2014

DIAD 5801.100 Personnel Security Program 26 Oct 2015

DIAD 6055.100 Occupational Safety Health and Environmental Protection 2 May 2014 Program

DIAD 6495.100 Sexual Assault Prevention and Response Program 20 Nov 2012

2

DIA Instructions

DIAi 1015.001 DIA Civilian Drug-Free Workplace

DIAi 1020.002 Reasonable Accommodation of Qualified Individuals with Disabilities

DIAi 1025.002A Personnel Training

DIAi 1025.003 Establishment and Support to Defense Intelligence Agency-Affiliated Private Organizations

DIAi 1100.003 Outside Employment Reporting Process

DIA 1300.001 Military Reserve Personnel Administration

DIAi 1350.001 Military Personnel Administration

DIAi 1400.00lA Civilian Performance management

DIAl 1400.002 Civilian Compensation and Work Schedules

OJAI 1400.008 Employment and Placement

DIAi 1400.010 Federal Benefits

DIAi 1404.010 Separation and Out-Processing Procedures

DIAi 1416.001 Performance-Based Bonuses

DIAi 1416.002 Injury Compensation Program

DIAi 1422.002 Time and labor Reporting

DIAi 1424.00lA Civilian leave Programs

DIAi 1426.001 Remedial Disciplinary Adverse and Performance-Based Actions Change

DIAi 1426.002 Employee Grievance System

DIAi 1432.00lA Civilian Awards Program

DIAi 1500.002 Telework Program

DIAi 1700.001 Recording Person Administrative and Career Information in eZHR

DIAi 4515.001 Temporary Duty Travel Request and Reimbursement Procedures

DIAi 5000.035 DIA Acquisition Regulation Supplement and Instruction (DARSI)

DIAi 5010.003 Internal Control Program

DIAi 5145.001 Alternative Dispute Resolution Program

DJAI 5230.002A Use of Copyrighted Materials

DIAi 5340.001 Administrative Investigations

DIAi 5400.001 Privacy and Civil Liberties Program

DIAi 5400.002 Freedom of Information Act Program

3

11Dec2015

19 May 201S

30 Apr2014

03 Jun 2015

14Jul2015

2S Jul 2014

20 Nov 2015

06 Mar2014

20 Sep 201S

03 Jun 2015

OS May 201S

19Jun 2014

lOJul 2014

23 Jul 2013

29 Oct 2013

14 May2015

06 Oct 2014

14 Dec 2015

23 Nov 2015

26 Jun 2012

24 Mar 2015

23 Jun 2015

8 Apr 201S

8 Feb 201S

8Jan 2014

16Jul201S

23 Mar201S

OS Feb 201S

9 May 2014

DIAi 5400.005 Prepublication Review of Information Prepared for Public 19 Nov 2013 Release

DIAi 552D.001 Agency Support Agreements lOJul 2014

DIAi 5700.001 Visitor Access Control 21Aug2Dl4

DIAi 580D.001 Personnel Security Procedures 26 Oct 2015

DIAi 6010.001 Ergonomics Program 28 Apr2D14

DIAi 6050.D03 Accident Investigation and Reporting 1 Mar2012

DIAi 6050.004 Environmental Compliance Program for Facilities and Sites D2 May 2014

DIAi 6055.002A Occupational Health 16Jan 2014

DIAi 7D40.001 Participation and Reimbursement of DIA Personnel in 13 Mar2015 Government Conferences & Meetings and in Activities of Non-Federal Entities

DIAi 704D.003 Government Travel Charge Card and Travel Advances 29 Dec 2014

DIAi 7040.D08 Coin Program 25 Mar 2015

DIAi 7040.D09 Local Travel Guidance and Reimbursement Procedures 14 Sep 2011

DIAi 7040.016 Planning Conduct and Approval of DIA-Sponsored 22 Apr2D15 Conferences and Off-Site Staff Meetings Change

DIAi 705D.002 Whistleblower Protection 24Jun 201-3

DIAi 846D.002A Portable Electronic Devices 03 Mar2015

DIAi 850D.003 Systems Access 23 Mar2014

NIU must also comply with certain policies from other organizations within the Department of Defense (DoD) and office of the Director of National Intelligence (ODNI). The ones that most impact NIU are:

OONI Issuances

Strategic Guidance for the National Intelligence University 2 Jan 2015

ICD 205

ICD 660

ICPG 660.1

CJCS Issuances

Analytic Outreach

JC Civilian Joint Duty Program

IC Civilian Joint Duty Program Implementation Guidance

CJCSI 1800.0lE Officer Professional Military Education Policy

OoO Issuances

DoDD 5105.21 Defense Intelligence Agency

DoDI 3305.01 National Intelligence University

4

28 Aug 2013

11 Feb 2013

24Jul 2015

29 May 2015 with Modifications

18 Mar 20D8

24Jan 2017

NIU's Institutional Assessment Plan

Each year IE executes a university-wide assessment plan of the university. The goal of the assessment plan is to:

1) Assess student learning by ensuring students are acquiring and demonstrating the knowledge,

skills, and abilities taught in the classroom.

Evidence: student papers, tests, quizzes, presentations, theses, etc.

2) Assess university-wide effectiveness by measuring the quality of the NIU academic programs

and the effectiveness of university processes and policies.

Evidence: academic program review, surveys, focus groups, and interviews of NIU faculty, staff,

and students

3) Collect direct and indirect evidence (See Appendix B: Direct Vs. Indirect Measures of Assessment)

from a variety of sources that demonstrates student learning and the overall effectiveness of

NIU programs and faculty.

Evidence: The NIU Roadmap to Evidence repository

IE rolls out the Assessment Activity Plan at the beginning of each academic year. The plan includes the

list and schedule of the standard assessments, surveys, and reports t hat will launch for the year.

SURVEYS CSI Proeram of Study Completion Survey

End-of-course Surveys

End-of-Program Survey- Certificate

End-of-Program Survey - Dceree

External Stakeholder - Alumni

External Stakeholder - IC Leader

External Stakeholder - Incoming Student

Leadershlp and Management (L&M) Certifi cat

Monthly Executive Distinguished Lecture Seri es

eSurvey

Orientation Survey

Research Fair Survey

Research Workshop Survey

!f tt_!<ly President's Lecture Series Surveys

ASSESSMENTS Academic Program Review 2017 - Planni ng

Academic Program Review 2017 - Llunch

Survey

Leadership and Management Certificate Study (Support)

5

Fall Winter Spr ine Summer

~ Quarter Quarter_ Quarter

I I I

' . ---;::;

--- ., I I I

--- ., .,

Fall Winter Spring Summer Quarter Quarter Quarter Quarter . -----

~

--

REPORTS AY 2016 Annual Report

AY2017 Annual Report

AY 2017 Ata Glance (Facts and Figures)

AY 2017 Common Data Set Report (Facts & Figures)

AY 2017 FACTBOOK

Fall

Quarter

Wi nter

Quarter

Spring Summer

Quarter Quarter

AY ~aduation and Retention Report (Facts & Figures)

AY2017 Ori entation Report

AY 2017 Quick Facts Report (Facts~ Figure.s)

CSI Program of Study Completion Report

End-of-Course Summary Reports

End-of-Program (Certi fi ca te) Report

End-of-Program (Degree) Report

NIU H~torica l Data Report (AY 2017 Facts & Figures)

Research Fai r Report

Research Workshop Report

S& Tl Concentration Summary Reports

--Figure 1: Sample Assessment Activity Plan from AY 2016-17

Reports are distributed w ithin 2-8 weeks after the close of the assessment. IE initially distributes the reports as follows:

Annual Report and FactBook

CSI Program of Study Completion Reports

End-of-Course Reports/End-of-Course Executive

Summary/Program Level Summary Reports (including

Master's Core and Thesis)

I End-of-Program Executive Summary

External Stakeholder Senior Leader Report

External Stakeholder Alumni Report

External Stakeholder New Student Report

NIU Facts and Figures

Orientation and Research Fair Executive Summary Reports

President's Lecture Series (PLS) Summary Report

S&TI Concentration Summary Reports

Figure 2: Report Distribution List

ODNI, USDI and DIA Senior Leaders, BOV, University

students, faculty and staff, and stakeholders

I CSI Deans, Directors, and Faculty Chairs

6

Faculty (Course/Section Reports only)

NIU Leadership, Deans, and Academic Program and

Academic Center Directors

NIU Leadership, Deans, and Di rectors

NIU Leadership

NIU Leadership and Office of Outreach

Registrar/ Admissions

Public (NIU Website)

NIU Leadership and Orientation and Research Fair Coordinators

Office of Outreach

S& Tl Deans and Faculty Chairs

Assessment Strategies and Tools

IE Assessment of Institutional Effectiveness

IE utilizes a variety of assessment methods to collect data and evidence, and to identify and analyze performance outcomes at all levels of the university. Some of these methods include:

• Online Surveys: IE deploys numerous surveys throughout the academic year to collect data from

students, faculty, and/or staff about their opinions and experiences regarding university

programs, courses, services, processes, practices, and governance. (See Appendix C: Student

Surveys for information about the student surveys.)

• Focus Groups: A form of qualitative research in which a target or sample group of students,

faculty, and/or staff are asked about their perceptions, opinions, beliefs, and attitudes towards a

specific subject in order to understand and/or identify criteria that may help solve or resolve a

problem (or opportunity).

• Academic Program Review: A review process designed to look across NIU at both academic and

non-academic programs and review them for quality, demand, cost effectiveness, and centrality

to mission.

Faculty Assessment of Student Learning

Assessment of student performance is critical in determining the effectiveness and success of NIU. NIU

evaluations are continual and interactive, and involve students, faculty, and University leadership.

Evaluations of NIU classes and curricula are carefully constructed to ensure clear linkage among desired

learn ing outcomes, changes to course syllabi, joint educationa l goals, and the NIU mission.

NIU faculty use a combination of strategies and tools to evaluate and assess3 student learning. Some of

these strategies and tools include:

• Observation checklist: A list of questions an observer will be looking to answer in relation to

course or program outcomes when doing a specific observation of a classroom or student

academic exercise (problem solving, verbal communication, collaboration, etc.)

• Anecdotal notes: A record of specific observations of individual student behaviors, skills, and

attitudes as they relate to the outcomes in the course or program. Such notes provide cumulative

information on student learning and direction for further instruction4.

3 See Appendix E; Evaluation vs . Assessment to learn the differences between evaluation and assessment goals.

4 Learnalberta.ca/content/mewa/htlml/assessment/anecdotalnotes.html "Assessment Strategies and Tools: Anecdotal Notes.

7

• Portfolios: A purposeful collection of student work samples, self-assessments, and faculty assessment/feedback that reflect student progress.

• Questions and answers: Verbal or written tests of knowledge that can be delivered in multiple forms - exams, quizzes, papers, assignments, in-class activities or discussion, presentations.

• Grading scales: Tools that state specific criteria for levels of learning and/or meeting course/program outcomes.

• Rubrics: A tool that consists of a fixed measurement scale and detailed description of the characteristics for each level of performance that is used to evaluate a student's product or performance. The descriptions focus on quality {and not quantity) of the product or performance being evaluated. Bloom's Taxonomy is the standard used at NIU for developing criteria to describe acceptable levels of performance. {See Appendix F: Rubrics for more information.)

Unlike many military educational institutions, NIU does not use its core curriculum as a common element for determining class standing. At NIU, the core curriculum represents a framework of knowledge that the University believes the students must have to understand intelligence at the strategic and operational levels a~d to study more advanced topics.

8

Faculty Curriculum Vitae Requirements

In addition to managing the university's assessment plan, IE coord inates and monitors compliance activities for institutional accreditation. One of the key activities involves mainta ining a file of current curriculum vitae (CVs) for all faculty. This is essential to the support of MSCHE Standard 1115, criteria 2, stating:

• Student learning experiences that are designed, delivered, and assessed by faculty (full-time or part-time) and/or other appropriate professionals who are:

o Rigorous and effective in teaching, assessment of student learning, scholarly inquiry, and service, as appropriate to the institution's mission, goals and policies;

o Qualified for the positions they hold and the work they do;

o Sufficient in number;

o Provided with and utilize sufficient opportunities, resources, and support for professional growth and innovation;

o Reviewed regularly and equitably based on written, disseminated, clear, and fa ir criteria, expectations, policies and procedures.

Faculty must submit updated CVs at the beginn ing of each academic year. IE sends out a data call for CVs at the start of the academic year. Adjunct, part-time or new faculty starting at other t imes of t he year, will submit CVs upon arrival.

There is no prescribed format for CVs, however, items that should be considered for inclusion are:

• Name and contact information • Education: degrees, school names, thesis/dissertation titles, Professional licenses/certifications • Professional/Academic Honors and Awards • Professional Experience • Extracurricular and Volunteer Experience (Relevant to the position)

• Professional Affiliations and Activities (Specific to Academic Career) • Research Experience • Publications and Presentations

• Interests and Qualifications

5 Middle States Commission on Higher Education (MSCHE), "Standards for Accreditation and Requirements of Affiliation", Thirteenth Edition, pg. 7.

9

Academic Review Roles

Although IE leads a majority of the institutional assessment efforts at NIU, NIU faculty have several

opportunities to actively participate in the ongoing review of all NIU programs through various review

processes and forums.

Quality Circle Review

Through the Quality Circle process, faculty examine their own observations, peer reviews, student

critiques, and inputs from the Deans, Program Directors, and Director, IE, of the courses they teach. The

Quality Circle lead and faculty colleagues decide on modifications for the course description, mission

statement, topical outline, and desired student learning objectives. Minor modifications to electives can

be handled between the Quality Circle and Program Directors. Major modifications must be presented to

the Dean(s) and briefed to the entire faculty of the College and/or School that owns the program

curriculum. Program Directors from the Master of Science of Strategic Intelligence (MSSI) and Master of

Science and Technology Intelligence {MSTI) programs supervise all of the Quality Circles to ensure that

course design, revision, and updates occur regularly.

Curriculum Working Groups (CWGs). CWGs and Concentration Chairs represent critical components of

the NIU review process. The faculty's findings are necessary for a credible assessment of the University's

mission success.

The College of Strategic Intelligence CWG (CSl/CWG) is chaired by the MSSI Program Director and includes

facu lty members elected from the CSI faculty.

The CSl/Graduate Core Working Group (GCWG) is responsible for managing the MSSI and BSI program

curricula and ensuring all undergraduate and graduate courses and certificates nominated for inclusion in

the MSSI and Bachelor of Science in Intelligence (BSI) curricula meet academic minimums for course

credits and MSCHE accreditation standards. NIU/GCWG is chaired by Graduate Program Directors from the College and School, and ensures the quality and relevance of the graduate core curriculum.

Concentration Chairs. Within the Anthony G. Oettinger School of Science and Technology Intell igence

(S&TI), faculty selected as Concentration Chairs are responsible for managing curriculum concentrations. Findings and recommendations are made by these Chairs with the appropriate oversight from the S&TI

Dean or any other members of the University leadership. Concentration Chairs are selected by the Dean

of the School, and may also serve as Faculty Supervisors.

Academic Policy and Standards Committee (APSC)

Co-chaired by the Graduate Program Directors from the College and School, APSC is an administrative

committee responsible for reviewing academic policy, admissions criteria, and institutional standards, and

10

makes policy recommendations to the Deans. APSC also reviews student complaints, appeals, and grade protests, and conducts investigations into alleged academic misconduct.

Faculty Senate

Another important part of NIU program review is the Faculty Senate. The Faculty Senate advises the President, Provost, Deans, and Academic Program Directors on matters affecting the academic welfare of the University. The Senate promotes effective faculty participation and communication in the governance of the University as a whole. Participation as an NIU Faculty Senator is an inherently governmental function, so it is limited to government employees.

The NIU Chair of the Faculty Senate is elected by the senators, serves as a member of the President's Council, and attends the semiannual Board of Visitors (BOV) meetings as a guest. Lastly, the Deans solicit faculty inputs and prepare agendas for periodic faculty meetings. The faculty meeting is an important part of University communications and review, and includes discussion on curriculum changes, course additions, instruction, research, outreach, human subjects review, institutional assessment, institutional effectiveness, and student issues. Unresolved issues are considered for further discussion with Concentration Chairs, CWGs, or the APSC.

University Leadership

The recommendations and inputs from the various review processes within the University are discussed in many leadership fora: Provost's Deans and Directors meetings and other leadership meetings. When issues beyond the normal realm of University activities arise, the Deans, Provost, JE Director, and President may create special committees to review such issues and make recommendations. The Joint Professional Military Education (JPME) Quality Circle serves as one example. Formed to work in conjunction with the Quality Circles for courses containing Joint Learning Areas {JLAs), the JPME Quality Circle ensures that the JLAs are presented effectively in the course material, and has evolved into an important, ongoing body.

The various NIU review processes culminate at every other week President's Council Meeting that is attended by the President, Provost, Chief of Staff, Vice Presidents, CSI and S&TJ Deans, Program Directors and Front Executive Staff.

At this meeting, the University President leads a dialogue on all issues of importance to the University, including those outside the University environment that could affect the University's mission.

Decisions on curriculum issues and University policy are finalized, and typically take effect upon the issuance of the next catalog or appropriate handbook (policy or student). If a policy change takes effect before the issuance of the relevant publication, or immediately, the President directs University leadership to provide immediate and clear communication throughout the University.

11

Peer-to Peer Review

An important part of ensuring the quality of instruction is regular review of the faculty, their expertise and teaching abilities. While there are formal reviews conducted by the Deans, Associate Deans or leadership, another type of review is the peer-to-peer reviews. These can be conducted in a number of ways, from new faculty being reviewed by the mentor or other faculty to help them with their teaching styles or as part of teaching a new course; to regular reviews of faculty by supervisors or program directors to seasoned faulty being observed by less experienced faculty to learn from observations. A peer-to-peer review form has been developed to help observers to evaluate faculty and their teaching. (See Appendix G: Faculty Peer-to-Peer Review Form.)

12

Development of Learning Outcomes

Learning outcomes describe what learning students should have acquired or should be able to demonstrate at the end of the class or program of study. Learning outcomes emphasize the knowledge, skills and abil it ies that are going to be taught, rather than the form of instruction.

Some guidelines when writing Student Learn ing Outcomes:

• Include an action verb and include an observable student behavior. • Begin with the end in mind (what the student should accomplish.)

• Student learning outcomes should be measurable. • Students should be assessed. based on the Learning Outcomes.

Note: The action verb chosen for the learning outcome statements should focus on what is going to be assessed and the tools, instruments and metrics that will be used to assess the extent of t he intended learning. Bloom's Taxonomy provides an excellent source of action verbs and is NIU's standard for classifying learning objectives.

There are several different philosophies on writing objectives. Two examples are:

• ABCD Method

o A- Audience (Learners, who will be doing the performance) o B - Behavior (Performance, Skill or Knowledge learner w ill be able to do) o C - Condition (Under what circumstances) o D- Degree (How well must they perform)

• SMART Objectives

o Specific - Focuses on specific category of student learning o Measurable - Data can be collected o Attainable - Attainable given amount of experience knowledge o Relevant - Aligned with program learning outcomes o Time-bound - Any time restrictions

When writing learning object ives, you should answer the following questions:

• What do you want students to accompl ish?

• What knowledge, skills and abilities should the student demonstrate?

• How will the students be able to demonstrate what they have learned? (What steps will they take to accomplish the objective? What activities will they do? How wi ll they acquire the learning? Under what conditions will the learning occur?)

• How do the program and organ izational outcomes fi t w ith the program?

(What evidence do they have to demonstrate to show learning took place? What criteria will be used to evaluate the evidence? Who will conduct the evaluation?)

13

Verbs to Avoid

In order to measure the intended learning outcomes, it is best to avoid verbs that represent concepts that are difficult or impossible to measure. Examples of some of these are:

• Appreciate

• Be aware of

• Be familiar with

• Believe

• Comprehend

• Know

• learn

• Understand

14

Bloom's Taxonomy

The Taxonomy of Educational Objectives is a framework for classifying statements of what we expect or

intend students to learn as a result of instruction. The origina l hierarchica l model was created in 1956,

and named after Dr. Benjamin Bloom, who chaired a committee of educators (Bloom, Engelhart, Furst,

Hill & Krathwohl) that devised the taxonomy. It identified three domains of education activities or

learning:

• Cognitive - mental skills (knowledge)

• Affective - growth in feelings or emotional areas (attitude or self)

• Psychomotor- manual or physical skills

The cognitive domain involves knowledge and development of intellectual skills. The cognitive domain is

then further broken down into six major categories that are used to classify educational learning

objectives into levels of complexity or difficulty. The taxonomy is used to structure curriculum and

program learning outcomes, assessments and activities. The taxonomy was revised in 2001 by Lorin

Anderson and David Krathwohl and is known as the Revised Taxonomy.

The Revised Taxonomy identifies the following levels of cognitive learning6:

• Remembering - Retrieving, recognizing, and recalling relevant knowledge from long-term

memory

• Understanding - Constructing meaning from oral, written, and graphic messages through interpreting, exemplifying, classifying, summarizing, inferring, comparing and explaining

• Applying - Using information in new ways; carrying out or using a procedure or process through executing or implementing

• Analyzing - Breaking material into constituent parts; determining how the parts relate to one

another and to an overall structure or purpose through differentiating organizing, and attributing

• Evaluating- Making judgements based on criteria and standards through checking and critiquing;

defending concepts and ideas

• Creating - Putting elements together to form a coherent or functional whole; reorganizing

elements into a new pattern or structure through generating, planning or producing

6 International Assembly for Collegiate Business Education 2014-2016, "Bloom's Taxonomy of Educational Objectives and Writing Intended Learning Outcomes Statements.

15

Reasons for Using Bloom's Taxonomy7

• Accurately measuring student's abilities requires an understanding of different levels of cognition that are critical for learning.

• Developing intended student learning outcomes according to Bloom's Taxonomy helps students understand what is expected of them.

• Using Bloom's Taxonomy to develop intended student learning outcomes helps professors to plan and deliver appropriate instruction.

• Developing intended student learning using Bloom's Taxonomy helps faculty to design and implement appropriate assessment tasks, measures, and instruments.

• Having intended student learning outcomes based on Bio.em's Taxonomy helps to ensure that instruction and assessment are appropriately aligned with the intended outcomes.

Learning outcomes for upper level undergraduate and graduate programs should use the more complex skills on the right of the chart below.

7 International Assembly for Collegiate Business Education 2014-2016, "Bloom's Taxonomy of Educational Objectives and Writing Intended Learning Outcomes Statements.

16

Action Words for Bloom's Taxonomv8

Knowledge Understand Apply Analyze Evaluate Create

define explain solve analyze reframe design identify describe apply compare criticize compose describe interpret illustrate classify evaluate create label paraphrase modify contrast order plan llst summarize use distinguish appraise combine name classlfy calculate infer judge formulate state compare change separate support invent match differentiate choose explain compare hypothesize recognize discuss demonstrate select decide substitute select distinguish discover categorize discriminate write examine extend experiment connect recommend compile locate predict relate differentiate summarize construct memorize associate show discriminate assess develop quote contrast sketch divide choose generalize recall convert complete order convince integrate reproduce demonstrate construct point out defend modify tabulate estimate dramatize prioritize estimate organize tell express interpret subdivide find errors prepare copy identify manipulate survey grade produce discover Indicate paint advertise measure rearrange duplicate infer prepare appraise predict rewrite enumerate relate produce break down rank role-play listen restate report calculate score adapt observe select teach conclude select anticipate omit translate act correlate test arrange read ask administer criticize argue assemble recite cite articulate deduce conclude choose record discover chart devise consider collaborate repeat generalize collect diagram critique collect retell give examples compute dissect debate devise visualize group determine estimate distinguish express

illustrate develop evaluate editorialize facilitate judge employ experiment justify imagine observe establish focus persuade infer order examine illustrate rate intervene report explain organize weigh justify represent Interview outline make research judge plan manage review list question negotiate rewrite operate test originate show practice propose trace predict reorganize transform record report

schedule revise simulate schematize transfer simulate write solve

speculate structure support test validate

8 Center for university Teaching, learning, and Assessment, http://uwf.edu/cutla/SLO/ActionWords.pdf.

17

Course Syllabi

The following guidance recommends best practices for syllabus construction as well as the guidelines for

including course learning outcomes on course syllabi. Use the NIU Syllabus Template {See Appendix H:

Course Syllabus Template) to build your syllabus.

Overview

Courses at NIU must satisfy the common student learning objectives of their respective program. Each

faculty member determines the specifics of course implementation, provided the established

competencies and learning objectives for the course are addressed.

Instructors often have differing teaching styles for a particular course, but all must have the same course

description (and this must match the one published in the University Catalog), the same contribution to

mission statement, the same overall lesson topics, and the same desired student learning objectives on their syllabi.

Instructors may choose their own texts (in addition to at least one text that may be used by all the classes}, and define their own expectations, deliverables, teaching methodology, and grading practices.

Syllabus Guidance

The syllabus provides an overall layout for a course. University policy establishes all courses shall have a

syllabus and t he syllabus shall be uploaded into Blackboard and made available to students at or near the

start date of the course. The syllabus shall include, but not be limited to the following:

• Basic course and instructor contact information

• Course description and contribution to mission

• Course goals and student learning outcomes or learning goals

• Joint Learning Outcomes (if applicable)

• Course format, procedures, and/or policies

• Course requirements

• The instructor's grading criteria or policies

• Course schedule or outline

In addition to the required components, NIU recommends the following university statement s as an

addendum to the course syllabus:

• Academic Integrity

• Attendance

• Incomplete and Withdrawal grades

• lnclusivity

• Surveys

18

I. Basic Information: Provide basic information about the course, such as course numbers and titles, instructor contact information, and class time and location.

II. Course Description/Contribution to Mission: The description on the syllabus must be consistent with the description listed in the current NIU catalog (http:l/ni-u.edu/wp/niu-academic-catalog/). Additional information may be included after the bulletin description.

Contribution to Mission: Map course learning objectives (CLOs) to program learning objectives (PLOs). Why does this course exist? How does it fit in with the rest of the field/area's curriculum?

Ill. Learning Outcomes: For all students, clear learning objectives for the course provide a roadmap of the material to be covered over the semester. Clearly articulated [earning objectives are also important for faculty members to ensure that assignments and learning activities align with the course objectives. Learning objectives also provide a mechanism to ensure that evaluation methods and activities align with the content that is presented.

IV. Joint Leaning Outcomes: For all classes that have Joint Learning Outcomes (JLOs), those objectives must also be included in the syllabi. It is important for both students and faculty to understand whatJLOs will be covered during the course.

V. Course Format and Procedures: This is where specifications for attendance, participation, how class will be carried out, instructor expectations, etc. should be spelled out. In addition, if the course has multiple formats (like lecture & recitation, lab and discussion, group learning projects and/or presentations) these should be explained clearly. The syllabus should provide instructions to students on how they are expected to meet the learning objectives for the course. For example: Will most objectives be met through examination? Through participation? Or through projects?

·i. Class attendance and participation: 2. You will meet the objectives listed above through a combination of the following activities in this

course:

• Attend • Complete

• Participate

3. Late Work: (Example: Be sure to pay close attention to deadlines-there will be no makeup assignments or quizzes, or late work accepted without a serious and compelling reason and instructor approval.)

19

VI. Course Requirements: Whatever tasks and assignments are included in your course, they should be aligned with the specified learning outcomes (final learning state, skills, knowledge, attitudes and values the students leave the course with) you have defined and specified earlier.

Provide full text citations of all required materials required resources.

1. Course readings: a. Required text: [Title. Author]

b. Background readings, (what and where can they be accessed) 2. Course materials: lab manuals, technology, supplies, and any other materials required or

recommended for the student to complete the course requirements. 3. Use of the course Blackboard or other web site? 4. Resources?

VII. Evaluation Procedures/Grading Criteria: The syllabus should indicate how students are evaluated, including tests, quizzes, papers, assignments, weight of the assignments, etc. and clearly identify how the course grades are determined.

Criteria for grading includes the grading scale used for the course. If points are earned, be sure the total number of points is correct and all points are accounted for in the grading scale.

Grading Scale Examples Final Exam Total Points

40% of final grade 100 points

D=

F= 60 to <70%

<60%

VIII. Course Schedule/Outline/Calendar of Events: Provide students with a tentative projected outline of significant events that occur throughout the semester, including assignments, projects, examinations, field trips, guest speakers, etc. (May change to accommodate guest presenters & student needs)

For example:

T . OPICS d" Rea 1ngs to b d" e 1scusse d Assignment Weeki Topics/Major Concepts covered [Text] Chapter#, additional readings

from course packet, handouts WeekZ Experiential and collaborative learning - [Text] Chapter#, additional readings

Guest Lecturer from course packet, handouts Week3 Exam #1 Week4 Experiential and collaborative learning - [Text] Chapter#, additional readings

Guest Lecturer from course packet, handouts Etc. Uses of technology [Text] Chapter#, additional readings

Guest Lecturer from course packet, handouts

20

IX. Academic Integrity: Each student in this course is expected to abide by the NJU Code of Academic Integrity. Any work submitted by a student in this course for academic credit will be the student's own work. [Optional: For this course, collaboration is allowed in the following instances: list instances.]

• You are encouraged to study together and to discuss information and concepts covered in lecture and the sections with other students.

• You are expected to credit properly and accurately the source of materials directly cited or indirectly used (i.e., paraphrased) in any oral or written work. All students' work shall be their own, unless otherwise properly noted.

• You may not use entire papers or substantive selections of a paper from one course to complete work for another course or courses.

A grade of F will normally be assigned for any work proven to be undertaken or performed in violation of academic integrity. All instances of alleged violations of academic integrity will be handled in accordance with published NIU policies.

X. Attendance: Students are expected to attend all scheduled class sessions. Students missing more than one session face, at the discretion of the faculty member, penalties ranging from the lowering of the final grade to failure in the course. A student who misses three or more sessions and does not withdraw faces removal from the course and a failing grade.

XI. Incomplete 'I' and Withdrawal 'W' Grades: A faculty member may assign an incomplete (I) grade to a student whose work is satisfactory but is unable to meet all course requirements for extenuating circumstances. It is the student's responsibility to discuss with the faculty member the possibility of receiviFJg an 'I' grade. All requirements must be completed by the 9th week of the following quarter, or the 7th week of an eight-lesson quarter, and the faculty member must turn in the final grade by the 10th week of the following quarter. Jf a final grade is not submitted by this deadline, the 'I' grade is converted to an F. The Dean may extend the deadline in exceptional cases. As long as the 'J' remains on the transcript, it is treated as unsatisfactory academic performance.

Withdrawing from a course: Students may withdraw from a course until the midpoint of that course, such as the end of the 5th session of a 10-session course, or the end of the 4th session of an 8-session course. A notation of W is assigned to the transcript of a student who withdraws from a course before the withdrawal deadline. Withdrawal from a course after the midpoint of that course is allowed only for non­academic reasons and requires permission of the faculty member and the approval of the Dean. Students who are approved to withdraw after the midpoint of a course are assigned a grade notation of WP (withdraw passing) orWF (withdraw failing) by the faculty member, depending on the student's academic standing in the course at the time. The grade notation of WP carries no credit or academic penalty. A grade of WF is treated as an F when calculating the grade-point average and triggers academic warning. Students withdrawing at any time must complete the necessary documentcition through the Enrollment Services Office. Students who stop attending classes without an official withdrawal or the Dean's approval receive a grade of F for the course.

21

XII. End of Course Survey: Students are expected to complete a course survey near the end of this course. Course surveys are an essential component of the university's educational assessment process. Failure to complete the end-of-course survey may result in the withholding of all student grades by the Dean. Follow university instruction on accessing the university survey site (myCoursEval) and completing course surveys. Contact NIU Surveys@dodiis.mil if you have any questions regarding course surveys.

XIII. lnclusivity Statement: We understand that our faculty and students represent a rich variety of backgrounds and perspectives. NIU is committed to providing an atmosphere for learning that respects diversity. While working together to build this community we ask all members to:

• share their unique experiences, values and beliefs • appreciate the opportunity that we have to learn from each other in this community • value each other's opinions and communicate in a respectful manner • keep confidential discussions that the community has of a personal {or professional) nature • use this opportunity together to discuss ways in which we can create an inclusive environment in

this course and across the Intelligence and National Security communities.

22

Academic Program Review

NIU conducts academic program review (APR) on a four-year cycle in which existing academic programs

and services are reviewed for their quality, demand, cost-effectiveness, and centrality to mission. Each

unit takes a comprehensive look at its opportunities and challenges in regard to the NIU strategic plan and

university standards of accreditation. The results of the review are used by the unit or University to

develop and implement plans to sustain, improve, or change programs and services. Detailed instructions

are found in the NIU Academic Program Review Process Guide.

lf'HJ

h· ~' h "· "

\e~r 1 l~c.:i·~

Figure 3: The APR Schedule (AYs 2018-2021)

Purpose

• Self-assessment for planning

"f';1f l

U! Tf' H !1

11rlm ~ , i ,\.

f{, r.1 'r 1\1 n

• Improvement of programs and/or services

• Enhan.cement of student learning

/

• Support of Middle States Commission on Higher Education (MSCHE) requirements

• In cooperation with NIU's culture of assessment and continuous improvement

Review of Programs/Units

APR provides each unit the opportunity to pursue a path of inquiry, discovery, and assessment. The

emphasis of the review is on forward planning, informed by analysis of recent data trends with units

identifying a detailed plan to sustain excellence, correct deficiencies, or enhance the educationa l mission

of NIU. The unit is not expected to address issues during the review period, but will develop plans to do so in a timely manner and update progress in follow-up years in-between the next review cycle.

Annual Update

The Unit is expected to take actions to address the findings of the program review and report on actions

taken by submitting a progress report every year after the conclusion of the review. Reviews are

scheduled every 4 years. IE is responsible for scheduling the annual reports and next review.

23

NIU's Roadmap to EVidence

The Outcomes Assessment Form

It is important to measure the outcomes that are derived from the various means of assessment, e.g., changes to curriculum based on end-of-course surveys, changes to courses, changes to required readings,

or types of student assessments, etc. While taking notes during curriculum working groups, quality circles,

etc. may capture the data, it can be difficult to then find the information needed when trying to provide

the evidence for accreditation purposes.

The Outcomes Assessment Form (Appendix I: Outcomes Assessment Form) was developed to capture

information about the type of meeting, purpose of the meeting, alignment with strategic, MSCHE, and

JPME goals and objectives, evidence, discussion and analysis conducted, and then the decisions, outcomes or recommendations that come out of that meeting. There is also a space for any follow-up activities that

needs to be conducted.

This form is recommended to be used for meetings of groups eva luating curr iculum, goals, objectives, and

other significant events impacting the way NIU conducts business.

Documenting Evidence of Student Learning Outcomes and Continuous Improvement

NIU's Roadmap to Evidence (RTE) is an arch ive of documented evidence demonstrating NIU is meeting its

accreditation standards and requirements. The evidence is collected from leadership planning and

decision making, policy, student feedback, academic program review, faculty working groups (outcomes

assessment form and support documents (meeting notes, briefs, and other updates), and faculty evaluation and assessment of student learning.

The RTE is maintained by IE. RTE evidence is unclassified and processed through the prepublication

process (Appendix J: Pre-public Release Review Process) for release to NIU's MSCHE and JPME

accreditation reviewers.

IE maintains three versions of the RTE:

• A physica l database of all archived evidence that exists on the IE unclassified shared drive. This

database is regularly updated with the most current evidence.

• A virtual, password protected library of the most current evidence prepared for the latest MSCHE review. This on line RTE (http://ni-u.edu/RTE/) is normally updated in support of MSCHE reporting

(self-study; periodic review) and visits.

• A physical database of all JPME archived evidence that exists on the IE unclassified shared drive.

This database is normally updated before a J7 team visit.

24

Appendix A: Assessment Acronyms and Glossary

APR Academic Program Review

APSC Academic Policy and Standards Committee

BOV Board of Visitors

BSI Bachelors of Science in Intelligence

CL Os Course learning objectives

CSI College of Strategic Intelligence

CSIR Center for Strategic Intelligence Research

CVs Curriculum vitae

CW Gs Curriculum Working Groups

DIA Defense Intelligence Agency

DIAD DIA Directives

DIAi DIA Instructions

DoD Department of Defense

GCWG Graduate Core Working Group

IE Institutional Effectiveness Department

JLAs Joint Learning Areas

JLOs Joint Learning Outcomes

JPME Joint Professiona l Military Education

MSCHE Middle States Commission on Higher Education

MSSI Master of Science of Strategic Intelligence

MSTI Master of Science and Technology Intell igence

NIU National Intelligence University

ODNI Office of the Director of National Intelligence

PAJE Process for Assessment of Joint Education

PLOs Program learning objectives

RTE Roadmap to Evidence

S&TI Science and Technology Intelligence

Assessment: The ongoing process of measuring effectiveness for the purpose of improvement (to increase

quality).

Assessment Activity Plan: NIU's schedule of standard assessments, surveys, and reports that will launch

during the academic year. This plan is developed by the IE Department.

Evaluation: The process of observing or measuring, at a point in time, against an established target or

standard for the purpose of judging value or quality. (The primary focus is on scores or grades.)

Learning Outcome: Statement of which a student wou ld know and/or be able to perform as a result of a

learning experience (completion of a course; program; ce rtificate).

Student Learning Outcome Assessment: The process of using direct and indirect measures to evaluate

student learning, and using fact-based evidence to change and improve programs. The primary focus is

25

on learning, teaching, and outcomes. Findings may result in changes in learning, teaching, or non­academic support and services. Examples: Curriculum Working Group and Quality Circle assessments.

Measures: Means used to benchmark performance and to determine if an outcome or goal/objective is achieved.

Direct Measure: Measure that evaluates the actual performance or achievement of an identified outcome {administrative or student learning). A direct measure is preferable to an indirect measure because it is .tangible and visible. Examples: performance output (countj product), student output (paper; project; presentation), the application of a rubric to assess an identified outcome.

Indirect Measure: Measure that signifies performance or achievement through the evaluation of something else (i.e. opinion; perception; satisfaction.) An indirect measure may help substantiate a direct measure in that an indirect measure along with a direct measure is expected to provide a more comprehensive view of student learning (or process or service impact). Examples: survey, interview, peer review, graduation rate, course grade*.

Academic Program Review (APR): A systematic, comprehensive review of a unit's quality, demand, cost effectiveness, and centrality to mission that results in a self-assessment report and action plan, APR committee review report, and briefing to senior leadership.

Unit/Program Action Plan: Plan based on the self-assessment conducted in an academic program review that describes the direction and goals (including benchmarks and timelines) for the next 4-year review period.

Roadmap to Evidence (RTE): NlU's online repository of direct and indirect evidence of student learning and compliance with Middle States Commission on Higher Education and Joint Professional Military Education accreditation standards and requirements.

*Reasons why course grades are considered an indirect measure: 1) course objectives can cover multiple outcomes, meaning there is no direct measure to a specific outcome; 2) a course grade is the average of the overall work/performance and may include adjustments not related to learning outcomes (example: credit or penalty based on attendance).

26

Appendix B: Direct vs. Indirect Measures of Assessment

Direct Measures Indirect Measures

Course • Course and homework assignments . End-of-course/program surveys • Examinations and quizzes • Test blueprints (outlines of the concepts and skills • Standardized test covered on tests) • Term papers and reports . Percent of class time spent in active learning • Observations of field work, internship • Number of student hours spent on service

performance, service learning, or clinical learning experiences • Number of student hours spent on homework . Research projects • Number of student hours spent at intellectual or

• Class discussion participation cultural activities related to the course • Case study or scenario analysis • Grades that are not based on explicit criteria . Rubric (criterion-based rating scale) scores for related to clear learning goals

writing, oral presentations, and performances . Artistic performances and products . Grades that are based on explicit criteria related to clear learning goals

• Quick checks during class • Pre-test/Post-tests • Competitions • Juried evaluators

Program • Capstone projects, theses, exhibits, or • Focus group interview with students, faculty performances members, or employees . Pass rates or scores on licensures • Registration or course enrollment information

• Student publications or conference presentations • Department or program review data . Employer and internship supervisor ratings of • Job Placement students' performance • Employer or alumni surveys

' Student perception surveys

' Proportion of upper-level courses compared to the same program at other institutions

' Graduate school placement rates

Institutional • Performance on tests of writing critical thinking, • Locally-developed, commercial or national or general knowledge surveys of student perceptions or self-report of

• Rubric (criterion-based rating scale) scores fo r activities (e.g. National survey of Student class assignments in core course, or other courses Engagement) required of all students • Transcript studies that examine patterns and

• Performance on achievement tests trends of course selection and grading • Explicit self-reflections on what students have • Annual reports including institutional

learned related to institutional programs such as benchmarks, such as graduation and retention service learning (e.g. asking students to name the rates, grade point averages of graduates, etc. three most important things they have learned in the program, not focus groups)

27

Appendix C: Student Surveys

About Surveys

Surveys are an essential component of NIU's institutional assessment plan that allows NIU to collect indi rect evidence of student learning. NIU surveys provide students and faculty with an opportunity to tell us what is going well and what should be improved. This input provides the data necessary to analyze, maintain, and improve t he University's institutional effectiveness, academic programs, and student services.

NIU uses the CoursEva l survey tool for online survey processing. See Appendix D: Using CoursEval to Access Survey Results for more information about this tool.

The IE manages university assessment processes. If you have questions or concerns regarding NIU surveys

and survey reports. Contact IEat:NIU_Surveys@dodiis.mil.

NIU Survey Schedule

Weekly Surveys

• President's Lecture Series {PLS) Survey: The PLS Survey captures student satisfaction with the lecture and lecturer and collects additional information used to improve the quality of the lecture series. This survey runs weekly for the duration of the Fall, Winter and Spring quarters.

Quarterly Surveys

• End-of-Course (EOC) Survey: The EOC Survey provides students with the opportunity to share their course learning experience. NIU uses the data and information to help understand how well a course was presented, how it enabled student learning, and how the course may be improved to better meet learning objectives. This survey runs quarterly at/near the close of each course.

Annual Surveys

• Orientation Survey: The Orientation Survey captures student satisfaction with student on­boarding and orientation. NIU uses the data and information collected to improve admissions,

orientation, overa ll university processes and information services. The survey is open during the NIU Fall orientation and closes at the end of the August.

• Research Fair Survey: The Research Fair Survey captures student satisfaction and interest with the NIU Research Fair. NIU uses the data and information collected to improve the planning and execution of future Research Fair/Topics. This survey is open annually during the Research Fair.

28

• Methods & Research Workshop Surveys: The Research Workshop Surveys are designed to capture student interests at the Workshops. NIU uses the data and information collected to enhance workshop planning and experiences. This survey is open annually during the Research Workshops.

• End-of-Program (EOP) Survey: The EOP Survey provides degree-seeking students with an opportunity to share their views about their university experience and their satisfaction with their degree program and their views about the various services offered by the University. This survey is open annually from May through July.

Biennial Surveys

• External Stakeholder Surveys: External Stakeholder Surveys collect data and information from IC leaders, University alumni, and incoming students that help ensure that NIU is fulfilling its mission. The surveys also provide insight into what the Institution is doing well and opportunities for enhancement.

• Additional Surveys: Leader and Alumni Surveys are run every 2 to 3 years. The Incoming Student Survey runs annually as part of the Orientation Survey.

Other Surveys

• Faculty Orientation Survey: The Faculty Orientation Survey captures new faculty experiences and satisfaction with the onboarding and orientation processes. This surveys runs as needed throughout the year following the end of a faculty orientation session.

Faculty Access to Survey Results

Survey results are available for Faculty review after surveys close. Faculty members receive an e-mail notification announcing that reports are available that includes login information for the survey site.

E-mail notifications are sent to the faculty member's e-mail address on file in the Registrar's Jenzabar System (the initial unclassified or personal e-mail address provided during faculty setup). If there is no e­mail on file, the Survey Management Team may update the system using your dodiis.mil account. If you do not receive an e-mail notification, the Survey Management Team does not have an e-mail address on file and/or the Team could not find a dodiis.mil account for you.

NOTE: Please contact the Survey Management Team at: NIU_Surveys@dodiis.mil if you want your survey results to go to a different e-mail address.

Survey Results Access:

• Faculty: Individual End-of-Course (EOC) survey results are available to the faculty member assigned as the instructor for the course.

• Lead/Point of Contact for the Research Fair, Research Workshop, etc.: The respective survey results are available to the lead/point of contact.

29

• Faculty Department Chairs: EOC survey results for faculty members assigned to a department are available to the Faculty Department Chair. Some reports will automatically show in your Report queue. Others may need to be manually e-mailed to you by the Survey Management Team.

• Program & Center Directors: EOC survey results for program courses are available to the respective Program Director. Academic Center Directors have access to all courses taught at their center, using the course section number.

• Deans & Associate Deans: EOC survey results for degree courses and results from shared degree courses (i.e. graduate core courses) are available to all Deans and Assistant Deans. Deans also have access to the EOP survey results and all EOC and EOP summary results reported to the NIU Provost and President.

• Outreach Office: PLS survey results are available to the Vice President and Deputy Vice President of Outreach and the PLS Manager.

• NIU Leadership: A summary of EOC and EOP survey results are available to the NIU President, Provost and Chief of Staff. A summary of EOP survey results are reported in the annual NIU FACTBDDK.

Survey Results Utilization:

• May be reviewed by NIU Leadership, Deans, and Program Managers/Chairs who make course, program, teaching, and faculty decisions.

• May be incorporated into the Institution's accreditation report of other critical assessment reports, and made available for future data requests or publishing. Reports are edited to maintain anonymity and ensure they are unclassified.

• Are stored and used as evidence of NIU's assessment processes in accordance with Middle States Commission on Higher Education {MSCHE) and Joint Professional Military Education {JPME) assessment standards.

Survey Reports:

IE provides specialized reports as requested. Requests should be submitted on the data request form. Requests should specify the information requested: course{s), academic year(s), and quarter(s). They should also include specific format requested (i.e., average scores, percentages, etc). All report requests must be signed or approved by the respective Dean or Associate Dean. The following form should be utilized.

30

Survey Data Request Form

The below information must be included in the email request OR a scanned copy of the completed form must be sent to the Surveys Team at: NIU Surveys@dodiis.mil

I. Description of Request including course number(s): ---------------

II. Timeframe of Data Requested. Please include a date range if applicable.

AV

Ql _ Q3_

Q2 _ Q4_

Ill. Numerical Data and/or Comments Requested?

Data

Comments

IV. If Numerical Data is requested, please select the way you would like it presented.

Percentages _

Numbers

V. Requester Name-- --- --------

VI. Approval for release of data. Signature of Dean or Associate Dean required.

VII. Date needed ----------

Please allow a minimum of a week to complete data requests. Someone from the Surveys team will be in touch if they require any clarification regarding the request

All requests and any questions should be emailed to NIU Surveys@dodiis.mil

IE received date ----------~

31

Appendix D: Using CoursEval to Access Survey Results

NIU uses the CoursEval survey tool for online survey processing.

View ing Your CoursEval Results:

l. Log into the NIU Survey Site (mycourseval) at:

https://p3.courseva I. net/etw I ets/et.asp ?nxa pp id= HWl&nxm id=sta rt&li=t

2. Enter your username and password and click Sign In. The survey site will open the "mycourseval" page.

3. Select the Reports icon from the mycourseval menu.

o Click on Recent Reports to see your course survey results.

o Click on Evaluation Reports or Individual Reports to see historical reports.

4. Select the course results you would like to view from the courses listed and use the options ava ilable on the screen. The options vary depending on which 'Reports' menu option you choose to view or print.

Viewing Departments, Degree, Program, or Center Results: (Deans and Directors):

l. Log into the NIU Survey Site (mycourseval).

2. Select Admin from the mycourseval menu.

3. Choose the course results you would like to view from the programs listed under Department.

4. Click the # Reports for that Department.

5. Select the course results you would like to view and click on the Report icon to the right of the row.

NOTE: If a null symbol appears on top of the Report icon, there are no results available.

View ing Historical Program Course Results:

l. Click Reports on the menu at the top of the screen.

2. Click Eva luation Reports.

3. Use the menu fi lters at the top of the screen to locate desired reports.

NOTE: Academic Year 2015 (and half of Academic Year 2014) survey results are not in CoursEval. PleasecontactNIU_Surveys@dodiis.mil to have these surveys e-mai led to you.

32

Printing Survey Results:

From the IE browser:

a. Click on View Report to open the .pdf.

b. Right click anywhere on the .pdf and select Print from the list of options.

From the Mozilla Firefox browser:

a. Click on View Report to open the .pdf.

b. Click on Open Menu icon located at the top far right of the screen far right (This icon looks square with 3 lines inside of it).

c. Click Print from the list of options.

Things to Know about Student Surveys

Survey responses are anonymous. The survey ManagementTeam can view who has or has not completed a course survey, but NOT individual student responses. Composite results are reported to faculty for each course. Course results are also aggregated at the program level and presented to NIU Leadership.

Student survey invitations are sent to their personal {or unclassified) e-mail address provided during registration and admission. Students must contact NIU_Surveys@dodiis.mil if they do not receive their survey invitation(s) or would prefer the Survey Management Team to use a different e-mail address than the one provided during registration.

End-of-Course surveys are launched around the conclusion of the course. Students are required to complete a survey for each course in which he/she is enrolled. Failure to complete surveys may result in the withholding of all grades by University Deans.

End-of-Program surveys are launched at the beginning of the summer quarter and close prior to graduation. All degree-seeking students are invited to participate in the survey regardless of graduation year. Survey participation is mandatory in the academic year in which the student is graduating. Survey completion is required before a student/graduate can check out of the University.

President's Lecture Series (PLS) attendance is required for full-time students. Students are strongly encouraged to take the PLS Survey even if they have not attended the lecture. We understand there are various reasons why a student may not attend a lecture. There is an indicator at the tqp of the survey students can check to indicate they did not attend the lecture. Checking this indicator and submitting the survey removes the survey from the student's survey queue.

33

Appendix E: Evaluation vs, Assessment

Evaluation: The process of making a judgment about the quality of student work for the purposes of

determining grades and identifying level of performance. Evaluation is part of the assessment process, by

systematically gathering, ana lyzing and interpreting the data/evidence.

Assessment: An ongoing process aimed at understanding and improving student learning. Assessment

helps faculty understand how well their students understand course/program topics, lesson and objectives.

The Assessment Process:

Steps involved in the assessment process include:

• Establishing students learning outcomes for the course

• Systematically gathering, analyzing, and interpreting evidence on how well student's learning

matches:

o Faculty expectations for what students will learn, and

o Stated expectations/learning outcomes

• Faculty members should use this evidence of student learning to:

o Provide feedback to students about their learning

o . Adjust teaching methods to ensure greater student learning

Examples of evaluating student learning against student learn ing outcomes:

• Exams

• Quizzes

• Papers

• Homework Assignments

• In-class Activities

• Class Discussion

• Briefs

Examples of assessment activities for assessing expected learning outcomes.

• Non-graded quizzes

• Reaction papers

• Application papers

o What is the most important concept from this class?

o Identify their questions related to lesson

• End of course surveys

34

Appendix F: Rubrics

Rubrics are a tool for assessment of performance. They provide a scoring guide used to evaluate the quality of st udents' work and can be used for a variety of assignments. Rubrics should contain evaluat ion

criteria, definitions for t hose criteria, and a scoring strategy. A rubric can help define expectations of

quality around a task and help ensure consistent criteria for grading. They can also save t ime in grading.

Scoring rubrics include dimensions on which performance is rated, such as:

• Criteria - t raits that serve as the basis for j udging student response

• Descriptors - define or clarify the meaning of each cr iteria

• Levels - scale of values to rate each criteria

• Standards - spells out performance required for each level

Rubric Examples

Assignment Rubric

Course Content Excepcional Good f air/Adequ11e Limiced Poor/Un 1 ccepca ble Points Points

4 l 2 I 0 ..\\•aila Awardtd 9~100% 8~9·/. 7~79°/. 60-69% ~S9% ble

Allention 10 Follows and exceeds Follows Follows some but Gestures toward Disregards instructions CXJ>l..-Ctations noted in instructions not all instructions instructions but instructions and instructions demonstrates little assignment comprehension or

competency

Creative Demonstrates a high Shows some Shows minimally Demonstrates very Shows no Thoughlf degree of originality, origmality, acceptable little creative original thought; l'roblcm- insight, and/or insight, and/or originality, insight, thought or insight: all second-hand Solvi ng problem-solving skill problem-solving and/or problem- consists mostly of ideas

skill solving skill second-hand ideas

Thes is Thesis statement is Thesis statement Thesis statement Thesis swtcmcnt No thesis statement specific, significant. arguable and minimally s implistic or statement

arguable, and easy to easy to find acceptable but incomprehensible find somewhat general

or obvious

Development Thoroughly and Devdops and Inconsistently Inadequately or Docs not make a and Support insightfully explores, supports key develops and ine lTectivcly meaningful

explains, and supports points supports ideas explains and a1tempt to each idea defends idem; explain or

support ideas

Source use Demonstrates Most source Source material is Source material 1s No use of source compr~hension of all material is not consistent ly rard)' appropriate material source material: appropriate, but appropriate or and/or rarely integrated sources some is not fully integrate into the integrated into the appropriately and explained or paper paper cfTcctivcl}' integrated into

the paper

Clarity Difficult material is Most ideas are Wordy: some points Unclear and Largely made clear and presented require rereading to dilTicuh to incompr.:hensible presented in academic clearly, but understand fully understand language sometimes too

simplistically

35

Thesis Rubric

-D'd I not meet reau1rements -- u stan ? s b d d ar - tan ar - ro 1c1ent 3S dd 4Pfi' :'\ot l\lccting Substnndnrd Stnndnrd rroliekn1 l\1n:dmum Requirements Possible

Points Research question or Research question that is Research question or Novel question or probknl that is trivial, not significant, is often problcin that tends to be addresses nn important weak, unoriginal, not highly derivative, is on!)• sntall and traditionnl; is question or problem;

' related to the conte.'>t of tangentially related to the ne.'>t step in a clcnrly stntcs the Research the IC, intelligence or the context of the IC, research program (good problcn1 and \vhy it is

question nalionn\ security issues. intelligcnci: or national normal science), is important Question is security issues; is nnrrow clearly related the perfectly nligncd \Vith in scope conte.,.t ofthi: IC. the conte.'>t ofthc IC.

intelligence or national intelligence or national security issues security issues

Lacks originality. Workma11like, a11d Clear, coherent, concise, Original insightful, insight, thoughtfulness generally coherent; creative, thoughtful: pcr.mnsive. ruul clarity; onen loses rarely insightful: sometimes insightful; sophisticated, !llld 4 coherence; may rnmblc; argument may be has a comprehensive and thoughtful: exhibits has n weak, inconsistent, coherent or coherent argument Is mature. independcn\

TI1csis sclf-contradictory, comprehensive but not structured nround a thinking; nrgumcnt is Development unconvincing. or invalid bolh. Is only somewhat research question. focused, logical.

argument. Is not structured around a rigorous, and sustained; structured nround a research qucslion connects components in rescruth question. a seamless \\·ny. Is

structured around a research question.

Relics on inappropriate Uses standnrd methods; Dcmonslrnles technical Original, unique, or or incorrect methods; data arc son1ctimes competence; uses OtilCT\l'iSc highly data are nawcd, wrong, incontplcte appropriate (St!llldnrd) crrectivc research 4

Research or false theory, methods, and design; uses or develops l'l-le1hodology techniques: new tools, ntethods, or

approaches; has rich data fron1 multiple sources

lnfommtion is presented >lot enough evidence has !las sufficient Has n great deal of •I but it is mostly not been provided to nnswcr infom1ation to support information from related to answering the 1he research question. the main assertion of1hc concrete or statistically resci!TCh question or Information within research question, but valid sources. Research

Oma there is no data to back thesis is only partially contains some question is fully up tl1c research question rclntcd 10 the research extraneous material or answered, \l'ith no answer question. leaves some potential unc.,.plored issues or

questions unexplored. connections. All of material is related to research question.

36

\Vrong, inappropriate, Unsophis1ic.11cd Obtains solid, expected Comprehcnsi\'e rurnlysis, incoherent, or confused analysis: docs not results or ans\vers: complete. sophisticated mmlysis; includes resulis explore all possibilities: n1isscs opportunities to (findings based on 4 lhat nn: obvious, already n1isscs connections; has completely explore nnal}'Sis and follows know. unexp!Bincd, or predictable results. interesting issues and logie.1lly fron1 research,

Anal)·tie, n1isinterpreted; has Exhibits gaps in critical connections. li'\hibits thesis includes Critical, & unsupported or thinking and little solid critical thinking. contmtlictoiy evidence); Creative e.\aggcrnted crcati\•ity. and/or so1nc degree of results arc significant: Thinking interpretation. Exhibits creativity. f'...lultiple ideas conclusion tics the \\'Ork

little in the \1•ay or are nt least son1cwhat together. E.xhlbits solid critical or creative syntl11:si1.cd. critical thinking and thinking. demonslratcs creativity.

S}nthcsizes mulliple concepts and ideas.

Docs not understand or Displa)'S 11 narrow Shows understanding Displays a deep misses relevant understanding or the and mastc1y or the understanding ora literature; docs not field: reviews literature subject mailer. massive amount of ' understand theory well; adequately·; knows the complicated litcrnturc: thco1y is missing or literature but is not exhibits command nnd

Tl100I)' \\Toni;;: docs not critical or it or docs not authority O\'Ct the understand basic discuss what is material: is theoretically

Knowkdi;;c concepts, processes, or important~ demonstrates sophisticmcd and shows convcnlions ofthe understanding of theory a deep understanding of discipline al a simple level, and theory. Author

lheOI)' is minimally 10 challenges the literature. competently applied to lhcproblcm

The outcome or rurther The outcome or rurthcr The outcome or rurther The outcome or rurt11er applianions of the appliciuions of the applications orthc applications orthc research is of no interest research m11y be of research are or interest to research arc of interest to 4 to intelligence interest 10 a nnrrow a moderate segment of a larger con1munity 11nd communily: docs not segment of the the comn1unity; unlikely changes the \\'llY people contribute to the field community: nmkcs a to change the \I'll)' think; pushes tl1e

Impact small contribution to the people think: makes 11 discipline's boundaries field modest contribution to and opens new areas ror

the field but docs not research; can open it up, inight extensively change 1hc ntodcratcly change the beha\'ioror altiludes of behavior ornttitudcs or proressionals in the professionals in the community community

Points awarded for the research proposal out of 28 points available: ___ _

37

Writing Rubric

I -Did not meet renuirements 2-Substandard 3-Standard 4 - Proficient Not i\leellng Subs1nndatd Sl11ndnrd Profkknt l\h1dmum Jtcquln:n1cnl~ Possible

Points Style Content not11ppropria1c ror Content is inappropriate Content is appropriate ror Content is appropriate for 4

audience for audience audience audience Style is inappropriate for Style is inappropriate for Style is uppropria1e for Style is appropriate for audience audience audience audience Content does no\ sausf)' Some content im:le\'llnt to Content m~'l:ts assignment Content meets assignment assignment assignment rcquircmcn\Sfpurpose rcquirementsfpurpose n:quircments/purpose rcquin:mcn\Sfpurposc No th~'lll)'fpt:rspt.-cth·e Does notaclmowledge Aniculat~'S some elements Addn:S,So:s howthdr utilized thcoiy/pcrspei:tive that of their thcoiy/pcrspectivc thcoiy/pc rspei:t ive

shapes argument or influenei:s their analysis or 11na!ysis argument

Uses inappropriate Uses some inappropriate Appropriate language for Appropriate language for language foruudieneo language for audience audience audience

Structure and llas no thesis statement 'Vcak thesis statement Effecti,·e thesis statement Effective thesis statement 4 Organir.uion

No organization uround Some organization around /I.lastly organiicd mound Fully orgnnb:cd around thesis statement thesis statement thesis statement thesis statement No organi1.ation nround raragraphs not fully Paragraphs mostly \\'ell· Fully dc,·eloped, coherent thesis statement dc\·clopcd, cohen:nt developed. coherent pamgraphs Paragr.iphs not developed, /l.finimal connection /l.linor problems \lith flow Organiicd into logic:il incoherent between ideas/concepts ofidcnstconecpts 11ithin flow of ideas/concepts

\\ithin [Xlragraphs paragraphs within pamgmphs Laci: of conclusion/ not /l.linimal eonncct1on /l.fi:aningful conclusion Meaningful conclusion consistent 1\·/ lho:sislboc.ly between thesislbodynnd

conclusion No section divisions Appropriate section Appropriate ors;mi1.ation Appropriate orgnnir.uion

divisions \\•llhin sections within sections lnapproprfatc content in Somcwha1jumblcd conient Appropriate content in Appropriate content in scrtions m sections ~':!eh section each sec1ion No discussion/no Unde~eloped Some1~ha1 undevclopi.'tl Developed diSCUS,Sion and conclus1on discussionfconclusion discussion I conclusion conclusion

Content, Thesis statement not Thesis stntc1ncn1 backed Thesis statement bac~l'tl Thesis statement backed 4 Argument, b.ickcd by coherent by inad1:quatearguments by ad1:quate arguments by ample and effective

and Evidence argumcn\5 nrgumcnts Argument(s) contam(s) no Arguments contain Arguments contain Arguments contain nm pk evidence i1mdcquate sources of adequate sources or and effective evidence.

C\'idcnce evidence Nof1nad1:quate numb<:r of Sources lack Sources ure mostly Sources :ire credible nnd citations ercdibility/rclc\'llllce cn:dible and relc111nt n:le\'lln\ No support/support is Suppon not relevant 10 Support is mostly fully Support is fully intt:gratcd irre[e\'iUlt tople integralcd Nofinadcqua\C numb<:r or Improper fomJat for Minor problems 11i1h Consistent format for eitaLions citations formal for citations chat ions Images/Maps not rc!atcd to Images/Maps no\ Images/Maps integrated Images/Maps integrated the paper integrated 1\tll and do not wdl and add to the paJl<!r well and udd to the p.iper

add to the paper Mechanics Writing errors make \Vrilmgerrors interfere Some writing errors - do Veiy few errors/no errors 4

crr~-ctive communication 11ith effective not interfere with crrcctivc in punctuation, grnmm~r. impoS,Sible communicauon communication spelling or usage (between (over 19 writing or (IJ and IS 1vriting or {7 and 12 writing or I and 6 writing or grammar errors) grammar errors) gramm~r errors) grammar errors)

'Vnting unclear and \Vriting unclear and Wt1tmg is generally clear Writing is clear and impn:eise im"n::cise and precise concise 'Vriting inerrci:1i,·c \Vriting inclfl-cti\'C Writing is mostly effective 'Vriting is skillfu!ly and

clfceti\-e[ ·era fled Does not meet \Vritingmos!ly meets Meets assignment McetS ussignment requirements for formal assignment requirements requirements for fomrnt requirements for format and kngth for formal and length and length and length

38

Appendix G: Faculty Peer-to-Peer Review Form

National Intelligence University

Peer Review of Classroom Instruction

INSTRUCTOR: Reviewer:

Course:

Date: Time:

Observations of teaching STYLE/DELIVERY of content and interaction with students:

Observations of STRENGTHS and best practices of the instructor:

SUGGESTIONS for the instructor and/or the seminar:

Please send a copy of the completed form to the reviewed instructor and to Institutional Effectiveness.

39

NIU Form MCE-2 Approved for use: 20151117

I. Basic Information:

Course:

Quarter/Year:

Instructor:

Contact:

Office Hours:

Class Time/Location:

Textbook:

Appendix H: Course Syllabus Template

NATIONAL INTELLIGENCE UNIVERSITY

SYLLABUS TEMPLATE

(Course prefix, catalog number, and title]

[Quarter, AV]

[Name]

[Office, e-mail, phone]

[scheduled + by appointment?]

[Day, time, classroom]

[Title, Author, Edition]

II. Course Description/Contribution to Mission:

Course Description: (must be consistent with the current NIU catalog)

Contribution to Mission: (must be consistent with the current NIU catalog)

Ill. Course Goals and Learning Outcomes:

Goals: (must be consistent with the current NIU catalog)

Primary Learning Outcomes: (must be consistent with the current NIU catalog)

Joint Learning Outcomes: (must be consistent with the OPMEP Instruction)

40

IV. Course Format and Procedures:

• Class attendance and participation:

• You will meet the objectives listed above through a combination of the following activities in this course:

o Attend

o Complete

o Participate

• Late Work:

V. Course Requirements:

• Course readings:

• Course materials?

• Use of the course Blackboard or other web site?

• Resources?

VI. Evaluation Procedures/Grading Criteria:

VII. Course Schedule/Outline/Calendar of Events:

VIII. Academic Integrity (optional):

Each student in this course is expected to abide by the NIU Code of Academic Integrity. Any work

submitted by a student in this course for academic credit will be the student's own work. [Optional: For

this course, collaboration is allowed in the following instances: list instances.]

You are encouraged to study together and to discuss information and concepts covered in lecture and

the sections with other students.

You are expected to credit properly and accurately the source of materials directly cited or indirectly

used (i.e., paraphrased) in any oral or written work. All students' work shall be their own, unless

otherwise properly noted.

You may not use entire papers or substantive selections of a paper from one course to complete work

for another course or courses.

41

A grade of F will normally be assigned for any work proven to be undertaken or performed in violation of

academic integrity. All instances of alleged violations of academic integrity will be handled in accordance

with published NIU policies.

IX. Attendance (optional) :

Students are expected to attend all scheduled class sessions. Students missing more than one session

face, at the discretion of the faculty member, penalties ranging from the lowering of the final grade to

fa ilure in the course. A student who misses three or more sessions and does not withdraw faces remova l from the course and a fai ling grade.

IX. End of Course Survey (optional but recommended}:

Students are expected to complete a course survey near the end of this course. Course surveys are an

essential component of the universities educational assessment process. Failure to complete your end­

of-course survey may result in the withholding of all student grades by the Dean. Follow university

instruction on accessing the university survey site (myCoursEval) and completing course surveys.

Contact NIU Surveys@dodiis.mil if you have any questions regarding course surveys.

X. lnclusivity Statement (optional) :

We understand that our faculty and students represent a rich variety of backgrounds and perspectives.

NIU is committed to provid ing an atmosphere for learn ing that respects diversity. Whi le working together to build this community we ask all members to:

• share their unique experiences, values and beliefs

• appreciate the opportunity that we have to learn from each other in this community

• value each other's opinions and communicate in a respectfu l manner

• keep confidential discussions that the community has of a personal (or professional) nature

• use this opportunity together to discuss ways in which we can create an inclusive environment in this course and across the Intelligence and National Security communities

42

Append~x I: Outcomes Asse1sment ~orm

The Outcomes Assessment Form was developed to capture information about the type of meeting,

purpose of the meeting, alignment with strategic, MSCHE, and JPME goals and objectives, evidence,

discussion and analysis conducted, and then the decisions, outcomes or recommendations that come out

of that meeting. There is also a space for any follow-up activities that needs to be conducted.

Th is form is recommended to be used for meetings of groups evaluating curriculum, goals, objectives, and

other significant events impacting the way NIU conducts business.

By utilizing this form, the documentation of decisions and changes will be clearly spelled out, and the evidence used to come to that conclusion.

43

Outcomes Assessment Form

Date:

Assessment Unit/Team/ Working Group

Ptoject/ASS~$-sment

Purpose I Objective I Goal:

Alignment: (MSCHE & JPME Standards/Strategic Gaols & Obj.)

Analysis

Data / Evidence:

Discussion I Analysis:

Results Decisions / Outcomes/ Recommendations:

Next Steps/ Implementation Plan (Post-Review Follow-up):

44

Appendix J: Pre-Public Release Review Process

There are specific procedures for assuring compliance with the agency policy for Prepublication Review of Information Prepared for Public Release. These procedures identify two key processes: those of NIU and those of the Office of Corporate Communications (OCC).

References: DIAi 5400.005, Prepublication Review of Information Prepared for Public Release, 19 November 2013.

Definitions:

• OFFICIAL: Material prepared and being released as part of one's official duties as a DIA employee or contractor.

• NON-OFFICIAL or UNOFFICIAL: Material prepared by a DIA employee or contractor as a private individual and who is not acting in an officia l capacity for DIA, DoD, or the U.S. Government.

Application:

• Pre-public release review applies to all DIA/NIU personnel, civi lian, military, and contractors, regardless of rank.

• All unclassified products that are intended for public release, whether the informat ion is intended to be an official or non-official product, must be submitted for prepublication review. This includes publications, speeches, library contributions, web biogs, etc.

• If you've been asked to speak to the public (conferences, panels, alumni events, etc.) about DIA or your mission area, you are required to have OCC approval prior to accepting the invitation.

Responsibilities:

• Directorates and Departments must submit all unclassified official and unofficial information and products prepared and intended for public release to OCC Prepublication Review after review and approval by component's management chain of command and security.

• Individuals are responsible for filling out the Prepublication Review Form and obtain ing permission of directorates and specia l offices which have equities in the material.

o Official products intended for public release must be coordinated with those directorates and special offices which have equities in the material prior to submitting the product to NIU leadership for Prepublicat ion Review.

o DIA personnel who are requesting review of material prepared in a non-official capacity shall include a statement that the material submitted for review is derived from unclassified

45

information and is, to the best of the submitter's knowledge, unclassified and appropriate for public disclosure.

o Draft material prepared in a non-official capacity must be submitted for review at each stage of development prior to disclosing such information to anyone, such as a publisher, editor, co-author, or other member of the public, who does not have the requisite clearance and "need-to-know."

• All employees must ensure that OCC Prepublication Review has been completed prior to disclosing such information to anyone who is not authorized by DIA to have access to the material.

Process:

• Individual requesting public release of information, must submit both the document which you are requesting release and the completed Prepublication Review Request form on JWICS. The form can be found at:

https://www.dia.ic.gov/homepage/occ/policy/prepub review form Sep2015.pdf

• Contractors must also clear their submissions through the COR.

• Prepublication Review Requests will be submitted via JWICS to the GG-15 Division Chief (or Higher) chain of command for processing and review.

o Requests from faculty or students will go to the Dean of the appropriate School or thesis advisor's School for NIU review.

o Requests from research faulty will go to the Director of the Center for Strategic Intelligence Research.

o Accreditation, Assessment or IE requests will be submitted to the Director for IE.

o Other requests will go to the Director of Office of Research.

• The Division Chief will determine if the information is Official or Unofficial, and will submit the proposed publication to Security for review and Foreign Disclosure review {FDO) as appropriate.

• Upon approval of security and FDO (if required), the Division Chief will submit the request to DCC for processing.

Academic Issues:

• In the interest of academic freedom and the advancement of national defense-related concepts in the DoD academic environment, students and faculty members of NIU may prepare academic papers and manuscripts for open publication. They may express their views in such materials as long as those views do not disclose classified or OPSEC critical information or jeopardize DoD interests and the author accurately portrays official policy, even if the author takes issue with that policy.

• Papers or other material prepared in response to academic requirements do not have to be submitted for review when they are not intended for release outside the academic institution.

46

• Papers or other material must be submitted for review and clearance if it is intended for public release or made available in libraries to which the public has access.

• DIA personnel teaching at a non-DoD schools must submit their written instructional material for review and clearance if it includes classified or "for official use only" information. It is incumbent on the individual to ensure any classroom discussion or anecdotes of one's experiences in or knowledge of the Intelligence Community or DoD do not reveal classified or otherwise sensitive information.

47