UNITED NATIONS CHILDREN’S FUNDREQUIRES

International Consultant to conduct an Evaluation of the Project “USING TEXTBOOKS IN DAISY FORMAT - Primary school teacher training in using audio textbooks”

Minimum Requirements: An advanced university degree preferably in Social Sciences, Education or other relevant field; At least 5 years of documented evaluation expertise and experience; Consultant with expertise in the area of education with a focus on inclusive education and children with

special educational needs; Proven knowledge on the education system in Montenegro, knowledge on the region is an asset; Excellent report writing skills; Good communication and presentation skills; Excellent written and spoken English; Ability to keep with strict deadlines; Knowledgeable of UN Evaluation policy; Experience in working with UN / UNICEF (obligatory).

Terms of Reference and P11 form can be found on the website www.unicef.org/montenegro or picked up at UNICEF Montenegro office.

If you are a dynamic person who would like to contribute to delivering of long-lasting, positive changes for Montenegro’s children, please apply with a CV, P11 form, financial offer and cover letter by email, mail or fax marked “Evaluation of the Project USING TEXTBOOKS IN DAISY FORMAT” to the following address:

UNICEF, UN Eco House, Stanka Dragojevica bb, 81 000 PodgoricaFax: +382 20 447 471

Email: podgorica@unicef.org

Closing date for applications is December 21, 2014.Only short listed candidates will be contacted for interview.

U N I C E F I S A S M O K E F R E E E N V I R O N M E N T

1

Terms of Reference

Consultant to conduct an Evaluation of the Project “USING TEXTBOOKS IN DAISY FORMAT - Primary school teacher training in using audio textbooks”

1. Context

Since becoming an independent State in 2006, Montenegro has officially joined the international community and in doing so a new blueprint has emerged for nation building. On 28 June 2006, it became the 192nd Member State of the United Nations, and on 11 May 2007, the 47 th Member State of the Council of Europe. Montenegro presented its official application for membership of the European Union in December 2008 and two years later it became a candidate country for EU membership. Like its neighbouring countries, it has a rich cultural heritage, a diverse history and is well poised to enjoy integration with the European community.

Montenegro belongs to a group of upper Middle Income Countries with GDP per capita EUR 5,063 in 2012. Montenegro experienced rapid economic growth ranging between 8.6-10.7 percent of GDP in 2006 and 2007 respectively. However, as a result of the global economic crisis that hit the country in the second half of 2008, the real GDP growth rate saw a steep decline. After slow and modest economic recovery in the period 2010-2011, Montenegrin economy returned to recession in 2012 with -2.5% real growth rate of GDP (Monstat, 2013). According to the latest estimates, the real growth rate of GDP in 2013 will be positive but modest (1.8%) (World Bank, December 2013).

The global economic crisis hit the economy of Montenegro hard having prolonged negative impact on well-being of citizens. The economic recession resulted in further increase of poverty rate in the country from 9.3% in 2011 to 11.3% in 2012. There are also significant spatial disparities in terms of poverty that represent serious impediment to the fulfilment of human and child rights. In 2012, poverty rate in the northern region was 18.3%, which was much higher than in the central region (7.9%) and southern region (9%) (Monstat, 2013). According to the Child Poverty Study in Montenegro (UNICEF, 2012), child poverty is concentrated in rural areas and in the north of the country with more than 3/4 of all poor children living in rural areas, and more than half of them living in the north of the country.

Delayed transition of Montenegro has had an adverse impact on the education sector which remains disconnected from the latest thinking and innovations, as well as from labour market needs - PISA results show that Montenegrin students lag behind their OECD peers by two years in terms of skills and competencies necessary for their future education and success in life and employment (50% of Montenegrin students have scored below basic functional literacy levels in math, science and reading).

Compromised quality of education also questions the quality of inclusion of children with disabilities (CwD). It must be emphasized though that the most visible results of the education reform are those related to inclusion of CwD. So far, the Government has adopted two Inclusive Education Strategies for the period 2008-2012 and 2014-2018 that are in line with the best international practices (the EU countries, above all Finland, then Great Britain, Israel, etc.) and rich domestic and regional experiences. The leading principles guiding these documents are good quality, accessible education for children with special education needs, in accordance with their interests, abilities and needs. Systemic efforts of the Government accompanied by UNICEF’s massive C4D campaign ‘It’s about ability’, resulted in a five-fold incremental increase in the number of CwD attending mainstream schools in the past 5 years.

However, CwD still do not have the same learning opportunities. It must be noted that there is no verified evidence of the noted inequalities affecting girls more severely than boys or vice versa. The Evaluation of Education Reform in Montenegro conducted in 2012 emphasizes that in terms of the legal framework and education policies concerning the establishment of an inclusive education system, Montenegro has made significant improvements, and a systemic environment for effective inclusion of children with special needs

2

has been created. However, there is a consensus between decision-makers, education professionals, teachers and parents that significant challenges remain when it comes to consistent implementation of adopted policies, specifically in relation to: enhancement of teacher training in this area (pre- and in-service), appropriate allocation of resources for the implementation of inclusion, inter-sectorial collaboration and activities, teaching material adapted to the needs of CwDs, etc.

In addition, there are no reliable estimates on child disability prevalence due to data gaps in all sectors. Child disability rate at the Census 2011 was 1.1%, while the MICS 2005 pointed to as much as 12.5% suspected prevalence for the 2 – 9 years old age group. According to MONSTAT data from 2013, 1399 children were recipients of child disability benefits, while the number of children with disabilities enrolled in primary schools in April 2014 was 1350.

With the aim of providing accessible and adapted teaching and learning materials for CwDs, in June 2013, UNICEF Montenegro Office, the Ministry of Education, Textbook Publishing Agency and the Resource Centre for CwDs initiated the first ever production of audio textbooks in DAISY format for children with visual impairments and children with learning difficulties. The initiative was implemented with the generous contribution of the Drama Academy, whose students recorder the books on a voluntary basis. Six audio textbooks/primary school readers in Mother tongue have been produced so far and approved by the National Education Council. This initiative is ongoing and the production team at the Resources Centre continues to work on making more textbooks accessible in DAISY audio format. However, further challenges are linked to scaling up the efforts, regular updates of the material and ensuring their effective use in classrooms.

2. Project to be evaluated: “USING TEXTBOOKS IN DAISY FORMAT - Primary school teacher training in using audio textbooks”

As a continuation of the initiative started in June 2013, the Project “USING TEXTBOOKS IN DAISY FORMAT - Primary school teacher training in using audio textbooks” aims to ensure that DAISY audio textbooks are effectively used in classrooms and that both children with and without disabilities are benefiting from improved teaching and learning environment. The project is being implemented by The Resource Centre for Children and Youth “Podgorica″ (Resource Centre) in close cooperation with the Ministry of Education, Textbook Publishing Agency and Bureau for Education, with financial support of UNICEF Montenegro. Total budget for this project is 7,553.00 EUR (UNICEF’s contribution: 6,563.00 EUR, Resource Centre’s contribution: 990.00 EUR). The project timeframe covers the period from September 2014 to June 2015.

Overall project objective is: Train teachers and parents to ensure quality teaching and application of reliable and modern learning methodologies for children with disabilities, as well as for all other children in 24 primary schools and the Resource Centre in Montenegro.

Specific objectives:

- informing teachers and parents about how innovative methods and modern technologies can be used to support the teaching process;

- training teachers and parents in how to use DAISY audio books and provide learner support to students/children;

- developing the capacity of teachers to use audio books in a way to support the quality and objectives of the teaching process;

- using audio books in the classroom, monitoring/testing student ability to listen attentively and truthfully represent the content to others;

- helping students to acquire knowledge from DAISY books as a modern teaching aid;- making an inventory of good practices, identifying lessons learned in using this book format for

future improvements in the teaching process;

3

- developing the training programme Using Audio Textbooks in DAISY Format and having it accredited by the Bureau for Education

Right holders of the Project “USING TEXTBOOKS IN DAISY FORMAT - Primary school teacher training in using audio textbooks” are children with and without disabilities attending inclusive primary schools and their parents.

The Project was managed by the Editorial Team composed of representatives of the Resource Centre, Ministry of education, Textbook Publishing Agency and Bureau for Education.

The Theory of Change (ToC) for the Project has been developed after the inception of the Project with the aim to contribute to its evaluability and better define expected outputs, outcomes and impact. The ToC, as well as the Project Document, can be accessed here:

3. Rationale for the Evaluation

Montenegro is the first country in the world producing a set of DAISY audio textbooks for inclusive education. Similar efforts have been made in other countries, but not led by the main educational institutions seeking to make the use of the audio textbooks part of the regular teaching material and introduce them in all primary schools in the country. Also, this is the first time that DAISY audio textbooks will be made accessible and downloadable to everyone. As such, this project has an enormous potential for knowledge generation and compilation of lessons learned to be replicated elsewhere. UNICEF Headquarter Disability Section has recognized this potential and made available additional funds for evaluating this Project.

The knowledge generated by the evaluation should be used by: Project Implementing Institutions – Resource Centre, Ministry of Education, Textbook Publishing

Agency, as an important source of information for the further scaling-up of this initiative - more specifically to identify lessons learned during the implementation, understand the impact of the USE of DAISY textbooks in the teaching and learning process in classrooms, areas that require additional attention, potential barriers and bottleneck in implementation, etc.;

Other primary schools in the introduction of DAISY textbooks in the teaching and learning process; Associations of Parents of Children with Disabilities and Children without Disabilities representing

the interest of children to further strengthen their monitoring and advocacy efforts; UNICEF Montenegro Office for future programming and support to strengthening the inclusiveness

of education for children with disabilities, support to scaling-up of the same initiative and similar initiatives;

UNICEF Headquarter Disability Section - for future programing, support and replication of similar initiatives in other countries.

Given that this is the first time that the DAISY textbooks are being used in teaching and learning in Primary schools, The recommendations will help the UNICEF to ensure provision of the most efficient and effective assistance to educational institutions in Montenegro in defining the future direction of the reform process with a focus on inclusiveness of education.

4

4. Objectives of the Evaluation

Given that this is the first time that the DAISY textbooks are being used in teaching and learning in Primary schools, the main immediate purpose of this summative evaluation is to evaluate the final (end) results and achievements of the project in relation to the project theory of change and indicators from the Project document.

More specifically the evaluation objectives are to:1. Provide feedback to UNICEF Montenegro office and its national counterparts on the soundness

(defined as relevance, effectiveness, efficiency, sustainability) and impact of the Project approach in introducing the DAISY textbooks in 24 primary schools and the Resource Centre for the benefit of improved teaching and learning of CwD and children without disabilities, with the aim to:

a. Reveal good practices and gaps in approaches, b. Evaluate Project Impact following Project Document and Theory of Change.

2. Based on the experience from the Project implementation to extract general lessons learned and recommendations aimed at further enhancement of the initiative;

3. Provide the UNICEF Headquarter Disability Section with information on impact of the Project and potential for replication and scaling-up.

5. Scope of the Final Evaluation

The Project evaluation should cover the entire project implementation period (September 2014 – June 2015). The geographic coverage is 10 municipalities where the initiative is being implemented: Podgorica, Bijelo Polje, Berane, Pljevlja, Nikšić, Danilovgrad, Tivat, Bar, Kotor and Herceg Novi (that is 24 primary schools from these municipalities and the Resource Centre).

Project monitoring data as well as other data sources that should inform the evaluation are listed within the section No. 7 on Methodology and will enable assessment of the project achievements. The Project Summary is portrayed through the Theory of Change Table (attached above). This Theory of Change Table should be used as the main Reference point – together with the Project document, because it captures the activities undertaken, the expected changes they were to produce and the intended impact.

There are no major limitations anticipated in relation to this evaluation. However, given that at the time the Project Document was being developed, there were no plans to conduct a full-fledged evaluation of this initiative, the Document does not contain an exhaustive list of indicators or the log frame. To overcome this, a ToC has been developed to guide the evaluation process and allow for development of appropriate indicators to measure and establish the evidence of impact. Especially so, given that the project implementation period is fairly short and that the evaluation will be conducted towards the end of the project.

6. Final Evaluation Questions

The focus of the evaluation is articulated under 5 main evaluation criteria (relevance, effectiveness, efficiency, impact and sustainability)1, each accompanied with guiding questions for the evaluation. These are not exhaustive and should be clarified at the start of the work when developing the evaluation framework:

Assessing relevance/ To what extent is the Project responding to the needs of stakeholders and beneficiaries?

To what extent does the Project address underlying causes of exclusion of CwD form the education system and respond to the needs of CwD in inclusive primary schools?

1 The humanitarian criteria for evaluation (coverage, coordination and coherence) are not applicable for the Montenegrin context or the context of this Project.

5

To what extent is the Project design relevant vis-à-vis the overall Project goal and the achievement of its objectives in the given period of time?

To what extent are the Project design and its objectives relevant vis-à-vis national policies and strategies?

Was the Project designed according to international norms and agreements on Human Rights (HR) and Gender Equality (GE) and in line with national strategies to advance HR & GE?

Assessing effectiveness / To what extent does the Project meet the outcomes as defined by the Project log-frame and the Theory of change?

To what extent have the planned results been achieved to date (quantitative and qualitative)? To what extent and how has the project contributed to enabling teachers to use the DAISY

textbooks in the teaching and learning process? To what extent has the project contributed to strengthening the skills of parents to better support

their children in the learning process? To what extent has the project enabled better accessibility of learning material to children with

disabilities and learning difficulties? To what extent has the project contributed to better individual and group learning of children with

and without disabilities in selected schools? What were the major factors influencing the achievement or non-achievement of the Project

objectives to date? Has the project provided any additional (not directly planned by the Project) significant

contributions/outcomes?Assessing efficiency / To what extent did the management of the Project ensure timelines and efficient utilization of resources?

How well have the implementation of activities been managed? To what extent were activities implemented as scheduled? What management and monitoring tools have been used?

How well have the financial resources been used / were funds managed in a cost-effective manner / what is the correlation between funds utilized and outputs / results achieved / could the same results be achieved with less resources?

Did the project ensure co-ordination with other similar interventions to encourage synergy and avoid overlaps?

Assessing impact / To what extent has the Project increased system’s capacities to ensure that more vulnerable and excluded children benefit from community-services in a way which contributes to their social inclusion?

Is there early evidence of the project having contributed to better integration of children with disabilities in the teaching and learning process in selected schools as compared to period before the introduction of DAISY textbooks?

Is there early evidence of the project having contributed to improved dynamics of group learning between children with and without disabilities in selected schools as compared to period before the introduction of DAISY textbooks?

Is there early evidence of the project having contributed to improving the confidence and empowering the children with disabilities in individual learning as compared to period before the introduction of DAISY textbooks?

Is there early evidence of the project having contributed to improving the teaching and learning environment in selected schools as compared to period before the introduction of DAISY textbooks?

Is there early evidence of the project having contributed to changing the attitudes and knowledge of teachers, schools management and parents on the inclusive teaching and learning practices?

Is there an indication that the project results/contributions have affected girls and boys differently, and, if yes, why?

Assessing sustainability / To what extent are the Project outcomes achieved sustainable? Did the Project design include an appropriate sustainability strategy to support positive changes in

selected schools after the end of the intervention?

6

To what extent do the developed training programme, related manuals and guidelines for teachers, supervisors and parents provide a ground for sustainable and coordinated up-scaling of the project?

To what extent are national level mechanisms, strengthened for monitoring and quality assurance of introduction of DAISY textbooks in teaching and learning in Montenegro?

To what extent will the new knowledge and skills integrated into regular activities of educational professionals working with children with disabilities?

What is the level of ownership of the initiative within the Ministry of Education, Bureau for Education, Textbook Publishing Agency and schools and what are the prospects for further development of related interventions after the end of external support?

To what extent has the Project promoted strengthening of already existing partnerships and establishment of new ones?

In addition to the 5 main evaluation criteria, the evaluation shall also focus on assessing human rights-based approach and relevant cross-cutting issues. More specifically, it should look into the extent of the project outcomes’ contribution to achievement of children’s rights and how the project contributed to addressing key cross-cutting issues?

Does the project actively contribute to the promotion of child rights? To what extent and how the project ensures an equity focus? Does the project reflect gender mainstreaming issues? To what extent are HR & GE a priority in the overall Project budget and implementation? Was the design of the Project ethical? How was the balance of cost and benefits to participants

(including possible negative impact) considered during the Project implementation?

7. Methodology

The evaluation methodology will be guided by the Norms and Standards of the United Nation Evaluation Group (UNEG):(http://www.uneval.org/normsandstandards/indexs.isp?doc_cat_source_id=4)2.

Evaluability assessmentThe theory of change and data available allow for the assessment of the progress achieved and evaluation of the project impact and results.

Existing sources of information are assessed as the most reliable:a. Ministry of Education Information System (MEIS) – specifically disaggregated data on the

number of children with disabilities in primary schools; b. Data from the Resource Centre – number of direct and indirect project beneficiaries involved in

project activities and implementation, evaluation form from the participants of the trainings on the usefulness of the same, etc.;

c. Bureau for Education – data on supervisory visits to primary schools;

It is expected that the evaluation methodology and instruments used will enable gathering the information on the situation before and after the intervention, specifically through the use of qualitative data collected from in-depth interviews, focus groups and survey conducted on an appropriate sample.

Contact list of all for the project relevant stakeholders, project implementing partners and consultants will be provided to the evaluator once a contractual agreement has been made.

Approach

2 UNEG Norms: http://www.uneval.org/indexAction.cfm?module=Library&action=GetFile&DocumentAttachmentID=1491 UNEG Standards: http://www.uneval.org/indexAction.cfm?module=Library&action=GetFile&DocumentAttachmentID=1496

7

The overall evaluation approach to be applied shall rely on the Project theory of change and shall have an equity focus. It shall combine qualitative and quantitative data and apply data collection strategy that relies on primary and secondary data collection and non-experimental design.

The consultant will use the desk review to get familiar with the policy basis, relevant project documents and other means of verification / sources of information listed above.

The consultant should develop more precise evaluation work. In addition to interviews with the Editorial Team and representatives of the Ministry of Education, the

Resource Centre, the Textbook Publishing Agency and UNICEF, opinions of the following actors shall also be acquired and analysed:

Key stakeholders and project beneficiaries:- Associations of Parents of Children with Disabilities;- NGOs representing parents;- School Parent’s Councils;- Teachers;- School management;- Children with and without disabilities in selected schools;- Families of children with and without disabilities.

The guiding questions for the evaluation against defined evaluation criteria shall be further elaborated and used as a basis for development of the main data collection instruments (interviews and focus groups).

All the data gathered will be analysed by the evaluator/s. Triangulation of data will be used to increase reliability of findings and conclusions.

Special measures will be put in place to ensure that the evaluation process is ethical and that participants in the evaluation process can openly express their opinion. Also, if proved necessary, it will be ensured that relevant experienced professionals be present and provide support during the interviews/focus groups with children with disabilities. The sources of information will be protected, and known just to the evaluator/s. The Evaluation Team will ensure that the evaluation process is in line with UNEG Ethical Guidelines. The consultant has to ensure that it is clear to all subjects that their participation in the evaluation is voluntary. All participants should be informed or advised of the context and purpose of the evaluation, as well as the privacy and confidentiality of discussions.

8. Work Plan

Description Responsible TimelinePreparatory phase:Development of ToR UNICEF By 12th

December 2014Selection / contracting of consultant UNICEF By 26 December

2014Data collection:Desk review of the existing documents Consultant May of 2015

Logistics (arranging meetings / interviews) Consultant End May 2015Field visit to Montenegro (meeting / interviews with UNICEF, MoLSW, EU Delegation, key partners and stakeholders)

Consultant with the support of UNICEF

Beginning of June 2015

De-briefing meeting with Implementing partners and UNICEF

Consultant Beginning June 2015

8

Reporting: Preparing inception report (methodology and work plan) Consultant Mid May 2015Submission of the draft report Consultant Mid/End June

2015 Feed-back on the draft report from UNICEF and Implementing Partners

UNICEF End June 2015

Submission of the final report Consultant Beginning of July 2015

Use of evaluation findings:Dissemination of the final report to all partners and stakeholders

UNICEF Beginning of July 2015

Agreement reached with Implementing partners on how to translate key findings into activities and integrate them into further activities / programming

UNICEF Mid of July 2015

The consultant is expected to produce the following key deliverables with the following tentative deadlines:

1. Inception Report, by 15 May 2015.2. Draft project evaluation report, by 20 June 2015.3. Final project evaluation report including executive summary and annexes, by 10 July 2015.

All deliverables should be submitted in English. UNICEF will ensure translation in Montenegrin.

Timeframe for this work assignment is from the 26th December 2014 to 10th July 2015. During that period total number of consultancy days available is up to 17 maximum, with estimated share of days as following:

- Desk review and submission of Inception report – 3 days,- Field visits and debriefing – 7 days,- Draft report development – 6 days,- Final report submission and presentation at the Final conference – 3 days

The consultant is expected to observe the UNEG ethical guidance to evaluation as guiding principle to ensure quality of evaluation process. (http://www.uneval.org/search/index.jsp?q=ETHICAL+GUIDELINES)

Management and Organization

Management: The evaluation will be managed by the UNICEF Country Office – Evaluation steering committee consisted of the Programme Specialist, Social Policy and Equity Officer (M&E focal point) and Education specialist. The consultant will be supervised by Programme Specialist. The management of the evaluation will include development of the terms of reference, assignment of the evaluation team, liaison between the evaluation team and partners / stakeholders involved (supporting organisation of meetings / interviews and field visits), as well as quality assurance of the report.

Representatives of Implementing Partners will be involved in designing the evaluation and will participate in the definition of recommendations through active contribution during debriefing meetings and providing feed-back to the draft report.

Organization: Individual consultancy is required for this task.

9

Schedule: This assignment will commence on 26th December 2014.

Key intermediate tasks of the consultant: Desk review of relevant documents and reports; Develop more detailed evaluation methodology and work plan – draft to be submitted as a part of the

Inception Report to UNICEF for approval, including key instruments / interview questions; To conduct data collection through field visits (realize interviews/focus groups with selected

stakeholders, partners and beneficiaries); Present initial findings through a de-briefing meeting with UNICEF and implementing partners; Prepare the draft report with key findings, recommendations and lessons learned based on all sources

of information used Based on feed-back provided by UNICEF and implementing partners prepare and submit the Final

Report with all key findings, recommendations and lessons learned following the UNICEF Evaluation Report Standards

Qualifications/specialized knowledge/experience required to complete the task

The consultant will be selected based on the following criteria:

- Experience of conducting project and programme evaluations;- Technical expertise in education and especially inclusive education;- Knowledge on the education system in in Montenegro;- Knowledge of the UN Convention on the Rights of the Child and relevant international

standards in the area of education;- Previous experience in working with UNICEF;- Communication skills.

The qualifications and skill areas required include:

Consultant with expertise in the area of education with a focus on inclusive education and children with special educational needs;

Documented extensive evaluation expertise and experience; Proven knowledge on the education system in Montenegro, knowledge on the region is an asset; Excellent report writing skills; Good communication and presentation skills; Excellent written and spoken English; Ability to keep with strict deadlines; Knowledgeable of UN Evaluation policy; Experience in working with UN / UNICEF (obligatory).

Consultant should be sensitive to beliefs and act with integrity and respect to all stakeholders. In the report the consultant may not refer to individual children. Consultant may not share findings with media in Montenegro or abroad concerning individual children or individual institutions.

UNICEF premises will be available during the time spend in Montenegro if needed. Printers, photocopying services, and other similar services will be provided by UNICEF. It is expected that consultant will bring their own laptop.

Budget and Remuneration

The request for services under this contract will require the prospective consultant to indicate their fees for the services to be provided. As part of the selection process, the office will select the consultant that

10

quoted the lowest fee from the list of prospective consultants who are deemed suitable for achieving all tasks in time, and as per the criteria and deliverables stipulated in the Terms of Reference.

9. Structure of the Evaluation Report

The final evaluation report should follow UNEG Norms and Standards, UNICEF Evaluation Report Standards and should follow the GEROS Quality Assessment System.

The report template to be used includes:

Title page and Opening pages Executive summary Project description (including the logic of the project design and/or expected results chain) The role UNICEF and Implementing partners and other stakeholders involved Purpose of the Evaluation Evaluation criteria Evaluation Scope and Objectives The evaluation design Description of Methodology The stakeholders participation Ethical issues Findings Analysis of results Constraints Conclusions Recommendations Lessons learned Annexes

The evaluation report will be assessed and rated against the following assessment tool available here:

11