HIGHER EDUCATION SOLUTIONS NETWORK – MIDYEAR REPORT (FY2017) COMPREHENSIVE INITIATIVE ON TECHNOLOGY EVALUATION

Agreement No. AID-OAA-A-12-00095

Submitted: THIS PUBLICATION WAS PRODUCED FOR REVIEW BY THE UNITED STATES AGENCY FOR INTERNATIONAL DEVELOPMENT. IT WAS PREPARED BY THE COMPREHENSIVE INITIATIVE ON TECHNOLOGY EVALUATION AT THE MASSACHUSSETS INSTITUTE OF TECHNOLOGY.

PHOTO COURTESY OF D-LAB MIT

CONTENTS

1. EXECUTIVE SUMMARY 3 2. MAJOR MILESTONES / ACHIEVEMENTS 3 3. KEY ACTIVITIES 4 4.A FUTURE (CORE) ACTIVITIES (YEAR 5) 14 5. ENGAGEMENT WITH PARTNERS 15 6.USAID STAFF AND OPERATING UNITS 17 7. MONITORING & EVALUATION 19 8. LESSONS LEARNED/BEST PRACTICE 20 9. ENVIRONMENTAL MONITORING 21 10. OTHER 21 APPENDIX 1. TANZANIA TRIP REPORT

2. CO-CREATING A CULTURE OF TECHNOLOGY EVALUATION IN A GRASSROOTS,

WOMEN-LED ORGANIZATION IN INDIA

3. EXECUTIVE SUMMARY The Department of Urban Studies and Planning (DUSP), D-Lab, the Sociotechnical Systems Research Center (SSRC), the MIT Center for Transportation and Logistics (CTL) and the Priscilla King Gray Public Service Center (PKGPSC) are the main partners in the Comprehensive Initiative on Technology Evaluation (CITE), a network of MIT faculty, staff, and students with expertise in technology design and testing, systems engineering, supply chains, community ethnography, institutional and public policy analysis, market systems analysis, and regional economics. CITE’s goal is to develop a rigorous product evaluation method that will help development organizations make educated product deployment decisions in emerging market economies. Our evaluations will also allow the development community to analyze specific product applications leading to better, data-driven programmatic decisions and overall product designs. By applying our evaluation methods, CITE will help to identify the bottlenecks that prevent products from achieving measureable impact.

At the mid-point of the fifth year of its program CITE is able to report milestones and achievements related to: (i) impact within MIT; (ii) contribution to USAID through the Global Development Lab (the Lab) and Higher Education Solutions Network (HESN); and (iii) global partnership.

2. MAJOR MILESTONES / ACHIEVEMENTS

During this reporting period (October 1st 2016 – March 31st 2017) the Malaria Rapid Diagnostic Test Kit (RDT) evaluation and Post-Harvest Storage Technology (PHST) evaluation reports were released. Both releases were covered by MIT News. To date there have been over 488 downloads of the Malaria RDT report and 584 downloads of the summary PHST evaluation report. Results from the work provide significant insights and identify development opportunities as well as provide data for decision makers. In addition, a report on the impact of CITE’s work over the first four years of the program was released. This report has been sent to partners and shared with organizations and individuals globally, in part to thank our partners for their contribution to the work, but also as part of CITE’s future planning activities. During the first half of year 5, CITE’s evaluation teams have been able to report significant, early progress in their work as a consequence of the rigorous process for product selection carried out in year 4. Four new product evaluations are underway in collaboration with nine partners. In addition, three year 4 evaluations have continued in collaboration with an additional seventeen partners. International field visits have taken place in India, Tanzania, Mali and Uganda to carry out needs assessments, develop partnerships and refine work plans for the evaluations. Scoping studies have been completed and sensor technology installed for data collection. Global organizations such as Mercy Corps, local NGOs, such as the Self Employed Women’s Association (SEWA) in India, USAID Bureaus and Missions and organizations such as the International Society of Wheelchair Professionals (ISWP) have all been instrumental in shaping the CITE year 5 evaluations and providing support in the field. CITE has continued to build a learning community at MIT, consolidating its relationship with (i) the Priscilla Gray King Public Service Center (PGKPSC) which has committed support for international travel for four CITE RAs during this reporting period, (ii) the Tata Center which is funding previous CITE RA Jaswanth Madhavan to carry out an evaluation of sewing machines in India, and (iii) the Office

of Digital Learning at MIT which is supporting development and launch of the CITE edX course. In addition, the Legatum Center are funding previous CITE RA, Amit Gandhi, to continue work to develop low cost sensors which is of significant benefit to the CITE program. MISTI Peru has also committed funding to support travel for faculty and students from Universidad de Ingeniería y Tecnología to MIT over the summer of 2017 to participate in a training workshop led by CITE. Significant progress has been made during this reporting period in relation to activities regarding CITE’s longevity, both on campus and off. CITE’s association with D-Lab has strengthened and CITE and SEWA Bharat and SEWA in Gujarat continue to remain committed to a longer term partnership. CITE has completed an initial needs assessment to understand whether a technology evaluation could add value to SEWA’s existing offerings and what organizational dimensions would influence the design and implementation of the technology evaluation process. CITE also signed a sub-contract in October with United Cerebral Palsy Wheels for Humanity (UCP Wheels) for work being carried out to develop a methodology for the Google User Voice Project, and received confirmation of a sub-award through Solar Sister as part of a project with The Bureau of Oceans and International Environmental and Scientific Affairs’ (OES) Office of Global Change at the Department of State. CITE course development has also continued as part of our year 5 activities with the focus on development of a new edX course on the evaluation of products designed for the developing world to be aimed at professionals. edX is an organization established by MIT and Harvard University that developed an open-source technology platform to deliver online courses. All course content, including videos, has been produced and reviewed and will be uploaded to the course instance on the edX platform by the first week of May, ready for a June 1st launch. Finally, a highlight of this reporting period was co-hosting the technical convening of the HESN network, TechCon, together with IDIN and USAID. The conference was held between November 10th and 12th and brought together researchers, students, entrepreneurs and innovators, development experts, field practitioners, and private sector representatives focused on highlighting cutting-edge solutions to critical development challenges.

3. KEY ACTIVITIES CITE’s overarching goal is to increase the impact and cost-effectiveness of products designed for people living in poverty by developing and implementing a product evaluation methodology. The annual work plan for the fifth year of the initiative was developed to guide project activities in line with this goal. Hence, CITE selected the following projects for year 5 using a rigorous evaluation protocol from proposals submitted by partners: (i) Evaluating the Impact of Solar Pumps on the Water-Energy-Food Nexus; (ii) Evaluation of RUDI Food Packaging; (iii) Evaluation of Vegetable Cooling and Storage Products and (iv) YieldWise Innovation Decision Support. Beyond these evaluations, during this reporting period (October 1, 2016 to March 31, 2017), CITE also focused its energy on completing year 4 evaluations, releasing the Malaria RDT evaluation and PHST evaluation reports and on-going central CITE activities related to: (i) development of CITE methodologies and courses; (ii) cultivation of hubs; and (iii) CITE’s longevity.

The key activities undertaken during the first half of year 5 are described below in relation to the evaluations and additional focus areas and central activities as outlined above.

Evaluation-Based Work Project 1: Sustainable Agriculture: Evaluating the Impact of Solar Pumps on the Water-Energy-Food Nexus (CITE-led) Jennifer Green, CITE Sustainability lead, is heading a team during year 5 to broaden the evaluation of solar water pumps carried out during year 4. Three additional, key challenges, identified through stakeholder feedback, will be addressed: finding appropriate financing models, identifying secondary uses of excess solar capacity, and long-term sustainability and adoption. Addressing these three issues, plus testing of SMS sensors in SEWA’s solar pumps, is the main focus of this year 5 evaluation. The project team also includes CITE Research Assistant, Jonars Spielberg, Legatum Fellow, Amit Gandhi, CITE Affiliates Sara Pesek and Vandana Pandya and Anurag Bhatnagar from SEWA Hariyali. Éadaoin Ilten and Brennan Lake from the Technology Exchange Lab have also participated actively in the evaluation. Solar powered pump systems have been promoted as an appropriate, beneficial technology. By replacing conventional diesel-powered systems, solar pumps mitigate air pollution and the release of greenhouse gas emissions. Relative to those who use manual systems solar pump systems are less labor and time intensive, freeing up energy and schedules so that other activities can be pursued. CITE’s evaluation focuses on two use cases, (i) irrigation for farmers living in rural areas where grid electricity is unreliable or non-existent, who are able to increase productivity through additional harvest during the dry season, which can lead to improved livelihoods and income generation and (ii) salt farmers in the Little Rann of Kutch where the use of solar energy for pumping is being promoted by SEWA Hariyali to address specific challenges of time (while diesel pumps are better than ox-driven water wheels, they are both inefficient and lead to low productivity), cost (cost of fossil fuels, and of low productivity) and drudgery for women.

In both cases, solar water pump systems are an important technology at the water-energy-food nexus. Yet, despite their advantages, the production, use and promotion of solar pump systems have been fairly paltry and uneven over time. The solar water pump team aims to understand the barriers to scale through a comprehensive evaluation.

The CITE team is working in close partnership with the Self Employed Women’s Association in Ahmedabad (SEWA) and their Hariyali program and the Technology Exchange Lab (TEL) who emerged as key partners in the work during year 4. We continue to keep Mercy Corps (MC) informed of progress as they originally proposed the work. However, the MC use case in Myanmar did not prove to be sufficiently mature to include in the final comprehensive study or year 5 extension. At the beginning of this reporting period CITE determined that a single consolidated report would be most appropriate to disseminate the findings from all the work completed during year 4 and year 5. We aim to release this report in June of 2017 and it will comprise of (i) the Scoping Study (including 5 possible Use Cases), (ii) Solar Pumps for Irrigation (including stakeholder interviews, field testing, sensors, a Pump Sizing Tool, and system dynamics model), and (iii) Solar Pumps for Salt Production (including stakeholder interviews, field testing, sensors, business model comparison, and lab testing of 5 pumps).

Between October 2016 and March 2017, the CITE team completed laboratory testing at MIT of 5 solar water pumps, continued to collect and analyze data from sensors installed in the solar pumps for irrigation and also developed a pump sizing tool – all of which relate to year 4’s scope of work. In addition, during this reporting period, the CITE team has focused on the three additional challenges identified in the year 5 scope of work: Comparison of Business Models An analysis of SEWA’s loan program for salt miners to purchase solar water pump systems has been completed including loan terms and payback period versus annual income, as well as a comparison with the use of diesel pumps. In addition, a comparative analysis between the SEWA model and two alternate ownership models has been carried out (Model 1: Farmers own entire system through financing, Model 2: A developer owns solar array and electronics, and farmers own their own pumps and purchase electricity through metered system). Finally, a preliminary analysis of alternative uses for solar panels in the off season has also been carried out. For farmers who own pumps, income generating opportunities for panels during the off season (monsoon) include: battery charging, collectively charging telecom towers, cottage industries and home lighting. Water-Food-Energy Nexus Modeling A computer simulation model of an integrated water-energy-food system has been developed to help prioritize investments and find the best mix of energy solutions, irrigation techniques, new seed technologies, and water resource management. The system dynamics (SD) model includes 6 interrelated systems, (i) water, food and energy are the core systems, (ii) livestock and population systems as "controls” and (iii) the welfare system as the outputs of interest. The unit of analysis is state policies (rather than household, as in the past). Installation and monitoring of sensors in SEWA Hariyali solar panels To determine appropriate scoping and deployment of the solar pumping systems for use in the SEWA salt farming pilot, we developed specialized remote-sensing prototypes to characterize the output and usage of the systems. The data-loggers were installed in 17 locations in the Little Rann of Kutch in January and designed to measure outputs of the systems until May-June of this year. The data from several pumps was aggregated from installation through March 31, 2017 to understand regularity of system usage and production. The data shows consistent usage of the pumping systems with varying levels of pump usage. We will continue to analyze the data to determine season variability and track longer term adoption rates for the different sensor systems. CITE aim to submit a draft report on this work for internal review at the beginning of May 2017 and to release the final report by the early summer of this year.

Project 2: Evaluation of RUDI Food Packaging (CITE and SEWA RUDI led) CITE and SEWA RUDI are carrying out a joint evaluation with SEWA RUDI being led by CITE Researcher Vandana Pandya. The aim of the evaluation is to improve shelf life of cereals, pulses and spices by identifying and developing affordable and environmental friendly packaging technology for RUDI, which is an initiative of SEWA. Specific objectives are to (i) improve the livelihood of sales women associated with RUDI by decreasing spoilage, (ii) provide food security to Rudi’s saleswomen’s customers and to reduce childhood malnutrition, and (iii) reduce post-harvest loss by choosing

appropriate packaging and storage technology. In addition, the learning that emerges from having SEWA RUDI conduct the evaluation will inform the creation of a CITE practitioner tool kit. The project team also includes Jennifer Green, Uma Swaminathan, from SEWA RUDI and a small contribution from CITE RA Mark Brennan. SEWA Rudi enables direct promotion and marketing of agro-commodities procured through Farmers Associations across all the villages of 14 districts of Gujarat. Specifically, SEWA Rudi sells pre-packaged “kits” — bundles of individually packaged spices and pulses — to a network of saleswomen. These saleswomen are often “bottom of the pyramid” entrepreneurs. They sell the individually packaged spices and pulses in and around their villages. Their customers are rural Indians, who are also often “bottom of the pyramid” consumers. Rudi is looking for affordable packaging technology using food grade materials that is environmental friendly and improves the shelf life of stored products. In the broadest sense, this evaluation will also build capacity for SEWA and Rudi to evaluate other products related to their operations, which will have similar impacts for bottom of the pyramid farmers, distributors, and consumers. This evaluation has been designed as a frugal evaluation that can be carried out by an organization such as SEWA. During this reporting period, Vandana has toured and conducted interviews at all points on the food packaging supply chain, including with (i) farmers, (ii) mills (rice) and SEWA processing centers, (iii) distribution centers, (iv) rudibens (saleswomen) and is currently compiling data in Excel and preparing for data analysis and a simple technical evaluation of the food packaging. We anticipate releasing a report from this work in August of this year.

Project 3: Evaluation of Vegetable Cooling and Storage Products (CITE-Led) Prof. Dan Frey, CITE Faculty Suitability lead, is overseeing a team during year 5 to perform an evaluation of evaporative cooling products in Mali. The CITE team is being led by Eric Verploegen, D-Lab lead for the Off Grid Energy Group, with support from Legatum Fellow Amit Gandhi, researcher Prithvi Sundararaman and three undergraduate students Claire Nobuhara, Julia Heyman and Cali Gallardo.

In areas without reliable energy access and/or suitable refrigeration equipment, the inability to refrigerate perishable foods presents a significant supply chain challenge and limits the populations that can access and benefit from such products. The products that are the focus of this evaluation have the potential to strengthen the perishable good supply chain and create opportunities for additional income generation through business activities selling perishable foods. In addition, increased availability of appropriate refrigeration technologies would allow consumers to benefit from increased access to nutritional foods.

The CITE team is working in close partnership with (i) the World Vegetable Center, (AVRDC) that supports the production and consumption of vegetables for economic and health benefits in Mali, builds ZECC (Zero-Energy Cool Chamber) units for horticulture cooperatives (funded by USAID) and is interested in understanding the performance of the different product designs that they use, (ii) the Institute of Rural Economy of Mali (IER) that is interested in Agronomical Research and Food Science and is also partnering with AVRDC on ZECC research, (iii) Mercy Corps Mali who have identified refrigeration as a key need in northern Mali during an assessment with D-Lab in 2015 and who has field offices and works with horticulture cooperatives in Northern Mali and (iv) product manufacturers based in Mopti. Between Oct 2016 and March 2017, CITE has worked to consolidate these partnerships and also establish a team of researchers in the field which comprise of Takemore Chagomoka Ph.D. (Production & Postharvest Specialist, AVRDC), Ousmane Sanogo (Project Coordinator), Aliou Coulibaly (Masters

student at University of Bamako), two regional site coordinators and six technicians. Our implementing partners are particularly important in this instance given the significant restrictions to travel in Mali for CITE researchers. The team has communicated closely with relevant stakeholders and partners in parallel with carrying out a literature review to refine the goals and desired outcomes of the evaluation as well as the scope of work. The evaluation will consider 6 different products, three pot-in-pot or pot-in-dish, and three ZECC products (brick, straw and sack) in three different regions in Mali representing significantly different climatic conditions. In addition, during this reporting period the sensors were prepared at MIT to monitor parameters such as ambient temperature, interior refrigerator temperature and usage behavior. The sensor technology is integral to the field based testing of the products for the suitability portion of this evaluation. Lab based testing of the sensors was also carried out before Eric Verploegen travelled to the field in February to work with our implementing partners on a sensor orientation and training as well as to install some of the sensors for testing in the field. Initial data collection was carried out to inform data analysis for the subsequent full study and Eric also completed initial interviews and focus groups with users of the technology. To date, all pots have been made in Mopti, sensors installed and products distributed to users who are participating in the study. The first round of data collection should be complete in May with the second and final round at the end of August. The time period for data collection was selected based on the seasonal weather patterns in Mali, August is the most humid month of the year, and thus it is important to collect data through this time in order for comparison with the less humid seasons. We anticipate releasing a final report by the end of September. The research plan has been approved by MIT’s Internal Review Board (IRB), the Committee on the Use of Humans as Experimental Subjects (COUHES).

Project 4: CITE-GKI YieldWise Innovation Decision Support (CITE and GKI led) Jarrod Goentzel, CITE Scalability Research Lead, is overseeing a study during year 5 on the processes by which “decision makers” identify, evaluate, and select products or practices for use in their post-harvest sector programming. We are conducting this study with The Global Knowledge Initiative (GKI), an implementing partner of The Rockefeller Foundation’s YieldWise program which is a $130 million initiative aimed at demonstrating significant reduction of post-harvest food loss in sub-Saharan Africa. By 2021, The Rockefeller Foundation aims to show 50% reduction in Sub-Saharan Africa, beginning with a focus on four value chains in three countries: maize in Tanzania led by AGRA, cassava and tomatoes in Nigeria led by Technoserve, and mangoes in Kenya led by PYXERA. The Global Knowledge Initiative (GKI) will serve as the Innovation Partner, which aims to improve the efficiency, effectiveness, and ultimately the impact of YieldWise through innovation cultivated internally as well as sourced externally.

The Tanzanian post-harvest sector has received substantial investment in the past few years. As well as the Rockefeller Foundation’s YieldWise initiative, the World Food Programme’s Purchase for Progress initiative, an effort to buy food from smallholder farmers, addresses questions of post-harvest loss and market access there. USAID- and other donor-funded projects like NAFAKA II are working with millers to improve post-harvest handling. And the Gates Foundation-funded Purdue Improve Crop Storage (PICS) network is active there. The Tanzanian post-harvest sector is, therefore, active, with a variety of

donors, international and non-governmental, and implementing organizations making up the stakeholder landscape.

This CITE project addresses a specific desire within this active region, by AGRA, to engage in an “optimized innovation decision-making process”, while also synthesizing CITE’s expertise on agriculture, decision-making processes, and providing evidence for development programs. The project team includes CITE RA Mark Brennan and MIT MBA student Kate Collins.

During this reporting period, the CITE team has been working in consultation with stakeholders from the Tanzanian post-harvest sector to refine the goals and scope of this project including a determination of the specific research questions to be addressed, the research methods to be used and identification of the interview subjects. In addition, the CITE team has been working in consultation with stakeholders from the Tanzanian post-harvest sector to ensure that (i) the study provides relevant feedback on initial results from the implementation of a Decision Making Toolkit, a report that provides tools (e.g., methods to rank or generate ideas) for YieldWise stakeholders to use to improve their decision making processes and (ii) the results are of use in on-going and future programming within Tanzania.

During the project scoping phase, which was carried out between October and December of 2016, a revision of the original scope of work was made in response to the evolving nature of GKI’s activities in relation to the Yieldwise project. In our original work plan, we aimed, with GKI to (a) conduct innovation scans and (b) co-develop the Toolkit. An innovation scan is a scan of, for instance, existing products and practices to solve a tangible, short-term need of an organization. However, during the first half of this reporting period, innovation scanning became a less substantial portion of GKI’s overall engagement with YieldWise; they only produced one scan, and it was independently led by a doctoral student visiting with them. Rather, during this time GKI focused on the Toolkit. While we had initially planned to co-develop the Toolkit, GKI and CITE identified a need for additional research that could be used to strengthen the Toolkit. Thus by November we opted to focus on designing a research protocol and data collection, in order to strengthen the toolkit, instead of to co-developing a Toolkit with already-apparent weaknesses.

The primary research questions that emerged were:

1. How do individuals and organizations identify, evaluate, and select different processes or products for their post-harvest programming?

2. What are attributes (e.g., years at organization) of these individuals and their organizations that affect decision-making?

3. What criteria are used in evaluating and selecting among options for products and practices?

The primary data collection method to be employed was a semi-structured interview, and the sampling method was purposeful, snowball sampling. In other words, respondents were selected on the basis that they would have relevant, revealing information, and identified by asking each respondent whom else the research team should interview.

In January, MIT CITE Research Assistant and PhD student Mark Brennan and MIT MBA student Kate Collins traveled to Tanzania to conduct interviews. They met with eighteen stakeholders from the Tanzanian agricultural sector over three weeks, and when possible stakeholders specifically from the Tanzanian post-harvest loss sector. Mark conducted three follow-up phone interviews after he returned to the United States. A detailed travel report and summary of initial findings is attached as an appendix to this report.

We will use qualitative case-based analysis to study the data gathered in the interviews. Codes will be used to annotate and draw connections among interview transcripts, allowing us to systematically identify differences across decision maker experiences and understand relationships among factors that affect decision-making. We will conclude this step of the study by late spring.

We aim to release a final report in early summer. The primary audience for the report will be the upper- and mid-level decision makers in the Tanzanian post-harvest sector. The secondary audience for the report will be state donor organizations and academics.

Completion of Year 4 Evaluations Evaluation of wheelchairs: the CITE evaluation of wheelchairs has continued during this reporting period. In particular, the field based data collection has been aligned with the work being carried out with UCP Wheels on the Google User’s Voice project. This has been done in order for both projects to benefit from the complementary components of the work: submission of documents for international IRB approval, distribution of wheelchairs and final development of the survey instruments.

Whilst leveraging the work with UCP will richen both the individual UCP pilot and CITE evaluation, it has also presented significant challenges which have resulted in further delays. Firstly, the location of the evaluation was changed to Bali due to difficulties distributing chairs in sufficient numbers for the study in our original choice of location in Indonesia. Secondly, an alternative donor for the chairs had to be found, which has resulted in a significant delay in distribution which is now scheduled to take place in June. Thirdly, a new academic partner was sought to administer the surveys in Bali. The Universitas Gadjah Mada is now playing this role (UGM). UGM has also handled submission of the international portion of the required IRB approval which has taken significantly longer than anticipated, but is due to be in place by the beginning of May.

Despite these delays, progress has been made during this reporting period. Prithvi Sundararaman traveled to Indonesia on behalf of CITE between October and November of 2016, to complete field based testing for the technical portion of the evaluation. This involved overseeing installation and testing of data loggers on wheelchairs in the field. The survey instrument is also now complete and Matt McCambridge travelled to Bali in March of this reporting period, to meet with partners, train UGM staff in preparation for the study and meet with Kopernik who will partner on a field based track testing portion of the work.

Laboratory based testing is underway at the University of Pittsburgh and preparations are almost complete for the track based testing at MIT. We now aim to release a technical report by the end of the summer with an addendum to follow in the autumn detailing the findings of the field based testing.

Evaluation of food aid packaging: The food commodities procured by USAID’s Office of Food for Peace for CITE’s evaluation of food aid packaging, all arrived at the foreign ports at the beginning of this reporting period. The 3 month storage experiment concluded in January of 2017 and quality analysis of this experimental data was carried out between January and March of 2017, with cost analysis being finalized in March 2017. The final integrated report is currently being compiled and will be sent to USAID and partners for review at the end of May 2017.

During this reporting period, CITE RA, Mark Brennan and CITE’s Communication’s Strategist, Lauren McKown, have also continued to work hard to raise the profile of this aspect of CITE’s work. They were successful in having articles featured in Devex, MIC and MIT News.

Release of CITE Evaluation Reports CITE released its comprehensive report and a 2 page “at a glance” document on the evaluation of Malaria Rapid Diagnostic test kits in October 2016 as reported by MIT News. The comprehensive report has been downloaded 488 times from the CITE website and the following publication:

Corinne Carland, Gilberto Montibeller, and Jarrod Goentzel. “Modelling values of humanitarian organizations in supply chains”, is under first revision for European Journal of Operations Research.

In addition, in February 2017, CITE released a 90 page comprehensive report, a 17 page summary report and a 2 page “at a glance” document on the evaluation of post-harvest storage technology. This was also featured in MIT News and the summary report has been downloaded 584 times from the CITE website. There are also plans to hold a combined Agrilinks webinar with WFP if time allows during the second half of this year.

CITE Central Activities CITE Methodologies and Course Development CITE’s Methodology Working Group has continued to meet periodically during this reporting period. The primary activity of the group during this time has been focused around launch of CITE’s edX course for international students and practitioners, now scheduled for the beginning of June. This activity is described in more detail later in this report. CITE’s group effort will be diverted towards development and release of the CITE Practitioner Guidebook during the second half of year 5.

In addition, the individual 3S teams continue to develop methodologies in line with their areas of expertise.

The Suitability team has worked to incorporate instrumentation into the wheelchair, solar water pump and vegetable cooling and storage product evaluations. Questions of rigor versus relevance of the data are being addressed in relation to what extent adequate data can be captured for evaluation using low-cost technology that can be implemented locally without undue burden on staff. The longer term aim is to provide more objective, quantifiable data in a distributed form to cover a wider geography over a longer period of time than current field methods allow.

The Scalability team has continued to work with Gilberto Montibeller, an expert from the London School of Economics, to apply an enhanced Multi Criteria Decision Analysis in order to understand the adoption of technologies among consumers in a developing country context and design business models for the private sector. This method was implemented as part of the malaria RDT evaluation and the analysis has been further refined during this reporting period and articulated as part of an academic article which is under first revision for European Journal of Operations Research.

In addition, the food aid packaging team are adding a level of rigor to the study of supply chain design by introducing design of experiments methodologies. This approach is a systematic, cost-effective way to evaluate multiple supply chain designs across various criteria, such as insect infestation, mold growth, and appropriate bag handling, and present insights to decision makers. In evaluating the costs of current

and proposed food aid packaging, the analysis will use a top-down cost analysis, based on data from procurement contracts, and a bottom-up cost analysis, based on domestic and international field interviews, to triangulate the costs of different packaging options.

The work of the scalability team will extend more broadly the application of commercial supply chain methods to less opaque markets in developing countries and will result in a toolkit for future CITE evaluations (and other organizations) to use.

The Sustainability team has worked to incorporate aspects of the Lean Research approach into the solar water pump evaluation. The Sustainability team have also piloted the CITE generic methodology as part of this evaluation. This has incorporated a more rigorous approach to carrying out a scoping study. Learnings from the field will be reviewed internally, and incorporated into the CITE Practitioner Toolkit.

Over the past six months, CITE has continued to collaborate with IDIN, Root Capital and Tufts University on Lean Research. The Lean Research approach is being implemented and documented as cases from Girl Effect, CITE, MIT D-Lab and a consulting group. CITE has also been working on a draft of the Lean Research Field Guide with IDIN, Root Capital, Sustainable Food Lab and Committee on Sustainability Assessment (COSA). The draft has been sent out for internal review. It should be available online this summer.

Course Development Significant progress on the development of the edX course entitled “Evaluating products designed for the poor” has been made during this reporting period. The Methodology Working Group, led by Dr. Joanne Mathias, was convened to mobilize effort around this activity and it has continued to meet regularly during this time.

CITE TAs Faizan Siddiqi and Vishnu Prasad, and DUSP MCP students Carey Dunfey and Haleemah Qureshi are playing a key role in supporting this work. In addition, staff members David Chotin, Lana Scott, Shira Fruchtman and Geoff Wilson, from the Office of Digital Learning (ODL) are also working closely with us to ensure that we meet our revised launch deadline of June 1st.

Between October 2016 and March 2017, the edX course outline was finalized and all lessons were recorded, edited and reviewed. The course instance on edX was structured and is ready to be populated. Assessment questions with solutions and rubric, as well as discussion forum prompts were finalized and 50% of the required IP review process has been completed.

In addition, during this reporting period plans have been developed to guide (i) implementation of the beta testing, (ii) the discussion forum, (iii) communications with and between course participants and (iv) moderation of the course while running.

All course content will be moved to the CITE course instance on the edX platform before May 5th in preparation for beta testing which will begin during the second week of May.

Finally, during this reporting period CITE has contributed to the following, three additional courses: (i) Prof. Bish Sanyal led the course “Development, Planning and Implementation” which draws

on some of the experience of the CITE program, (ii) CITE work was referenced by Jarrod Goentzel as part of the new course “Innovation for

Humanitarian Assistance and Disaster Relief” and (iii) D-Lab students did a very basic case study of solar panel capacity for SEWA as part of D-

Lab’s Energy class. The students looked at opportunities to rent out solar panels during off-

season usage and have been focusing on looking at solar-powered water treatment systems for the salt farmers that CITE is working with as part of the Solar Water Pump evaluation.

Cultivation of Hubs Pilot SEWA-CITE Project Prof. Bish Sanyal, CITE Director, is overseeing a pilot project with SEWA Bharat, to co-design a technology evaluation methodology that would be embedded within, and adapted to SEWA Bharat’s organizational context. The expectation is that an organically grown evaluation methodology will produce more effective and sustainable results, as it emerges shaped by – but at the same time changing – SEWA Bharat’s organizational culture. The CITE team also includes CITE RA Cauam Cardoso.

CITE is working in close partnership with SEWA Bharat. However, at the beginning of this reporting period, Dr. Sanjay Kumar, then Director of SEWA Bharat, notified CITE of his intention to transition from the organization. Dr. Kumar was instrumental in shaping the vision for an evaluation unit within SEWA Bharat and so, upon his departure, considerable time was spent in dialogue with Renana Jhabvala, SEWA’s National Coordinator and Sanchita Mitra, SEWA Bharat’s new Director, to ensure that both CITE’s and SEWA’s goals and objectives remained aligned. We agreed that concluding the pilot SEWA-CITE project remained an important first step towards achieving our longer term objectives from both perspectives. We have therefore, continued to progress this work, communicating closely with Renana, Sanchita, Ian Mulholland, an intern working in India who has been assigned as a direct point of contact for CITE regarding this project, and Kritika Narula, Research and Project Assistant. The primary focus of the pilot project remains to address the following three questions:

(i) What are the technology - related projects SEWA in Delhi is currently developing? How are they implemented?

(ii) What are the lessons that can be learned from SEWA in Delhi’s experience in implementing technology - related programs?

(iii) How can technology evaluations best support SEWA’s mission in Delhi going forward? The answer to questions (i) and (ii) will provide a basis for an organizational roadmap that will inform the design of future technology evaluation processes. Furthermore, in responding to question (iii), we intend to:

(i) Identify the broad technological areas that could be targeted for future evaluations, and,

(ii) Recommend strategies for technology evaluation within SEWA in Delhi, given the contextual characteristics of the areas in which the organization operates, and considering its current implementation capabilities.

During this reporting period, findings from the qualitative primary data collection carried out in year 4, have been triangulated by reviewing project documents and analyzing quantitative data from databases containing demographic and behavioral information from SEWA members. A summary report has been produced and shared with SEWA Bharat with a view to determining next steps together. The aim is to achieve the largest impact during the summer of 2017 and beyond. This document is included as an appendix to this report.

In November of 2016, Cauam Cardoso also secured co-funding for international travel for this work from the Priscilla King Gray Public Service Center at MIT. He intends to use this funding over the summer of 2017 to complete data collection.

Other Activities TechCon 2016 TechCon 2016 was co-hosted by CITE, IDIN and USAID's Higher Education Solutions Network on MIT's campus. The conference was a unique convening of researchers, students, entrepreneurs and innovators, development experts, field practitioners, and private sector representatives focused on highlighting cutting-edge solutions to critical development challenges. The conference agenda featured a student innovation marketplace, keynotes, panels, breakout sessions and workshops, and more. CITE team members Lauren McKown and Joanne Mathias led MIT’s overall organization and implementation of the conference. CITE team members including Bish Sanyal, Joanne Mathias, Dan Frey, Jennifer Green, Jarrod Goentzel, Jonars Spielberg, and Amit Gandhi all led sessions at the event.

CITE Impact Report Dr. Joanne Mathias, Lauren McKown and Kendra Leith, CITE’s M&E Manager, worked closely with the CITE team to produce a report detailing impact of CITE’s work over the first four years of the program. The report was released to coincide with travel to India to undertake a number of initiatives related to CITE’s future work, including seeking opportunities for collaboration and support for future programmatic efforts. The report has been shared with The Global Development Lab at USAID.

4.A FUTURE (CORE) ACTIVITIES (YEAR 5) During the final half of CITE’s fifth year evaluations of food aid packaging, solar water pumps, RUDI food packaging, and vegetable cooling and storage products will be completed and reports released. In addition, data gathered during the interviews carried out as part of the work with GKI will be analyzed to identify differences across decision maker experiences and understand relationships among factors that affect decision-making. A final report will be released in early summer. The evaluation of technology needs for SEWA will also be completed during the summer of 2017 and a final comprehensive report released. CITE’s Communication Strategist and Associate Director have worked closely with CITE faculty and staff to determine a schedule for release dates which has been shared with USAID.

In addition, CITE will disseminate its framework for evaluation during the second half of this year in three ways: (i) the edX course will be launched on June 01 and run until July 14th; (ii) CITE’s practitioner guide will be developed for release in the fall of 2017; and (iii) CITE will host faculty and staff from the Universidad de Ingenieria y Technologia as part of a training workshop with funding from MISTI Peru.

4.B Future (Core) Activities (No-Cost extension Requested for Core Activities) Activities that will extend into the first three to four months of year 6 include:

(i) Completion of the CITE wheelchair evaluation and release of the associated report, (ii) Completion and release of the CITE practitioner guide (iii) Completion of an academic reflection on CITE’s work (iv) Completion and submission of final performance reports for USAID (v) Activities in relation to reconciling financial matters and archiving material as appropriate for

DUSP and MIT

(vi) Activities in relation to making CITE data bases publically available (vii) Activities in relation to institutionalizing CITE at MIT

4.C Closeout (core) Activities and Sustainability Plan CITE will be taking advantage of the no-cost extension (NCE) issued to the overall CoAg and therefore, there will be no formal close-out process at the end of Year 5. However, CITE is working with the Department of Urban Studies and Planning (DUSP), Sponsored Accounting (SA) at MIT and the Office of Sponsored Programs (OSP), as appropriate, to ensure that program records and data are archived and processes and responsibilities have been identified and allocated for administration of the program during year 6 and for when formal close-out will take place

CITE’s key responsibilities will include equipment disposition, submission of the final performance report, archiving of all data bases and working with the DUSP AO to archive all relevant financial material for audit (including ensuring that all cost-share is documented). Any reports or findings generated will be shared with USAID.

CITE’s Associate Director, Joanne Mathias and CITE Finance and Program Assistant, Jesse Kaminsky have meet with DUSP AO, Karen Yegian to have initial conversations. Follow up meetings with Karen, and appropriate officers from SA and OSP are scheduled to take place early in the summer of 2017 to ensure individual responsibilities and roles have been accurately and clearly identified.

CITE aspires to continue as a program with a network of faculty, staff and students at MIT, beyond the current funding with USAID. We are referring to this next phase as CITE 2.0. During this final year we have continued strategic planning for CITE 2.0 with relevant Departments, Labs and Centers (DLCs) at MIT, most notably D-Lab. We anticipate contributing to D-Lab’s revised business plan in the autumn of 2017. In addition, we are planning to host an invitation only event for MIT faculty early in the fall of 2017, in order to widen CITE’s faculty engagement on campus and strengthen the academic rigor of the program.

In addition, CITE will continue to liaise with MIT administration and DLCs to leverage resources for the program, both continuing in its final year of core funding with USAID and beyond. To date we have been successful in obtaining travel funding from PKGPSC, MISTI Peru and the Legatum Center and our students have received fellowships from the Tata Center and the Legatum Center. CITE will consolidate current partnerships, including with institutions in India and Uganda, and reach out to new partners in order to maximize opportunities for potential future funding together.

The main risk identified is the length of time that such negotiations take and the limited funds available to support this aspect of the work.

5. ENGAGEMENT WITH PARTNERS

Other Departments on Campus CITE has continued to build the multidisciplinary ecosystem of MIT faculty, staff, and students focused on issues around technology in development. The Department of Urban Studies and Planning (DUSP), D-Lab, the Sociotechnical Systems Research Center (SSRC), the MIT Center for Transportation and Logistics (CTL) and the Priscilla King Gray Public Service Center (PKGPSC) continue to be the main partners in CITE. CITE and PKGPS continue to strengthen the relationship between the programs which has again resulted in four CITE RAs being awarded travel scholarships by the PKGPSC for the work they have/will

be carrying out in the field as part of CITE evaluations during year 5. CITE has also continued to partner with the Tata Center for Technology through the award of a Tata Fellowship to previous CITE RA Jaswanth Madhavan. Jaswanth has been carrying out a project of mutual benefit to both CITE and the Tata Center and travelled to the field in January to work with SEWA Bharat on this project. In addition, the Legatum Center has awarded a fellowship to previous CITE RA Amit Gandhi and a seed grant for travel to Kate Collins. Both awards have contributed significantly to the completion of CITE activities this year. CITE continues to try to leverage support from other centers, labs and departments at MIT and submitted an application for seed funding to the Abdul Latif Jameel World Water and Food Security Lab (J-WAFS) at MIT in January of this reporting period. In addition, two MIT undergraduates received a small stipend from MIT’s UROP office to support laboratory based work on the wheelchair evaluation and MISTI Peru have committed funding to support travel for faculty and students from Universidad de Ingeniería y Tecnología to MIT over the summer of 2017 to participate in a training workshop led by CITE. Finally, CITE has worked very closely with staff from ODL during the first half of year 5 in order to complete the edX course. The partnership is integral to the success of this endeavor. Other Donor Partners CITE has engaged with a growing number of partners during this reporting period, including NGOs, private sector partners, universities and global organizations. This engagement has been described in detail elsewhere in this document. Of note here, however, is (i) the work being carried out with UCP wheels as part of a sub-contract to develop a methodology for the Google User Voice Project, as well as (ii) a sub-contract awarded to CITE through Solar Sister as part of a project with The Bureau of Oceans and International Environmental and Scientific Affairs’ (OES) Office of Global Change at the Department of State. Solar Sister are aiming to grow its women-centered last mile distribution chain for portable solar and clean cooking solutions and improve climate change mitigation in rural communities. CITE will address the specific question of “What effect does an intentionally women-centric distribution model have on access to clean energy and livelihoods?”

Other HESN Labs CITE and the International Development Innovation Network (IDIN) continued to work closely together during October and November of 2016 on activities in regard to co-hosting TechCon with USAID. Each of the HESN Labs participated fully in the conference, leading sessions and contributing to the discussions that took place over the three days of the event. In addition, HESN staff and faculty took part in an HESN community meeting at a pre-conference event. At a critical juncture in the evolution of the Higher Education Solutions Network, this forum provided an opportunity to discuss the progress the Labs have made together over the past four years, and look to future collaborations around programming and partnerships. In addition, CITE has continued to work with IDIN on the Lean Research Initiative during this reporting period and CITE team members participate in monthly HESN M&E Specialist calls.

Students on Campus CITE students have participated regularly in the CITE seminar during this reporting period. In addition, CITE RAs continue to represent the program at events on campus and off. These have included participation at TechCon and at the IEEE Global Humanitarian Technology Conference in Seattle during October of 2016 where Jonars Spielberg presented work on the solar water pump evaluation. This work was also published as a paper as part of the conference proceedings (http://ieeexplore.ieee.org/document/7857257/). CITE students have continued to work closely with CITE’s communication strategist to highlight the work that we are doing, producing blogs and articles that have featured in Devex, MIC and MIT News. CITE students have made significant contribution through their work with our partners. All of our students have built trust, relationships, developed deep understanding of needs and been advocates for change within the organizations that we have partnered with.

CITE students have continued to have the opportunity to experience firsthand field work. Mark Brennan and Amit Gandhi travelled internationally in January as part of the CITE-GKI YieldWise Innovation Decision Support project and solar water pump projects. Jonars Spielberg and Cauam Cardoso will travel internationally to India and Tanzania over the coming summer with co-funding from the PKGPSC. All the students are expected to benefit significantly from immersion in the field. During the first half of this reporting period, six additional undergraduate students have also benefitted from a research opportunity with CITE carrying out laboratory and field based work. The students received small stipends as UROPS either directly through CITE funding or through the MIT UROP office.

6.USAID STAFF AND OPERATING UNITS

6.1 USAID Buy-Ins USAID/Uganda

The USAID/Uganda Feed the Future Market System Monitoring Activity (MSM), led by Jarrod Goentzel in the MIT Center for Transportation and Logistics in collaboration with The George Washington University, completed the first year of its four-year grant to develop new approaches that assess the impact of market facilitation activities in the ongoing USAID/Uganda Feed the Future Value Chain project (FTF-VC). The MSM team conducted two studies over the summer of 2016, analyzed the results during the fall of 2016, and published two-page summaries and detailed reports for both studies. The first study, focused on the “inputs subsystem,” aimed to understand whether and to what extent expected changes were occurring in the last four years of FTF-VC work by asking “How has the inputs subsystem been changing over time?” The second study, focused on the "output subsystem," discussed how quality differentiated prices interact with actors in the market. Over the fall, MSM continued to develop the conceptual map of the market system, releasing v1.0 in October. Before March of 2017, MSM restructured the map to enable workshop facilitation, releasing v1.1. In March 2017, USAID gathered partners and stakeholders for a three-day co-creation workshop where 168 participants and presenters explored a market systems approach, identified components within Uganda’s agricultural market systems, and developed a pipeline of actionable opportunities and challenges to inform future

programming. MSM contributed through vendor selection, design, and facilitation of the workshop. USAID/Uganda will continue to cultivate these co-creation inputs as part of their forthcoming project design work in agriculture market systems. USAID/Bureau for Democracy, Conflict and Humanitarian Assistance (DCHA)/Center of Excellence on Democracy, Human Rights and Governance

A $50,000 buy-in from USAID/DCHA/Center of Excellence on Democracy, Human Rights and Governance is partially supporting work on the evaluation of wheelchairs. The year 5 activities in relation to this reporting period were described in “Completion of Year 4 Evaluations” in this document.

USAID/DCHA/Food for Peace

A $186,889 buy-in from USAID/DCHA/Food for Peace is partially supporting work on the evaluation of food aid packaging. The year 5 activities are described under “Completion of Year 4 Evaluations” in this document. Gregory Olsen, from the Food for Peace Office, is in close, regular communication with the CITE team.

6.2 USAID/LAB Interactions USAID/LAB A particularly important interaction with the Lab during this reporting period has been preparations for and hosting of Techcon 2016. Dr. Mathias and Lauren McKown have worked closely with Michelle L'Archeveque Jones and Maggie Linak on activities related to the conference, including regular conference calls to discuss progress made and hosting of the event itself during November of this reporting period. CITE communication’s strategist, Lauren McKown, has continued to work with Danielle Somers to promote the release of CITE’s reports and events. 6.3 USAID (Non-Lab)/Washington Interactions USAID/E3/Education Jennifer Groff attended the Mobiles for Education Alliance Symposium on behalf of CITE which took place from Oct 18-20 in Washington. 6.4 USAID Mission Interactions USAID/India During this reporting period CITE has been in direct contact with the USAID/India Mission in preparation for CITE’s visit to India. In addition, CITE met with Anand Rudra, Senior Project Manager, WASH specialist, from the Mission, during TechCon, to provide an update in regard to CITE’s evaluation of solar water pumps in India. The USAID/India Mission provided concurrence for the evaluation of RUDI food packaging in India. USAID/Uganda Jarrod Goentzel continued to engage with the USAID Mission from Uganda in order to finalize and release reports for the Post Harvest Food Storage work.

In addition, Jarrod Goentzel leads a team working with the USAID/Uganda Mission on the Feed the Future Uganda Market System Monitoring Activity. Progress made during this reporting period has been described in detail previously in this report.

USAID/Djibouti we continue a low level of engagement with the USAID/Djibouti Mission as a consequence of the food aid evaluation.

USAID/Ethiopia we continue a low level of engagement with the USAID/Ethiopia Mission as a consequence of the food aid evaluation. USAID/South Africa we continue a low level of engagement with the USAID/South Africa Mission as a consequence of the food aid evaluation. USAID/Tanzania provided concurrence for the work in Tanzania with GKI and they have been kept informed regarding travel and progress of the project. USAID/Kenya were informed regarding travel to Nairobi, Kenya as part of the work with GKI. USAID/Indonesia Members of the CITE wheelchair evaluation team met with Ester Manururg, Hanif Saleh and Sutan Royansyah from the USAID/Indonesia Mission to discuss the evaluation and progress made. USAID/Mali provided concurrence for the evaluation of vegetable cooling and storage products.

7. MONITORING & EVALUATION

7.1 M&E Updates We are on track to meet or exceed most of our targets. We have exceeded our targets for the following areas:

x HESN_0in1 $ total dollar value of outside (non-USAID) resources utilized. This number is currently low, but we will more than meet our cost-share target for the year.

x HESN_2.3in2 Number of stakeholders engaged in problem solving with CITE. We have engaged

many partners in our evaluations and they have provided valuable feedback and insights. We recognize the importance and value of partnerships. Thus, we have engaged more partners than we originally predicted. Our new target is 15, but we have engaged 44 this year.

x HESN_3.2in1 Number of visitors to the CITE knowledge-sharing platform. During this the first half of the year, we had 2636 visitors to the website. This is consistent with the number of visitors over the past year, but is above our target for the year.

x HESN_3.4in1 Number of students participating in short-term practica. We have already engaged two students in short-term practica through the decision support project and the solar water pump evaluation. We will exceed this target for the year. However, our counts for the number of fellows was a bit lower than expected.

x HESN_2.2in1 Number of white papers, articles, assessments, analyses and evaluations on development challenges, innovations, technologies, approaches, and contexts. We have released three publications this year. We will likely exceed our target for the year.

x HESN_3.2zCITE-in1 Number of evaluation downloads. Our original target was 2000 for the year, but we exceeded that number as we have 9,135 downloads to date.

x HESN_3.3in1 # of classes supported by CITE with human, financial, or institutional resources contributed by CITE. We set a target of two, but we are currently contributing content to five classes.

x HESN_0zCITE-in1 # technology evaluations produced by CITE. We had a target of three, but are conducting eight evaluations for this year. We continued to work on some evaluations and completed others. We will likely need to revise our target for next year.

x HESN_3.3zCITE-in1 Number students taking class per year. We have already exceed our target. We will exceed it even further with the EdX course this year.

x HESN_2.2zCITE-in2 # users who provided feedback on CITE evaluations. We have already received feedback from five people this year. Thus, it is likely that we will exceed this target.

x HESN_2.3in1 # MOUs or other agreements signed with public sector, private sector, local community partners, and one HESN Development Lab. We have already signed two agreements for this year. Thus, we have more than doubled our target.

We are also below our targets in the following areas: x HESN_Oin10 # of beneficiaries reached. We have already exceeded this target for the life of the

grant. However, we are not on track to meet our annual target of five people. We have reached more intermediate beneficiaries.

x HESN_1.3zCITE-in1 # of partner organizations using CITE data in decision-making processes for product selection. We are on track for the life of the grant, but as far as we are aware, no additional people have used our results. We plan to follow up with others over the next six months. It is likely that we will reach or exceed this target for the life of the grant.

x HESN_2.2zCITE-in3 # requests for the CITE methodology. We have received some requests from organizations such as UTEC and United Cerebral Palsy. However, our target was based on potential downloads of CITE methodology papers and the EdX course. Given that neither of these things are public yet, we have not met this target. However, by the end of the program, we should exceed the target.

x HESN_0zCITE-in2 # intermediate beneficiaries. So far this year, as far as we are aware, no additional people have used our results. However, two have utilized our methods. This number should increase over the second half of the year, so we should meet this target.

x HESN_3.4in2 # Hubs created with human, financial, or institutional resources contributed by HESN Development Labs. We are supposed to create one new hub this year. It is still to be determined as to whether that will happen.

We have started to conduct in depth interviews with partners who have been using our results or methods. We have shared a copy of our impact report with USAID.

8. LESSONS LEARNED/BEST PRACTICE As CITE has matured over the four and a half years of the program, a significant number of lessons have been learnt. During this reporting period CITE would like to highlight the following broad insights. CITE continues to understand the importance of “true” partnership both strategically and on the ground operationally. Working closely with experts from NGOs, global organizations, university partners, USAID and USAID missions has been critical to the success of CITE evaluations in shaping an effective research question and facilitating access to the relevant stakeholders. The relationship with the grass root organization SEWA has been particularly important and we have learnt that the only way to understand context is through close collaboration with organizations such as this.

CITE has learnt that evaluation of technology should be an important consideration during the design of new products as well as once product are in the marketplace. Developing a framework to do this effectively is an important next step for CITE 2.0 as well as consideration of issues around generalizability. While CITE continues to focus on technology, we understand the importance of considering other related components such as financial models and producing frameworks and models to articulate the results of CITE’s work such that they remain relevant over time. Finally, CITE now more fully appreciates that the audience for its evaluations are consumers with the lowest levels of income and that generating data for decision makers should be the primary goal in order to improve lives. This is not a market driven space, but a social responsibility.

9. ENVIRONMENTAL MONITORING Not applicable

10. OTHER Not applicable

APPENDIX

CITE trip report: decision making in post-harvest sector interventions in Tanzania

Trip duration: January 13 to February 3 Trip Destinations: Nairobi, Kenya; Dar es Salaam, Dodoma, Iringa, & Arusha Tanzania Participants: Mark Brennan (MIT PhD), Kate Collins (MIT MBA / HKS MPP)

1. Background

The Massachusetts Institute of Technology’s Comprehensive Initiative on Technology Evaluation is conducting a study on the processes by which “decision makers” identify, evaluate, and select products or practices for use in their post-harvest sector programming. x We consider decision makers typically as individuals in donor, international or non-

governmental, or implementing organizations who have the ability to influence which products or practices are included in their organization’s post-harvest sector programming. These individuals are often program managers, grant managers, and technical staff.

x Designing interventions to introduce better products or processes is complex. A decision maker has to both identify options (i.e., diverging) and thoughtfully evaluate them to select top candidates (i.e., converging). The design of these interventions can happen within one organization or between multiple organizations.

x In our context, products might include bags or driers, and processes might include how farmers are aggregated or how agricultural extension is delivered.

2. Motivation Why Tanzania

The East African, and specifically the Tanzanian, post-harvest sector has received

substantial investment in the past few years. For example, the Rockefeller Foundation’s YieldWise initiative – a $130 million effort aimed at reducing post-harvest loss in Sub-Saharan Africa by 2021 – selected Tanzania as one of three countries in which to begin their operations. The World Food Programme’s Purchase for Progress initiative, an effort to buy food from smallholder farmers, addresses questions of post-harvest loss and market access in Tanzania; USAID- and other donor-funded projects like NAFAKA II are working with millers to improve post-harvest handling in Tanzania; and the Gates Foundation-funded Purdue Improve Crop Storage (PICS) network is active in Tanzania. Qualitatively, the Tanzanian post-harvest sector is active, with a variety of donors, international and non-governmental, and implementing organizations making up the stakeholder landscape. Why not farmers or supply chain actors

There is a growing body of evidence on the factors involved in a farmer’s decision to adopt technologies or practices in the post-harvest storage sector. For example, Bokusheva et al. (2012) in Food Security find that the desire for household self-sufficiency

as well as socio-economic characteristics such as age, land ownership, and completion of a training course determine adoption of post-harvest storage technologies. Prusky (2011), also in Food Security, cites the cost of the technology as a determinant of adoption; likewise, Addo, Birkinshaw, and Hodges (2002) find cost as the major barrier to adoption of integrated pest management strategies.

Other studies of post-harvest products and practices offer analyses of the impact they have on household food security (e.g., Abass et al. 2014) as well as the barriers to their intended use (e.g., Burke 2014), most of which at least color farmer’s adoption decision. Why decision makers Thus the post-harvest community is beginning to understand how farmers make choices among available products or practices, but the question of how “decision makers” identify, evaluate, and select which products or practices to make available to farmers is less understood. For example, this study helps explain, Given the abundance of post-harvest storage products that exist globally that range from plastic drums to bio-pesticide treated bags, how is it that in Tanzania (like most other developing countries) only a subset of these products are available to farmers through programs? Tanzania, because of the abundance of post-harvest sector programming at a variety of different scales, provides a fertile context in which to answer this question.

This avenue of inquiry is also complimentary of other recent studies in health and nutrition. Of relevance to this study, Warren and Frongillo (2017) recently in Global Food Security, also using multi-site qualitative interviews, describe how the “system” and resources constrained the activities of mid-level actors working on nutrition sensitive programming in Ethiopia. Abbreviated research design Research question

1. How do individuals and organizations identify, evaluate, and select different processes or products for their post-harvest programming?

2. What are attributes (e.g., years at organization) of these individuals and their organizations that affect decision-making?

3. What criteria are used in evaluating and selecting among options for products and practices?

Data collection The research team used semi-structured interviews with theoretically sampled individuals from organizations across the Tanzanian post-harvest sector. Individuals were identified and recruited into the study because they had knowledge of why certain products or practices were included in their post-harvest programming. The research team used snowball sampling, a process by which one interviewee would introduce us to other

interviewees, as well as leveraged CITE’s and their personal networks to identify interviewees. Data analysis

We will use qualitative case-based analysis to study empirical data drawn from semi-structured interviews. Interview transcripts will be analyzed using coding to develop themes and answer the research questions. Codes will be used to annotate and draw connections among transcripts, allowing us to systematically identify differences across decision maker experiences and understand relationships among factors that affect decision making. This analysis aims to build or extend theory related to the role of decision makers in identifying, evaluating, and selecting products or processes for use in post-harvest programming. Case analysis methods from operations management (e.g., Eisenhardt 2007) and social sciences (e.g., George and Bennett 2005) will be used. Atlas.ti software will be used to code the various documents and aid in analysis. Trip Schedule To preserve the anonymity of our respondents, especially in cities in where there is a limited sample size, we present the itinerary and interviewee list separately. Our itinerary was as follows

M Jan 16 Nai./Dar M Jan 23 Dar M Jan 30 Dar/Iringa T Jan 17 Dar T Jan 24 Dar T Jan 31 Iringa W Jan 18 Dar W Jan 25 Dar/Arusha W Feb 1 Iringa/Dodoma Th Jan 19 Dar Th Jan 26 Arusha Th Feb 2 Dodoma/Dar F Jan 20 Dar F Jan 27 Arusha F Feb 3 Dar

And our interviewees are as follows1

Country Director Large IO

Operations Director Implementing Partner

Director Local NGO

Product Team Large, International NGO

Senior Leadership Implementing Partner

Program Manager Implementing Partner

Program Director Large IO

Sector Leader International NGO

Director Local NGO

1 Note this coding of organizations is very preliminary. Some organizations listed as implementing partners (i.e., operationalize and implement programs) may be closer to NGOs (i.e., shape programs and pitch ideas to donors), and visa versa. Likewise, some donors may actually be closer to NGOs. We will review the transcripts to find the best descriptions of these individual’s organizations.

Country Director Large, International NGO

Project Manager International NGO

Project Manager Implementing Partner

Ag. specialist Large, International NGO

Program Manager International NGO

Project Manager, International NGO

General Manager Implementing Partner

Field Coordinator Implementing partner

Consultant International NGO

Summary of primary findings Below is a preliminary summary of themes that emerged from the interviews:

No one particular way to identify products and practices: We asked each interviewee how they identify products or practices for use in the programming. Several used professional conferences to identify products. One used government best practice guides. Others relied on their organization’s or program’s networks to help them identify products or practices. Thus some individuals looked toward other field or even country offices within the same organization for insights, while others found ideas with their donors and other, co-implementing partners. Many, when discussing products, noted that it was often salesmen of particular brands that would bring technologies to them.

“Menu of options”: Many interviewees made a point of clarifying that they do not make decisions for farmers, but rather that they present farmers with a “menu” or “set” of options and let the farmer make the decision. The idea of the menu was nearly universal in interviewees’ comments. There is a nuance to this statement that we began to try teasing out in later interviews: if a product is not “right” for the farmers for one reason or another (e.g., reputably bad quality), decision makers may decide not to include it in the menu. Thus while they do not make “the final decision” for the farmers, they may make a decision at the outset that constraints the farmer’s decision space. That said, it is hard for people to, in hindsight, remember the ideas that never got absorbed in programming.

Role of government in the process: Our questionnaire did not include any questions about the role that government plays in either determining which products or practices are available or used in Tanzania, nor the success of their introduction to farmers. Yet several interviewees, unprompted, brought up the role that government plays (a) implicitly in technology identification, evaluation, selection and (b) explicitly in the implementation of a program.

Cost of products: Cost is a major factor in how decision makers characterize products. Due to time constraints, we did not ask whether the farmers for whom the decision makers were designing programs were credit constrained.

Appendix: Semi-structured interview guide First, I want to get a sense of you and your organization, and where you and your office fit in within it and its post-harvest loss work.

1. What organization do you work for? 2. How long have you been at [your organization]? 3. Have you personally completed projects related to post-harvest extension and

training before? a. Have you personally completed projects related to post-harvest storage

before this? b. Have you personally completed projects related to video extension

training before this? 4. How many people work in this office? 5. To which other office in [your organization] do you travel to the most? 6. From which other office in [your organization] do you receive staff from the

most? 7. Has your organization completed projects related to post-harvest loss before?

a. Has [your organization] completed projects related to post-harvest storage before this?

b. Has [your organization] completed projects related to video extension training before this?

8. How many times do you communicate with staff from [your organization’s] head quarters per week?

9. How many times do you communicate with people from [your organization’s] field locations per week?

10. Generally, who in your office makes key decisions related to products and processes used in programs?

11. Generally, who in [your organization] makes key decisions related to products and processes used in programs?

I want to understand, generally, your role in identifying, evaluating, and selecting among options for solutions to post-harvest loss in Tanzania.

12. Do you help make decisions in [your organization’s] programs related to post-harvest loss in Tanzania?

13. Can you tell me about a recent technology product or process related to post-harvest loss that your organization used in a program?

14. In you and your organization’s decision making, what were the steps in deciding on the technology product or process to use in the program? prompt for identifying, evaluating, selecting steps prompt for hold separate meetings for each step, do different people help with different steps

15. How long did this process take? What step took the longest? 16. Did you consider other products [processes]?

a. If yes: What other product [or process] options did you consider? b. If not: Why not?

17. How did you identify them?

a. Was it a group effort to identify them? i. If so, who led the conversation?

b. Did you need to gather information from sources (e.g., people, internet) outside of your office to identify them?

c. What information did you gather? d. Were there any barriers to gathering this information? e. Was there information you lacked?

18. How did you compare and select them? a. Was it a group effort to compare them?

i. If so, who led the conversation? b. What were some of the factors you used to compare them? c. Was it a group effort to select them? d. What were the factors that led you to eventually select what you did?

19. Do you feel like you made the right choice? What is one thing you did not know during their decision making process that they wish they did in retrospect?

References Abass, Adebayo B. et al. 2014. “Post-Harvest Food Losses in a Maize-Based Farming

System of Semi-Arid Savannah Area of Tanzania.” Journal of Stored Products Research 57: 49–57.

Addo, S., L. a. Birkinshaw, and R. J. Hodges. 2002. “Ten Years after the Arrival in Ghana of Larger Grain Borer: Farmers’ Responses and Adoption of IPM Strategies.” International Journal of Pest Management 48(4): 315–25.

Bokusheva, Raushan et al. 2012. “Factors Determining the Adoption and Impact of a Postharvest Storage Technology.” Food Security 4(2): 279–93. http://link.springer.com/10.1007/s12571-012-0184-1 (August 11, 2014).

Burke, Marshall. 2014. “Selling Low and Buying High: An Arbitrage Puzzle in Kenyan Villages.” Department of Agricultural and Resource Economics.

Eisenhardt, Kathleen. 2007. “Theory Building From Cases: Opportunities and Challenges.” Academy of Management Journal 50(1): 25–32.

George, Alexander, and Andrew Bennett. 2005. Case Studies and Theory Development in the Social Sciences. Cambridge: MIT Press.

Prusky, Dov. 2011. “Reduction of the Incidence of Postharvest Quality Losses, and Future Prospects.” Food Security 3(4): 463–74. http://link.springer.com/10.1007/s12571-011-0147-y (September 10, 2014).

Warren, Andrea M, and Edward A Frongillo. 2017. “Mid-Level Actors and Their Operating Environments for Implementing Nutrition-Sensitive Programming in Ethiopia.” Global Food Security. http://10.0.3.248/j.gfs.2017.01.010.

Massachusetts Institute of Technology - MIT Comprehensive Initiative on Technology Evaluation - CITE

Self-Employed Women’s Association of India - SEWA Bharat

Co-creating a Culture of Technology Evaluation in a Grassroots, Women-led Organization in India

PROJECT SUMMARY REPORT

April 2017

Author: Cauam Ferreira Cardoso

Research Assistant: Kritika Narula

Research Support from SEWA: Ian Mulholland

2

OUTLINE

1. Overview

1.1. Objectives

1.2. A brief story of SEWA and its Delhi operations

1.3. Research Questions

2. Methods

2.1. Research Design

2.2. Characteristics of the Sample

2.2.1. SEWA Members

2.2.2. SEWA Staff

3. Results and Discussion

3.1. Question A: Description of SEWA’s technology-related activities

3.1.1. Producer-owned Embroidery Cooperative (RUAAB)

3.1.2. Microfinance Services

3.1.3. Linking members with Government Schemes (SSKs)

3.1.4. Skills Development Programs

3.2. Questions B: Lessons Learned from the Implementation of Technology-related programs

3.2.1. “Technology Evaluation by Network” Can Limit Organizational Learning

3.2.2. Lead-user Bias Undermines the Validation of New Technologies

3.2.3. Staff and Members Learn about New Technologies Differently

3.3. Question C: How can technology evaluations best support SEWA’s mission in Delhi

3.3.1. Improve Technology Evaluations by Network by Creating Data-sharing Systems

3.3.2. Reduce Lead-user Bias by Incorporating Comprehensive Baseline Studies

3.3.3. Use Technological Learning as a Complement for Project Incentives

4. Conclusions

5. Recommendations

3

1. OVERVIEW

1.1. Background and Objectives

The Comprehensive Initiative on Technology Evaluation (CITE) at MIT and Self-Employed

Women’s Association of India (SEWA) have collaborated since 2015 in the implementation of the

project Co-creating a Culture of Technology Evaluation in a Grassroots, Women-led Organization in India. The

broad goal of this partnership is to assess how technology evaluations can add value to SEWA’s

existing activities in Delhi.

CITE’s research framework characterizes organizations implementing poverty alleviation projects as

facing a two-stage decision-making process, when choosing how to engage with technology. At the

framing stage, which takes place before any actual technological assessment is performed, organizations

ask questions about why, whether, and what technologies to use. For example, prior to asking which

brand of water filter makes the water cleaner for the lowest possible cost, the managers must decide

whether increasing access to drinking water should be prioritized over other local demands.

Afterwards, a new set of decisions emerges, with dissimilar characteristics. The procedural stage focuses

on how to evaluate a set of technological options available within a given market. That is, the

organization has already set up its priorities, and must now make a narrower choice based on

specific information about existing products. In the case of solar lanterns, for instance, this means to

ask which brands are the easiest to use, have the lowest price, and offer the best value for money.

The current research effort focuses on the Framing Stage of a technological assessment. It entails the

development of a strategy for the identification of social, organizational, and contextual issues that

can influence SEWA’s technological choices. Our assumption is that increased awareness of such

factors will help the staff make informed decisions about the demand for, and potential impact of

the technology-related services offered to their members. Our expectation is that the time and

resources spent on the co-production of this project will improve the quality of SEWA’s work, as

well as generate lessons that CITE can use to develop its own, broader evaluation methodology.

With this summary report, we share a snapshot of the preliminary findings produced so far.

Therefore, this document does not contain the final conclusions of our work. There is more data to

be analyzed, which may or may not change our interpretation of the research results. It is our hope,

however, that our preliminary findings will provide a useful background for SEWA and CITE teams

as they continue to co-produce more effective technology evaluation methods.

4

1.2. A Brief History of SEWA and its Delhi Operations1

The Self-Employed Women’s Association of India (SEWA) is a women-led labor union founded

under the leadership of Smt. Ela Bhatt in Gujarat, India, in 1972. It was the first organization of its

kind to organize women working in the informal economy. Over 45 years later, SEWA has become

a federation of local grassroots organizations with a total of 1.9 million members in 13 Indian states.

In addition to labor organizing and advocacy, SEWA provides a number of services aiming at

increasing economic and social autonomy of women in the informal sector, which include

microfinance, support to cooperatives, skills training, and linking members to government schemes.

In 1984, SEWA Bharat was established in Delhi as part of the expansion of SEWA organizations

throughout India. The entity’s goal was to formalize a federation of local grassroots associations

under a unified national identity. SEWA Bharat provided start-up resources and advice to new local

associations and promoted knowledge exchange between existing groups, at the same time

harmonizing organizational practices and strengthening service quality. In Delhi, SEWA Bharat

provided critical support to the local association to between 1999 and 2007, after which SEWA

Delhi formally became an independent organization. SEWA Bharat and SEWA Delhi have

continued to collaborate on specific issues, especially in areas like leadership training, management

strategy, and project monitoring and evaluation ever since.

Both because of this organizational restructuring, and in recognition of the multi-dimensional

challenges affecting female informal workers in India, SEWA Delhi has continued to expand its

portfolio of services. Campaigns promoting women and labor rights, and social welfare projects

implemented with the support of external partners are some of the most important ones. But

increasing demand has also led to specialization and decentralization, and consequently the creation

of semi-autonomous sub-organizations, such as the Mahila SEWA Urban Thrift and Credit Cooperative

(microfinance services, 2007), SEWA Shakti Kendras (SSKs) (linking members to government

schemes, 2007), RUAAB (embroidery cooperative, 2010), and SEWA Polytechnic (skills training

programs, 2011). Today, SEWA reaches over 53,000 members in Delhi across 6 areas, namely

Jahangirpuri, Raghubir Nagar, Sunder Nagri, Rajiv Nagar, Mullah Colony, and New Ashok Nagar.2

1 Section written based on information from SEWA Bharat, 2013 Annual Report. Available at: http://sewadelhi.org/wp-content/uploads/2014/06/sewa-delhi-final-report-ab2.pdf 2 The study also contains references to two additional field offices – Mustafabad and Anand Vihar – which were operational when the data were collected but have since closed.

5

Figure 1 – SEWA Delhi Timeline

1.3. Research Questions

The broader motivation for this project was born out of SEWA’s need to make decisions involving

technology more systematically – thereby improving their capacity to identify the products and

services with the best chance to work for their members. Our strategy consisted of an exercise of

deep organizational learning, in which researchers looked for answers based on SEWA Delhi’s own

experience implementing technology-related projects, rather than relying on case-studies of other

peer NGOs. Specifically, research activities revolved around three questions:

A. What are the technology-related projects SEWA Delhi is currently developing? How

are they implemented?

B. What are the lessons that can be learned from SEWA Delhi’s experience in

implementing technology-related programs?

C. How can technology evaluations best support SEWA’s mission in Delhi going

forward?

6

The answer to questions A and B will provide a basis for an organizational roadmap that will

inform the design of future technology evaluation processes. In fact, the very methodology used in

this research has been designed in a way that SEWA can use it independently as a template to

identify organizational and contextual priorities that can be addressed by technological

interventions. Furthermore, in responding to question C, we intend to:

i. Identify the broad technological areas that could be targeted for future evaluations, based on

the findings from questions A and B;

ii. Recommend strategies for technology evaluation within SEWA Delhi, given the contextual

characteristics of the areas in which the organization operates, and considering its current

implementation capabilities.

2. METHODS

2.1. Research Design

Following Yin’s (2013)3 classification, the research is best characterized as a revelatory, single case study.

This methodology is used when researchers are allowed broad access to an organizational context in

ways that had not been previously available to other researchers. Although SEWA has been involved

in a variety of academic efforts over the years, it is the first time a study is looking at how

technology-related projects have been implemented in Delhi. As a result, our findings are to a large

extent descriptive, but have value because these insights can help SEWA and other peer

organizations to learn broader and original lessons from the work they perform.

In addition, our approach follows an embedded case study design4, joining a group of recent research

efforts5 that examine the implementation of projects within specific socio-political contexts. This

design allows us to specify technology-related projects currently implemented by SEWA-Delhi as the main unit

of analysis, and at the same time examine these projects within the larger system of policies,

organizations, and populations they are embedded. 3 Yin, R. K. (2013). Case study research: Design and methods. Sage publications. 4 Ibid. Yin (2013) 5 See for example, Bhatt, P., Ahmad, A. J., & Roomi, M. A. (2016). Social innovation with open source software: User engagement and development challenges in India. Technovation, 52, 28-39; Bhatt, P., Ahmad, A. J., & Roomi, M. A. (2016). Social innovation with open source software: User engagement and development challenges in India. Technovation, 52, 28-39; and Hoey, L. (2015). “Show me the numbers”: Examining the dynamics between evaluation and government performance in developing countries. World Development, 70, 1-12.

7

Following a mixed method approach, we conducted qualitative primary data collection through

semi-structured interviews and focus group discussions with the staff implementing technology-

related projects, as well as its beneficiaries. Having both perspectives captured in detail is paramount

for the development of accurate assessments of the projects’ performance and identification of

lessons learned. We further triangulate these findings by reviewing project documents and analyzing

quantitative data from databases containing demographic and behavioral information from SEWA

members.

2.2. Sample (Primary data collection)

Primary data collection was conducted in January, July and August of 2016 resulting in 245

interviews (64 staff and 181 members) across the eight regions of Delhi where SEWA works, as well

as in its headquarters. The sampling strategy was guided by two main principles: first, to identify the

people who were best positioned to answer the research questions at hand and; second, compose a

group of respondents that is representative of the SEWA staff and membership in Delhi more

broadly. A summary of the data collection activities is displayed in Figure 4 (below):

Figure 4 – General Characteristics of the Sample

8

The recruitment of participants was conducted in consultation with SEWA staff, but the semi-

structured interviews and focus group discussions including SEWA low-level employees and clients

occurred without the presence of senior staff, in condition of anonymity, and after formal consent.

The project team collecting the data was composed by a MIT researcher accompanied by a female

translator and a research assistant responsible for recording the interviews and take notes for future

coding. All events followed the same five-fold structure:

i. Survey collecting basic demographic information on the participants

ii. Questions about the respondent’s background (ex: where do they come from, their trade, etc.)

iii. Questions about the respondents’ relationship with SEWA (ex: which kinds of services they use or

projects they work on, and their overall experience with them)

iv. Questions about the role of technology in shaping respondents’ relationship with SEWA (ex: how does

the introduction of computers in field offices changed the ability of the staff to get their work

done, and, from the perspective of users, how it has changed their level of satisfaction with

SEWA’s performance)

v. Questions about the role of technology in respondents lives beyond SEWA (ex: how the increasing

availability of mobile phones has changed the nature of their work, or power relationships

within the household).

2.2.1. SEWA Members

To strengthen the external validity of the research findings, we used the SEWA member database to

recruit participants that generally represent the socioeconomic characteristics (age, educational level,

occupation, and geographical location) of the broader SEWA membership (see Figure 5). However,

there are some noteworthy exceptions that were caused by constraints that emerged during

fieldwork, which go from the lack of available participants to prioritization of groups that are more

relevant to our specific research questions.6

6 Location: The sample is over-represented in Jahanjipuri and under-represented in Mustafabad. Occupation Type: one of the key objectives of this study was to examine the operation of skill development programs, the sample has a greater proportion of “students”, which reflects the prioritization of data collection from this population. Age Group: The proportion of both “adolescents” and “senior workers” is overrepresented. In the former case, this is partly due to the high proportion of skills training programs participants. In the case of “senior workers”, the cause is less clear: older workers were easier to recruit despite of our similar attempts to reach both groups. Educational Level: The greater proportion of participants having “completed secondary education” is partially a consequence of a higher proportion of students from skills training programs (which tend to be more educated than the average SEWA member).

9

Figure 5 – Comparison between Sample (in blue) vs. SEWA Members (in green) Socioeconomic Profiles

2.2.2. SEWA Staff

Since the research team did not have access to a complete database with information about the staff,

the research was designed to capture diverse points of view, across work locations but also: i)

organizational hierarchy; ii) professional expertise; iii) length of employment with SEWA and; iv)

project lines. The total sample (n=64) consisted of approximately 25% of all SEWA Delhi

employees and is described in greater detail in Figure 67 (next page):

7 In analyzing the sample, it is important to note, regarding the specific variables adopted:

Position: The staff was divided in three different hierarchical categories. Given that that the number of people in higher positions is smaller than low-level ones, managers were overrepresented in the sample. Still, 70% of the participants work directly with members, giving them a dissimilar vantage point to observe how the organization works.

Program: There is a considerable level of autonomy within SEWA operations, which is coupled with a diverse portfolio of activities, from microfinance to skills training. As such, participants were recruited from programs in which technology has an important role, and in numbers that are proportional to the size of their operations.

Expertise: The sampling reflects the variety of professional backgrounds that exists among the staff.

Length of association: The recruitment of research participants was designed to capture the opinions of staff with different levels of seniority within the organization. The distribution of respondents according to this criterion sought to be proportional to their respective populations within SEWA.

10

Figure 6 – SEWA Delhi staff profile

3. RESULTS & DISCUSSION

3.1. Research Question A: What are the technology-related projects SEWA Delhi is currently

developing? How are they implemented?

The research identified technology-related project areas according to the following criteria:

• Systematic Technological use: Recent acquisition of a software or hardware by the project to

achieve at least one of the following goals:

o Improve the capacity of the organization to implement its activities with better quality and

at a higher scale;

o Introduce a new tool used in trainings for members interested in learning technologically

advanced skills;

• Length of Operations: The project has been implemented for at least one year and continues

to operate in Delhi. This indicates that there was enough time to assess meaningfully the

project’s performance, and the experience of both staff and users.

11

• Scope: The project is using at least one of the technologies acquired beyond the pilot stage.

That is, the technology has influenced the implementation of activities in most the areas where

SEWA Delhi operates (equal to or more than the area covered by 4 field offices).

Following these criteria, we identified four project areas8 (Figure 7, below, indicates the types of

technology used):

• RUAAB: a livelihoods project supporting a producer-owned embroidery cooperative;

• Microfinance: a micro-credit cooperative that helps SEWA members to create and maintain

saving accounts, as well as provides loans;

• SEWA Shakti Kendra (SSKs): a service that connects members with a wide range of

government schemes, in addition to legal assistance and business registration support;

• Skills Training: projects that support members in the building of marketable skills, such as

advanced garment making and computer literacy.

Figure 7 – Technology-related programs currently implemented by SEWA Delhi

8 For reference, examples of projects areas that were not included in our analysis because they did not fit the criteria are: SEWA union initiatives for community mobilizing and public lobbying, pilot projects in public health, after-school tutoring programs, among others.

SEWA Technology-related Programs in Delhi

Livelihoods (RUAAB)

Inventory Software

Sales Software

Tech. Rewards Program

Microfinance

Mgmt. Software

Airtel Money App

App for Bank Saathis

Govt. Services (SSKs)

Member Database

Skills/Youth Development

Pankh Portal Software

Adv. Garment

12

3.1.1. Livelihoods Program (RUAAB)

RUAAB is a producer-owned embroidery cooperative led by skilled SEWA members. Their work

currently reaches an estimated 200 artisans in four locations across Delhi -- over 800 beneficiaries

since 2010. Originally, the cooperative was formed to aggregate workers so they could take orders

from larger clients and, consequently, increase their earnings. But eventually SEWA started its own

brand, Loom Mool, taking advantage of the silk cooperative active in Bihar and its 150 artisans.

Initially, RUAAB and Loom Mool operated separately, though they started collaborating shortly

after, in most cases, dividing the labor between finishing of products and production of raw

materials, respectively.

Both organizations are still on their way to stabilizing their businesses. Still, they have been able to

reach a diversified portfolio of clients, which includes exports to countries like Germany and the

Philippines. SEWA continues to support the cooperative by providing management support, quality

control, and the option to use one of the four production centers to conduct their work (although

many chose to work from home).

Technology has played an increasing role in RUAAB’s evolution in at least three ways. The first

started with the adoption of two software applications -- one that controls inventory and another

that supports sales –, which were acquired to improve operational efficiency and support scaling-up.

Interviews with staff and artisans indicated that the implementation of these computer programs

created mixed results. On the one hand, the new systems did enhance the staff’s ability to coordinate

clients’ orders with their internal production chain. On the other hand, challenges with staff training

and software limitations prevented the organization to fully benefit from these applications.

A similar result happened when RUAAB started to sell products online. The internet offers an

attractive market for SEWA products, especially due to the rise of an informed clientele that

prioritize brands that maintain an economic and social mission. Entering e-commerce proved to be

difficult, though, because of the speed and efficiency that it requires from suppliers, at least in the

case of large online platforms. SEWA had to adjust its strategy over time, focusing on niche websites

that are less popular but more flexible about delivery dates.

Finally, the third technology-relevant feature of the project is a rewards program that started in 2016,

through a partnership with a professor from Columbia University. The project developed a system

13

in which members earn points according to their productivity, reliability, and even some behavioral

indicators, such as sending their children to school. As one of the respondents explains:

Women artisans [that] are giving their 100% quality, keeping their houses clean, sending their kids to school; all these activities have points attached to them like 20 or 50 points. If they keep their community clean, its 100 points. We do some random checks also. So a verification person goes and checks all this. Once all these points are collected, then there are redemption things. For example, for 4000rs, there is a table fan, for 500rs, there are food coupons for mother dairy milk, and then there are school uniforms for kids, toys, emergency kit, spectacles, and radio. - RUAAB Staff [KII 013]

SEWA staff reported a sustained positive response from members during the year, which was

corroborated in interviews with artisans. However, the future of the program remains uncertain

because of the need of an external partner for financing and technical advice.

Figure 8 – Images of RUAAB activities

List of Products offered by the RUAAB rewards program

SEWA Staff assessing the quality of an embroidery piece.

3.1.2. Microfinance

Founded in 2007, the Mahila SEWA Urban Thrift and Credit Cooperative provides microfinance services

to more than 7,000 women in Delhi. Opening and maintenance of savings accounts correspond to

70-85% of all operations, while 15-30% of clients request loans. The average size of a loan between

2007 and 2015 was approximately 32,000 Rs, with the mean monthly instalment of 2,000 Rs.

Loan services at SEWA operate similarly to other mainstream micro-finance organizations: it targets

women, and stipulate the amount borrowed according to the potential for repayment and amount

14

deposited in the savings account. Over the years, however, there has been a considerable shift in

relation to the kinds of activities SEWA was open to provide loans for. As Figure 10 shows, most

borrowers would take money to finance productive activities, but since then the cooperative’s

portfolio became much more diverse. Credit for housing, debt payments, education, health, and

social obligations such as funerals or weddings turn out to be more common, while only a third of

loans continued to be used in productive activities.

Figure 9 – Proportion of Loans provided by SEWA by Type (2008-2015)

Achieving this scale was only possible due to SEWA’s technology acquisitions, especially different

computer programs that would allow systematic data analysis and record-keeping. SEWA is

currently on its third custom-made software (called BIJILI), acquired in 2013. This program has

since allowed the staff to increase operational efficiency by reducing the time spent in data

processing, as well as by improving the quality of services via digitalization. At the same time,

similarly to RUAAB’s example, the implementation of a new software also created challenges related

to the training of skilled personnel. The process of transferring paper files to the system online took

several years to complete and required sustained efforts from project managers.

15

With the emergence of new mobile phone applications and renewed interest from donors to support

financial inclusion activities, two new proposals are currently being considered by the cooperative.

One is a nascent partnership with Airtel (Indian telecommunications company), which intends to

support SEWA members to sending remittances through the Airtel network, expanding SEWA’s

existing capabilities. The other is the development of a new mobile app compatible with BIJILI,

which would allow microfinance field workers to have a simplified platform to manage their clients.

Figure 10 – Images of Micro-finance activities

Figure 8 – Focus Group Discussion with Microfinance Clients

Figure 9 – Payment slips of microfinance loans

Indirectly, the relationship between technology and microfinance is also important because at least

some of the resources distributed by the program are used to purchase technological goods and

services. The staff confirmed during the interviews that there have been conversations about

opening lines of credit for specific products that are in high demand and, at the same time, have

good potential for generating positive social impact. This is the case of cellphones, computers,

among other electronic devices.

3.1.3. Connection to Government Services (SEWA Shakti Kendra - SSKs)

SEWA has been connecting its members to social schemes offered by the government since its early

stages of operation in Delhi. But in 2007, this relationship was formalized and expanded through the

implementation of Gender Resource Centers (GRCs) throughout the city. GRCs were designed as

an institutional tool to improve access to public services that until then were inaccessible to women

16

living in poverty. These included the preparation of applications for identification and ration cards,

skills training, legal assistance, and information about a variety of schemes in areas such as social

security, health, and education.

Partly in recognition of the important role of civil society organizations already had in the delivery of

public goods to vulnerable populations, public-community partnerships were established to facilitate

the implementation of the program. SEWA was selected as one of the program’s grassroots

collaborators, becoming part of the network of over 100 GRCs and 2,000 workers in charge of

managing day-to-day operations9. Although managerial and budgetary issues led the government to

shut down the project a few years later, SEWA has continued to offer GRC-related services

independently with the help of external funders. Between 2010 and 2013, SEWA has linked 23,742

people linked to government schemes10, while this year the number has decreased to 5,000.

Technology has influenced the implementation of this program both in terms of supply and

demand. The supply of government schemes has become increasingly digitized, following the

expansion of “e-government” systems. Such systems often require the submission of applications

for services online, or utilize machines to capture individual information such as fingerprints. From

the demand side, technological change has been constantly redefining the requirements for social

and economic empowerment. For example, the computer skills have become an important tool for

accessing high-paying jobs, while the proliferation of mobile phones have created, alongside

communication and information benefits, challenges with online harassment and technology-

enabled violence against women that often require specific legal counseling.

SEWA has itself undergone technological transformations by adopting a custom-made software for

client management. Since 2015 the organization has been able to register information about the

clients, create records about their requests, and follow-up on each case until the service or inquiry is

satisfactorily completed. Like other software used within the organization, the research has identified

challenges with training, infrastructure, and management of workflow (e.g., relying solely on

computers can be harder during peak hours, when many clients come to the center at the same

time).

9 The Times of India (2016) “Govt pulls the plug on gender resource centres”, (Online edition of 05/29/2016).Available at: http://timesofindia.indiatimes.com/city/delhi/Govt-pulls-the-plug-on-gender-resource-centres/articleshow/52484932.cms 10 SEWA Delhi website. Available at: http://sewadelhi.org/resources/factsheets/

17

Figure 11 – Images of SSK activities

SEWA staff collected the fingerprints of a client’s son

as part of the identification card application.

CITE researchers interview one of the SEWA staff in

charge of SSKs.

3.1.4. Skills Development Programs

Supporting female informal workers to improve their professional and personal skills has been at the

same time one of SEWA’s priorities and an area intensive experimentation. Whereas “training”

activities have been regularly conducted since SEWA’s early stages in Delhi, the variety of courses

offered is such that it was difficult to identify one comprehensive list including all of them. Broadly,

it is possible to group the courses offered over the years in four different areas: i) after school

programs for members’ children; ii) hands-on professional courses (embroidery, fashion design,

advanced garment making, beauty parlor, henna application, among others); iii) English and

computer literacy courses; iv) Soft skills activities (career counseling, job preparedness courses,

leadership, peer-to-peer mentorship, among others).

In addition to their diversity, SEWA Delhi skills development projects have also been characterized

by the short duration and variety of external partners associated with each of them. Many of such

activities were established as standalone pilot projects, but others have been implemented as the

“training” component of broader initiatives. For example, in 2015 the United Nations Development

Program partnered with SEWA on a project focused exclusively on skills, in which members were

offered career counseling services as well as English and computer literacy classes. Alternatively, the

aforementioned Gender Resource Centers program created in collaboration with the government of

Delhi included vocational training as one of the many activities taking place within the buildings.

18

Figure 12 – Images of Skills Training activities

SEWA Polytechnic classroom for garment-related

courses

SEWA researchers interviewing the instructor of SEWA

advanced garment making course

A degree of permanence was added to these skill development efforts in 2011, with the founding of

the SEWA Polytechnic. This new space enabled the organization to concentrate at least part of its

training activities into one location, and have the necessary infrastructure (equipment such as

computers, sewing machines were installed in the classrooms) to offer courses with quality and at

scale. Despite these advances, there are only two projects currently operational: the UNDP project

(career counseling sessions were provided to 13,000 adolescents, and job preparedness were

attended by 300) and one advanced garment making course (200 students currently enrolled).

From a technological perspective, as one SEWA Delhi reports published in 2014 underlines, “skill

building has emerged as a new thrust area for unorganized workers, to adapt to changing

technologies and labor market demands” (p. 22).11 Increased productivity and access to higher

paying jobs are now mostly available to workers who can familiarize themselves quickly with new

equipment and information platforms. But most importantly, a dynamic economic environment

requires (especially) from women a higher degree of self-confidence in their capacity to learn

techniques and enter industries previously only provided to men. In this context, technical knowhow

and soft skills are equally important for sustainable economic and social mobility.

11 ibid. SEWA (2013)

19

Figure 13 – Images of Skills Training activities

CITE researcher interviewing students from SEWA

job preparedness course

Computer literacy student taking a lesson on typing

SEWA has incorporated new technologies in the delivery of its own skills programs too. Advanced

garment-making course uses sewing machines identified as mechanically reliable and economic

accessible to the students. The designs and materials used in class are supposed to follow what is

currently in demand. Keeping up with technical trends while prioritizing access and affordability are

some of the key challenges the instructors must take on. The UNDP project, on the other hand, has

computer literacy courses as part of its core activities. It also created a custom-made online platform

(Pankh Portal) for women and girls seeking job-market information. Overcoming infrastructural

barriers, such as internet access and availability of computers and/or smartphones at members’

households is one of the most visible issues faced by the project. However, existing social norms,

gender-based violence, and other structural factors play an even greater role in preventing these

kinds of opportunities from reaching a wider audience.

3.2. Research Question B: What are the lessons that can be learned from SEWA Delhi’s

experience in implementing technology-related programs?

3.2.1. “Technology Evaluation by Network” Can Limit Organizational Learning

SEWA currently adopts a technological decision-making strategy that is relatively efficient. The

organization keeps the costs low by having few in-house experts, and relying on third-party technical

20

advice whenever there is a need for it. They leverage a range of professional networks – across

SEWA units in other parts of India, NGOs, donors, and the staff professional contacts – to ask

questions about which products to use inside the organization and with clients. When more detailed

advice is needed, they rely on consultants and other contractors to help. Moreover, this evaluation by

network is also useful because it works as an information exchange mechanism by which SEWA

continues to learn from others about what is new in their field. All four projects analyzed by this

study make technology decisions following that same strategy.

While the flexibility and low-cost fit well with SEWA’s interest to limit overhead and minimize

disruptions caused by fluctuations in revenue, this approach has important limitations. Among the

most important of them is the fact that relying excessively on external knowledge leaves the

organization with few avenues for internal technological learning. Evaluation by network facilitates the

implementation of specific projects in the short-term, and it is adequate for projects in their pilot

phase. In the medium and long-term, scaling-up with this same strategy weakens project

performance because while the size of operations grows, knowledge asymmetries remain the same.

Every time a contract with a consultant ends, or one key staff leaves the organization, most of the

knowledge leaves with them.

3.2.2. Lead-user Bias Undermines the Validation of New Technologies

One of the main characteristics of SEWA’s operations in Delhi is the quantity of new projects being

piloted every year. As an organization with national and international reputation, SEWA faces no

shortage of foundations, government agencies, and companies seeking to start new partnerships.

Connecting this new supply of products and services with the people who need them the most is

one of SEWA’s comparative advantages. But doing it fast and simultaneously on multiple fronts, has

important implications.

The variety of new pilots tends to be negatively correlated with their duration, implying that

decision-making processes, technological or not, must also be accelerated. The pattern that emerged

from the four projects we studied was that such speed was possible because decisions are made by a

relatively small number of people. In every case analyzed, both staff and SEWA members had been

at some point consulted about the new technology before it was formally acquired. However, the

people who composed this decision-making group had similar characteristics: one or two top

managers, one or two field workers, and five to ten community leaders or members.

21

Such practice can be described as one version of what Von Hippel (1986) 12 calls a lead-user strategy:

Lead users are users whose present strong needs will become general in a marketplace months or years in the

future. Since lead users are familiar with conditions which lie in the future for most others, they can serve as a

need-forecasting laboratory for marketing research. Moreover, since lead users often attempt to fill the need they

experience, they can provide new product concept and design data as well. (p. 791)

Lead-user strategies are widely known as a tool to assess demand for products and services, and are

particularly useful for short-duration pilots because their limited target audience is more likely be

compatible with the small group of people consulted during project design. On the other hand, this

assumption compatibility is not necessarily applicable to SEWA’s membership base, which is highly

diverse. At SEWA, this risk is compounded by the fact that the high turnover of projects gives an

incentive to team leaders and lower level staff to reach out to people they know well, or have a close

work relationship with.

As sociologist Mark Granovetter (1973)13 noted, the key characteristics of groups formed by people

with strong ties to each other is that participants tend to have (relatively) similar opinions about

fundamental issues. As a result, when it is time to scale, this small and homogenous group making

the decision will likely create a bias in the validation of the project designs, impacting the ideas,

products and services that are selected. The final decision is therefore representative of the views of

the group making the decision, and not necessarily of the larger audience it is supposed to reach.

The data from the interviews and focus group discussions indicated that there seems to be a

correlation between the type of project and the intensity of the bias, though, measured in terms of

the demand for the service or product offered. Skills training courses and livelihoods activities such

as the creation of producer-owned cooperatives seemed to be more susceptible to this kind of bias,

given the challenges research participants reported to find new clients. This was not the case for

services like those provided by SSKs, which experience stable demand for their activities. Going

forward, reducing user bias when selecting technologies is likely to require a more comprehensive

analysis about the nature of the project itself and the people being consulted for validation of its

design.

12 Von Hippel, E. (1986). Lead users- a source of novel product concepts. Management science, 32(7), 791-805 13 Granovetter, M. S. (1973). The strength of weak ties. American journal of sociology, 78(6), 1360-1380.

22

3.2.3. Staff and Members Learn about New Technologies Differently

Technological learning is the process through which a new product (hardware or software) is

incorporated in the organization creating new routines and procedures and, in turn, generating new

or improved output. From our interviews with the staff at various hierarchical levels and members,

we could detect a stark difference between how one sees technological change affecting their work

and relationship with the SEWA depending on where they were in the organization.

Top level managers were usually open to new products and services that helped them to control and

scale their operations. In the case of microfinance, for example, BIJILI gave managers the ability to

maintain default payment at sustainable levels, as well as follow the growth of their portfolio in real

time. From the point of view of the lower level staff, however, this represented a challenge. Going

from a paper-based to a paperless system includes a considerable amount of training with unfamiliar

technologies, in addition to a perception of loss of ownership over their expertise and work material

– having one’s files, with one’s handwriting, was described by respondents as necessarily different

from having documents stored in a computer network.

Another interesting example was found at RUAAB, where managers decided to use smartphones

and Whatsapp to receive live updates from workers operating remotely. In this case, weavers resisted

the new technology not because of their unfamiliarity with mobile phones, but because it increased

managers’ control over them. Weavers often have side jobs and therefore prefer a flexible schedule,

so the introduction of smartphones prevented them from allocating their time and work as they saw

fit. Eventually, most master weavers started to use smartphones, but it took time for all people

involved to adapt.

From the perspective of SEWA members, the response was more diverse, following three distinctive

patterns. One group expressed that the incorporation of new technology changed their perception

of SEWA as a “modern” organization. For instance, the establishment of computers in SSK offices

was mentioned by participants as encouraging them “to take SEWA more seriously”. Similarly, some

micro-finance clients appreciated the fact that being able to see their account details in the

computers located at SEWA field offices helped them to have more confidence that the information

was true. This was not the case, though, when clients were offered transactions confirmations via

SMS, rather than by paper receipt. Many still wanted a hardcopy that they could keep at a safe place.

As it was explained to us, male household members often have control over mobile phones, which

makes it harder for women to keep that information private.

23

3.3. Research Question C: How can technology evaluations best support SEWA’s mission

in Delhi in the future?

Apart from the identification of SEWA’s technology-related projects, and the lessons that can be

learned from their implementation, there is a complementary set of findings from our study that

speak more broadly to the role of technology evaluations as opportunities to improve technological

learning within the organization. In the following sections, we approach each of the same five

lessons learned from the perspective of what can be done to address them in the process of

conducting a technology evaluation.

3.3.1. Improve Technology Evaluations by Network by Creating Data-sharing Systems

SEWA relies on a complex network of professional and organizational partners that are consulted

when a decision about which technology to acquired needs to be made. As mentioned before, the

solution is both efficient and incomplete. Efficient because it does not rely on fixed resources (high

overhead) for decision-making, but it is incomplete because by following this strategy SEWA

continues to depend on external partners to make important decisions.

Reversing this trend would involve the adoption of different tools from those currently available.

One of the options would be to build in-house expertise on technology evaluation. New or

internally promoted staff could be allocated on an “intelligence unit”, whose function would be to

conduct research on technological trends, define protocols for product selection, and perform

evaluations internally. However, the cost implied in hiring skilled personnel and building the

necessary infrastructure is excessively high, especially when one accounts for all the different

technical specialties required in a portfolio of projects that covers areas as dissimilar as microfinance

and skills training.

Furthermore, the idea of an “intelligence unit” is not necessarily compatible with the kinds of

technological choices that SEWA has to make in Delhi, which are both sporadic and discrete. That

is, it is difficult to predict when decisions about technology have to be made during project

implementation, and when they do, the advice needed is usually specific to that project at that time,

and not necessarily relevant to future decisions. Advice regarding, for instance, which financial app

to adopt, which device to use to collect fingerprints for government IDs, or which sales

management platform for RUAAB to acquire, is the most valuable just prior to purchase. Recruiting

24

and maintaining such diverse and specialized staff for that purpose alone would be not only difficult

but also very expensive.

On the other hand, there are ways to build on the existing “evaluation by network” system in a less

disruptive fashion. If the key problem with the current system is that it limits technological learning,

evaluations can be used to improve organizational capabilities by matching them with tools that

increase staff communication and data sharing. For instance, it was a recurrent theme during

interviews the existence of “silos” during project implementation that were reinforced by the

technologies they used. Each of the four projects analyzed uses a different custom-made software to

manage their operations, and none of them are compatible with one another.

Data-sharing benefits individual projects because it exposes their staff to a larger pool of

information. It is often the case that microfinance clients are the same people that use SSK services

and send their daughters to skills training courses. The combination of insights from these differed

initiatives can be helpful both in terms of providing a more comprehensive understanding of clients’

needs and allows staff members become familiar with activities beyond their specific job description.

It is a system that is at the same time more integrated and less dependent of individual employees.

3.3.2. Reduce Lead-user Bias by Incorporating Comprehensive Baseline Studies

The existence of multiple, concurrent pilots allows SEWA to maintain a dynamic testing ground to

innovative ideas. Pilots have to be conducted within a short timeframe, though, precluding the staff

from spending too much time validating each initiative. In this context, lead-user bias can have a

negative impact during project implementation when the group selected to validate the usefulness of

a new technology is not representative of the population that it is designed to reach. The

implications of such bias can vary depending on the nature of the project – lower influence on SSK

services, and higher on livelihoods and training projects --, but the risks involved are important

enough to deserve the attention of SEWA managers from all areas.

Grassroots organizations like SEWA build their comparative advantage vis-à-vis other development

stakeholders -- such as government agencies and international NGOs -- by having a better

understanding of what their members need and want. However, our research indicates that what

constitutes “members”, “needs”, and “wants” changes significantly over time. SEWA’s membership

in Delhi has a high turnover due to migration, forced displacement, and other factors affecting the

geographical mobility of informal workers in the city. At the same time, economic growth,

25

technological change, and changing social norms have a direct impact on the specific needs

members have, as well as their perception of how SEWA can help them.

In such conditions, maintaining an up-to-date understanding of their members vis-à-vis these

dynamic processes poses many organizational challenges. SEWA’s work already entails a significant

amount of interaction between staff and members as they build a relationship that both can trust.

And when turnover is high, this means that the staff must dedicate a large share of their time and

resources trying to build relationships with new clients to replace those who left. Having to hit a

moving target is not only hard; it can also prevent the organization from expanding services to a

broader population.

SEWA has been dealing with these and other structural issues for a long time, but there seems to be

a lesson to be learned in terms of not taking “demand for granted” during implementation – a

phenomenon also called the “need equals demand fallacy”14. In many ways, SEWA has a variable

clientele that needs to be “re-learned” and “re-characterized” before new activities take place. This is

the case when pilots are established targeting a general audience of female informal workers and it

does not have the intended impact because it failed to recognize the diversity of backgrounds, needs,

and wants SEWA members have, at any given time.

Technology evaluations offer an opportunity for reducing lead-user bias because they allow the staff

involved to learn more about their members before a new technology is acquired. In this context,

comprehensive baseline studies represent a counter example to lead-user strategies by following

three main principles: triangulation, contextualization, and historical analysis. Triangulation means

that it is important to validate the adoption of new technologies and project ideas with staff working

at various hierarchical levels and clients living in different areas of the city – preferably from

different backgrounds. Such strategy is more likely to reduce bias because it is based on a more

diverse sample.

The identification of lessons learned from projects implemented in the past is important because it

prevents new projects from repeating mistakes unnecessarily. Besides, familiarizing the staff with the

experience of earlier teams is one of the ways to build institutional memory. Finally,

contextualization refers to research efforts that look for information outside the boundaries of the

organization. Data from censuses, for instance, include information about both SEWA members

14 The “Need Equals Demand” Fallacy concept was discussed first by Hussein, T & Plummer, M. (2017) Selling Social Change. Stanford Social Innovation Review. Available at: https://ssir.org/articles/entry/selling_social_change

26

and non-members, providing a better picture of the structural factors influencing the live in these

communities. Evaluating technologies in this broader context can help staff signal potential

bottlenecks for project implementation and future expansion.

One critical observation, though, is that not all types of projects require baseline studies of the same

depth. For example, SSK services have been consistently highlighted by research participants (both

staff and members) as the most valuable contribution SEWA provides to their communities. They

praised the fact that it is a female-friendly space that provides accurate information about essential

public goods (i.e. education, social security, health, etc.), at a low cost. In part, the nature of SSK

services is helpful in explaining this outcome: they offer a transaction-oriented services, where the

product is well-codified and the benefit is clear. More importantly, it demands relatively little from

clients that only have to visit SEWA’s office a limited number of times.

This is almost the opposite experience of skills training projects, where only two courses are still

operational despite the establishment of many different courses over the last decade. These types of

courses involve deeper relationships of trust between the staff and their clients, have a product

whose utility is less certain (there are many variables at play between receiving the training and

getting a job), and require consistent interaction.

As we have learned, activities contingent on women’s or girls’ continued attendance to field offices

are much harder to maintain. Issues such as lack of money, safety concerns, and restrictive social

norms are just some of the structural factors raising the threshold for how much women must

overcome to be part of activities promoted by SEWA. In this case, the transformation of the

perceived need for new skills into an effective demand for training opportunities is inherently

different (and more complicated) than matching people seeking government IDs and the agencies

providing them.

3.3.3. Use Technological Learning as a Complement to Project Incentives

Once it is established that staff at different hierarchical levels learn about new technologies

differently, one of the key issues that emerge is the relationship between learning and incentives.

Incentives mediate the process taking place between the acquisition of a new technology, and the

building of a workforce with the adequate skills to use it effectively. The data collected from SEWA

indicated that taking incentives into consideration during technological learning is important in at

least two ways: in the relationship with vendors and with the staff.

27

One of the most salient themes in interviews was that training was a pre-requisite for successful

technology implementation. In particular, the quality (especially that of the instructor), and the

length of the training were cited as factors facilitating new technology adoption. Whereas the

positive association between the quality of activities and their effectiveness is not surprising, the

importance of timing is less straight-forward.

Staff indicated it was essential to preserve the ability to communicate with technology providers

after the initial period of sale, given that it was impossible to know all relevant questions in advance.

In fact, most of the bottlenecks mentioned by respondents referred precisely to technical assistance

as a proxy for continued training, since it was harder to find technology providers that would

respond timely and satisfactorily after the software or hardware had been purchased.

SEWA managers reported that they deal with incentive-related issues using three main strategies.

One of them is to highlight to vendors the possibility of future purchases based on the quality of

their technical assistance. The second, applicable in the case of hardware, was to prioritize products

with better warranty options. Third, they would establish a separate contract for technical assistance

that guaranteed them greater control over the quality of assistance they get.

All these strategies have produced mixed results so far, as it is not always possible to enforce such

agreements as an NGO without having a large purchasing power. Such structural issues are not

always possible to address directly, but it is important for SEWA to consider in advance, in the

context of technology evaluations, that when choosing to adopt a new technological product or

service, the provision of continued technical assistance should be prioritized.

From the staff perspective, more and better training is cited as a necessary, but it insufficient to

promote technological learning. Rather, they see skill-building and knowledge acquisition as a

complement to additional incentives, and not as activities that are solely desirable based on its

intrinsic value. For example, some of the issues mentioned as motivating factors to employees is

having the ability to understand why that new technology is necessary, and what improvements can

be expected after its implementation. Another example is the association between technological

learning and specific (better) career opportunities. Knowing that learning a new skill can increase the

prospects of a promotion or a salary raise can make a difference between employees seeing a new

technology with excitement, instead of simply “more” work.

28

5. CONCLUSIONS & RECOMMENDATIONS

CITE’s comparative advantage as a university-based initiative is to conduct and share the results of

applied research on topics that are relevant to its partner organizations and society more broadly.

Institutional building, program decision-making, and project implementation are tasks that ultimately

fall beyond CITE’s role, representing instead exclusively a partner’s prerogative. By co-producing a

methodology of technology evaluation, our research intends to generate evidence that advances

knowledge about development problems, thereby representing a basis for more appropriate

solutions.

Despite extensive data collection to-date, it is still possible – and encouraged – to add to the scope

of the analysis produced so far. However, additional efforts must consider the project’s nearing end

line (Summer 2017). With this in mind, we recommend the development of targeted research efforts

in one or more of the following eight topics:

1) Direct Demand (Comprehensive analysis of SEWA membership databases) – Until now,

most of our analysis about the SEWA membership not covered by our sample relied on

information retrieved from the “Union” database. However, there are other sources worth

exploring, such as the records kept through the microfinance, SSKs, RUAAB, UNDP projects.

A quantitative analysis of these datasets can help CITE to create a more detailed demographic

profile of SEWA members, and consequently a more accurate assessment of the existing

demand for technological products and services.

2) Indirect Demand (Research on the demand for SEWA services by non-members) - The

data from the Census (2001 and 2011) and the Indian Human Development Survey (2008 and

2012) could only be partially analyzed so far, despite the wealth of relevant information they

contain. Comparably to the efforts to assess direct demand described above, the creation of a

detailed demographic profile of the non-members can have a similar result. More information

about the people living in the districts where SEWA operates will allow the organization to

identify social demands beyond its existing membership base. Such information can be critical to

evaluate which technological products and services are the most appropriate for each specific

context.

3) Partnerships (Review of SEWA’s experience with external partners) - Over the years,

SEWA Delhi has worked with many external partners supporting its projects, which allowed the

29

organization to learn lessons from each of these experiences based on their performance. Yet,

little is known about how partners viewed these same experiences. Technology evaluations are

inherently a collective effort, and therefore require the leading organization to be knowledgeable

of the kinds of partnerships that have the potential to work, those that don’t, and why. The

original design of our research included a literature review of donor reports, as well as semi-

structured interviews with partners’ liaison officers. Time and logistical constraints prevented

these activities to take place in 2016, but there may be an opportunity in 2017 to carry them out.

4) Organizational Mapping (Study of the Delhi NGO Landscape) – SEWA is one among

many organizations offering similar services and targeting the same populations. In fact, the

most recent registration from the government released in early 2017 has identified approximately

2,000 NGOs operating in Delhi15. An assessment of this database, in addition to interviews with

key informants, can reveal broader patters related to sources of funding, priority areas,

organizational modalities. SEWA can use this information to identify which products and

services are undersupplied in their areas of work. A complementary study along the same lines

can be developed around organizations currently specifically involved with technology

evaluations.

5) Advances in program design and implementation (Study of social policy and practice in

Delhi) – Across India, and even within Delhi, there are several new NGO and government

initiatives emerging every day. Most of SEWA managers we interviewed expressed that

identifying their clients’ demands and delivering quality service was as important as being

updated on the technologies and practices being used by their peers. Even a compact literature

review could provide basic information on some of the new social interventions targeting

women, especially those involving technology.

6) Institutional Analysis (Research on the Delhi institutional context influencing women,

informal workers, and technology) – Identifying the major organizations, regulations, and

policies with direct impact on women, informal workers, and technology can help CITE and

SEWA to further contextualize future evaluations. Part of this work has already been completed

as part of this study, but further research is encouraged because of the sheer number and

complexity of social, economic, and cultural transformations that have taken place in Delhi over

the last two decades. 15 Government of India – NGO-Partnership System. Available at: http://ngo.india.gov.in/

30

7) Research Preparedness and Data Management – Effective evaluations rely on well-

developed information systems. It is through them that the organization can identify

technological demands, support program design, and generate evidence that helps to measure

the impact of technologies they select. One of the key findings from our study revealed that

SEWA collects a large amount of data from their members, but only uses a fraction of it for

program design and monitoring purposes. Taking advantage of the good working relationship

built over the last 15 months of collaborative work, it is possible to use next few months of the

project to help shape SEWA’s next round of data collection exercises across the four programs

studied.

8) Trade-based Technological Ethnographies – The first question SEWA staff asks of

potential members is about their trade. Street vendors, construction workers, artisans, and most

of all other informal workers represented by the organization have complex relationships with

technology. The research until now has prioritized specific SEWA programs as a unit of analysis,

but there is also space to include in-depth accounts about how technology influences members’

work. It has already been discussed among the research team the use of participant observation

methods to identify the direct role of technological change on their professional activities.