Promising Practices Final Report - Unicef

How do you know what’s good for me? An Overview of Promising Practices in Adolescent Programming in Indonesia by UNICEF (and other partners). Final Report

Elayn M. Sammon, Mehjabeen Jagmag, Monica Martinez, Revita Wahyudi

8th November 2017

How do you know what’s good for me? An Overview of Promising Practices in Adolescent Programming in Indonesia by UNICEF (and other partners).

Acknowledgements This report represents a collaborative effort made possible by the contributions of experts in the field of child and adolescent development and wellbeing. Special thanks are extended to the adolescents who shared their views and opinions so openly and honestly, to their parents, guardians and community members as well as the professionals and programme staff who work with them. We are also immensely grateful to the adolescent U-Reporters who responded so enthusiastically to the on-line poll. This review would not have been possible without the collaboration and commitment of the organisations who agreed to participate in the case study reviews. We sincerely acknowledge and extend our thanks to the management and programme staff at UNICEF Indonesia; Sabang City Administration; Micronutrient Initiative (MI), Bandung; Save the Children (StC), Bandung; Perkumpulan Sangaar Sahabat Anak – Bandung Street Children Programme (BSCP), Bandung; Indonesia Planned Parenthood Association (PKBI), Banda Aceh; Pusat Kajian Pendidikan dan Masyarakat – Centre for Education and Community Studies (PKPM), Banda Aceh. Sincere thanks are further extended to the UNICEF partners in government and non-government organisations, including youth networks, who supported the process through their attendance at stakeholder workshops, through the provision of documentary evidence, and by facilitating access to local organisations involved in the data collection for case studies; Youth Network KAMUS, Rumah KitaB, Plan International. RCA Palladium, Ditjen Pemasyarakatan, Childfund Indonesia, Sinergi Muda, Pannafoto, JHUCCP, Nutrition International, PKBI, KPAN, Save the Children, UNFPA, BKKBN, Wahana Visi Indonesia, Fokus Muda, Helen Keller International. We are indebted to the Ministry of Social Affairs, Ministry of Health, Ministry of Education and Culture and Ministry of National Development Planning and their local departments in Sabang and Bandung. The authors acknowledge the contributions of UNICEF Indonesia to the overall review process. Our appreciation is extended to Gunilla Olsson, Representative and Lauren Rumble Deputy Representative for their encouragement and guidance. Hiroyuki Hattori, Chief of Education, Ticiana Garcia-Tapia, Youth and Adolescent Development Specialist, Aidan Cronin Chief of WASH, and Andi Yoga Tama, Chief of Banda Aceh Field Office and their teams made themselves available for discussion, provided comment and inputs and were consistently and reliably supportive throughout the process. UNICEF Indonesia were also particularly helpful during the design and administration of the adolescent on-line poll and sincere thanks are extended to Valerie Crab, Programme Specialist/Innovation Lead and colleagues for sharing their experience and diligently supporting this exercise.

© Oxford Policy Management

ii


Executive Summary Purpose The purpose of the Overview of Promising Practices in Adolescent Programming by UNICEF and other partners, is to examine the strategies that have been identified as most effective in contributing towards improvements in outcomes for adolescents as they move towards adulthood to become “the agents of change for economic growth and torchbearers for sustainable development” (UN General Assembly 2015). Context This focus on 10-19 year olds comes at a time when significant evidence is building to suggest that adolescence provides, “a second window of opportunity to influence developmental trajectories and make up for poor childhood experiences”, (UNICEF 2017a). UNICEF Indonesia report that, “In Indonesia, there are approximately 46 million adolescents, aged 10 to 19, which represent 18 per cent of the total Indonesian population”, (UNICEF undated), and that, “this group represents an important foundation for the nation's long term growth and development … and will be key to Indonesia achieving its global sustainable development goals” (“IndonesianYouth – a foundation for the future,” 2017). UNICEF and other stakeholders’ adolescent programming is designed against this background of inequality, high need and limited resources. Method To inform UNICEF’s future programme and policy strategy, we designed a framework for planning and measuring promising practice in programming for adolescent wellbeing as a robust and lowcost qualitative methodology to help establish the credibility of a programme. This participatory methodology consists of 11 evidence-based criteria which were derived from a review of literature related to promising practice in human development and with technical inputs from key stakeholders in Indonesia. The criteria were refined to describe what works to improve the wellbeing of adolescents and to determine if a specific programme is scalable or replicable in each context. They were then cross-referenced with the five domains of the UNICEF adolescent development framework – health and wellbeing, education and learning, protection, transition to work and participation and engagement. The framework was field-tested using a case study approach in two locations, one urban located close to the capital Jakarta, and one remote rural in Western Indonesia. A sixth light-touch case study was conducted with Papua programme staff (including filed staff) in Jakarta. It was complemented using a virtual platform to engage adolescents through a poll developed by youth networks. Findings We identified 19 programmes for review in the preliminary screening. Of these, three programmes met the criteria for in-depth assessment against the promising practice criteria established in this review. Finally, only one programme met the criteria for a promising practice – a government school grant programme for children in Sabang Island, in Aceh Province. One UNICEF-supported programme met the criteria for an emerging practice, the Menstrual Hygiene Management programme in Bandung, West Java and one UNICEF-supported programme, was considered undetermined practice, the Support for closing the HIV prevention and treatments gaps for Young Key Populations in Indonesia (LOLIPOP) programme in Bandung West Java. A further six programmes were subject to a light-touch review; two of these are considered emerging practice and four are considered undetermined practice. Several programmes did not meet the criteria because of insufficient documentation. This points to gaps in planning and monitoring by the development partners, which are urgently needed to influence scale up and sustainability.


iii


Implications for policy and practice In a middle-income context, where development partners like UNICEF are practicing an upstream approach with limited resources, improvements to development planning and monitoring are needed. Our review finds that investments are most impactful when designed together with young people, have a clear theory of change and are designed to scale. Results also suggest that UNICEF and its partners concentrate investments in proven interventions which impact on overall adolescent wellbeing. This approach should also be applied to a rigorous 'weeding-out' of programs which do not have a clearly articulated results chain, with measurable indicators and targets. Further research is needed to critically review selected adolescent-focused interventions within the broader context of UNICEF's program in Indonesia and the evolving Sustainable Development Goal Agenda 2030 (SDGs). The adolescent programming stream may benefit from a dedicated strategy to address critical issues affecting adolescent wellbeing and opportunity (such as school drop-out, HIV and AIDS, and violence, amongst others). It is suggested that UNICEF consider a formative evaluation of the combined cross-sectoral support provided to Indonesia through its programme strategy 2016-2020. This can take account of the newest evidence baseline for the SDGs as well as recent emerging evidence on HIV and access to quality education etc. and can consider how to improve coherence and integration of adolescence and other crosscutting themes across programme sectors. This formative evaluation should place an emphasis on results-based management and adaptive design with concern for outcomes, impact and sustainability. Alternatively, a series of targeted evaluations of key pilot inititatives showing improvements in adolescent well-being and generating lessons learned for evidence-based policy is recommended. In summary, •

Generate more and better independent evidence to support policy advocacy

•

Shift the focus from quantity and coverage to quality and outcomes/impact

•

Conduct internal critical review of programmes for evidence of outcomes and impact and value for money

•

Introduce an explicit adolescent strategy

•

Pay attention to local initiatives and build on success

•

Reflect on UNICEFs comparative advantage in the context of high inequality, high need and limited resources – conduct a formative evaluation of the combined cross-sectoral support provided to Indonesia.


iv


Table of Contents Acknowledgements

ii

Executive Summary

iii

List of Tables and Figures

vii

List of Abbreviations

viii

1

Introduction 1.1 Purpose 1.2 Definition of Adolescence 1.3 Definition of Promising Practice 1.4 Background to the Review 1.5 The Current Situation for Adolescents in Indonesia 1.6 Adolescent Perceptions of Priority Concerns

1 1 1 2 3 4 7

2

Methodology 2.1 The Promising Practices Framework 2.2 Key Research Questions 2.3 Respondents 2.4 Ethics 2.5 Research Guide 2.6 Training for Field Researchers 2.7 Pretesting 2.8 Data Collection 2.9 Data Management 2.10 Data Analysis 2.11 Quality Assurance

10 10 14 17 18 18 19 19 20 20 20 20

3

Results 3.1 Overview 3.2 Three In-depth Case Studies Sabang Education Grant Menstrual Hygiene Management Linkages of Quality Care for Young Key Populations (LOLIPOP) 3.3 Six Light-touch Case Studies Anaemia prevention in school-going adolescent girls through IFA supplementation Skills to Success The Bandung Street Children Programme Youth Programme. Indonesia Planned Parenhood Association Strengthening Child Protection (Restorative Justice) Life-skills Education 3.4 Promising Practice Methodological Guide

22 26 27 28 31 34 38 38 41 43 46 48 50 52

4

Summary Conclusions

54

5

Annexes

57

References / Bibliography

58

Annex A Detailed Case Studies A.1 Sabang Education Grant In-depth Case Study

60 60


v


A.2 A.3 A.4 A.5 A.6 A.7

MHM In-depth Case Study 72 LOLIPOP In-depth Case Study 81 IFA Light-touch Case Study 92 Skills to Succeed Light-touch Case Study 99 Bandung Street Children Programme (BSCP) Light-touch Case Study 105 Youth Program. Indonesia Planned Parenthood Association (PKBI) Light-touch Case Study 110 A.8 Community Youth Justice Light-touch Case Study 115 A.9 Life Skills Papua Light-touch Case Study 119 Annex B

“How do you know what’s good for me?; “A guide for planning and measuring promising practices in programming for adolescent wellbeing” 123


vi


List of Tables and Figures No table of figures entries found. Table 1: The Promising Practice Framework ................................................................................. 11 Table 2: Number of interviews and FGDs per in-depth study ........................................................ 17 Table 3: Respondents for in-depth case studies ........................................................................... 17 Table 4: Training Agenda .............................................................................................................. 19 Box 1 Three definitions of Promising Practices ............................................................................... 2 Box 2 UNICEF Adolescent Country Tracker.................................................................................... 4 Box 3 U-Report Poll Questions ....................................................................................................... 8 Box 4. U-Report Respondents - Methods for Problem Resolution ................................................... 9 Box 5 Overview of in-dpeth and light-touch case studies .............................................................. 22 Box 6. Locations of Promising Practice Case Studies .................................................................. 27


vii


List of Abbreviations BSCP

Bandung Street Children Programme

CDC

Centers for Disease Control

FGD

Focus Group Discussion

IFA

Iron Folic Acid

KII

Key Informant Interviews

LOLIPOP

Support for closing the HIV prevention and treatments gaps for Young Key Populations in Indonesia

MENARO

[UNICEF] Middle East and North Africa Regional Office

MHM

Menstrual Hygiene Management

OPM

Oxford Policy Management

PKBI

Indonesia Planned Parenthood Federation

PKPM

Pusat Kajian Pendidikan dan Masyarakat (Centre for Education and Community Studies)

PSSA

Perkumpulan Sanggar Sahabat Anak (also BSCP)

SDGs

Sustainable Development Goals

SEG

Sebang Education Grant

StC

Save the Children

STS

Skills to Success

SRH

Sexual and Reproductive Health

UNICEF

United Nations Children’s Fund


viii


1

Introduction

1.1

Purpose

The purpose of the Overview Promising Practices in Adolescent Programming, is to examine the strategies that have been identified as most effective in contributing towards improvements in outcomes for adolescents as they move towards adulthood to become “the agents of change for economic growth and torchbearers for sustainable development” (UNICEF 2011). The review process was conducted in three phases, Phase One Pre-inception Document Review (finalised 21st July 2017), Phase Two Inception Stakeholder workshop for a) validation of promising practice criteria and b) identification of programmes for case study review, Key informant interviews, Inception Report (finalised 16th August 2017) Phase Three Data collection Field work for case study review, Data analysis and identification of promising practice. During Phase Three and in agreement with UNICEF, an additional deliverable was developed, “How do you know what’s good for me? A guide for planning and measuring promising practices in programming for adolescent wellbeing”, describing how to apply the methodology. This Final Report relates to Phase Three and to the consolidation of the overall assignment. It is supplementary to the Document Review and Inception Report. It includes the, • • •

outcomes of fieldwork for case study review – in-depth case studies of three programmes and light-touch case studies of six additional programmes, promising practices conclusions and recommendations, the detailed methodology for the data collection phase, including tools and instruments for data collection.

1.2

Definition of Adolescence

This review applied the UN definition of adolescence, namely the period from 10 to 19 years of age, acknowledging that characteristics of this stage may extend up to age 24, and that adolescent wellbeing is also determined by early child development before age 10 (UNICEF 2011). There is also an acknowledgement that the 10-14-year-old age range is of critical importance because this is the onset of physical, emotional, cognitive and social changes - a rapid and rich learning period and associated risk factors (Ibid.). The broader review of adolescent programmes notes that programmes for young people in the age group between 15 and 24 years of age whilst not adolescent-specific, nonetheless include adolescents and can be considered for inclusion. The terminology of youth and adolescence in Bahasa Indonesian language is given some consideration by Ramadhan (2013), who notes that both “Anak muda” and “Pemuda” have been used to describe youth with the former seen as more useful to describe the active engagement as © Oxford Policy Management Ltd, November 2017

1


equal partners. Many programmes also use the term "remaja". It refers to both adolescents and youth. UNICEF report that, “remaja is not only related to age group, but is also linked with social status. In many area, once adolescents get married, they are no longer considered adolescents anymore - they are considered as adults”, (written communication, October 2017). Equally the age group suggested by ‘youth’ can extend up to 30. Ramadhan (2013) further suggests that the term ‘adolescent’ has been used to position young people as not civically engaged, or perceived solely as an object, with an image attached to consumerism. The implication is that adolescents are considered as targets for acquisition of goods and services in ever increasing amounts rather than as citizens who can actively participate in community life. The author relates this to a perceived apathy amongst adolescents to political activism1 (Ibid.).

1.3

Definition of Promising Practice

For this review a promising practice is defined as, “A programme or intervention which meets a specific set of criteria, which describes what works to improve the lives of individuals and which is sustainable or replicable in a specific context”. This definition is derived from an extensive literature review and is based on the combined experience of the Promising Practices Network, the Centers for Disease Control (CDC) and UNICEF MENARO 2015, adapted for the context of this review, (Box 1). Box 1 Three definitions of Promising Practices Promising practice belongs to a continuum of practices that “represents the ongoing application of knowledge about what is working to improve desired outcomes in each context”. (Spencer and others 2013) Promising practice is “evidence-based, [and demonstrates] equity, values orientation, innovativeness and youth involvement”, but where, “no evaluation of outcomes has been conducted and thus there is no evidence of effectiveness”. (UNICEF MENARO 2015) Promising practice is, “evidence-based information about what works to improve the lives of children, youth, and families” Promising Practices Network http://www.promisingpractices.net (archived)

This promising practice review includes evaluative elements but differs from an evaluation in that it asks not only if a project or programme achieved or is on track to achieve its intended outcome and/or impact, but also asks if it is sustainable and replicable within a given context. Programmes which produce results for adolescents are considered a promising practice only if there are enough available resources to allow the intervention to be sustainable and replicable. To make a distinction between those practices which are definitively a promising practice and those which with additional forthcoming evidence or with a course correct could in the future be considered a promising practice, programmes are considered on a continuum, from undetermined practice to emerging practice and then promising practice. This allows for variable results to be reported.

1

Ramadhan (2013) recommends further reading on the subject, Nugroho Yanuar & Tumenggung, Adeline M. Marooned in The Junction: Indonesian Youth Participation in Politics. Part of Go! Young Progressives in Southeast Asia. Friedrich Ebert Stifftung

© Oxford Policy Management Ltd, November 2017

2


1.4

Background to the Review

Indonesia’s growing adolescent population is one of the largest in the world, a population which also holds the key for Indonesia’s economic growth, if supported and encouraged by policies and services that are responsive to their needs and capabilities (Economist Intelligence Unit 2016 a). Programming for adolescent wellbeing is therefore a critical intervention area for UNICEF. However, there is not enough robust evidence on what works and what does not. The lack of documented evidence makes it particularly difficult to identify interventions that are effective in addressing the risks facing many adolescents. Applying an evidence-based methodology is a useful mechanism to understand if and how programming for adolescent wellbeing works. To inform UNICEF’s future programme and policy strategy, we designed a framework for planning and measuring promising practice in programming for adolescent wellbeing as a robust and lowcost qualitative methodology to help establish the credibility of a programme. This participatory methodology consists of 11 evidence-based criteria which were derived from a review of literature related to promising practice in human development and with technical inputs from key stakeholders in Indonesia. The criteria were refined to describe what works to improve the wellbeing of adolescents and to determine if a specific programme is scalable or replicable in each context. They were then cross-referenced with the five domains of the UNICEF adolescent development framework – health and wellbeing, education and learning, protection, transition to work and participation and engagement, Box 1. The framework was field-tested using a case study approach in two locations, one urban located close to the capital Jakarta, and one remote rural in Western Indonesia. A sixth light-touch case study for a programme in Papua was conducted via discussions with programme staff (including field staff) in Jakarta. It was complemented using a virtual platform to engage adolescents through an on-line poll developed by youth networks. This review is a first attempt to collect and systhesise available data on adolescent programming in Indonesia. The methodology proved suitable for determining useful strategies and interventions which can contribute to effective targeting of limited resources in contexts of inequality and huge need. Further, it supports identification challenges and obstacles that programmes need to address to be considered promising practice and provides guidance for course correction. Although this framework was developed and tested in Indonesia to review adolescent programmes, the principles of the framework are considered universal and can be adapted for use in any programme context to assess promising practice. The detailed methodology is provided in Section Two of this report.


3


Box 2 UNICEF Adolescent Country Tracker

1.5

The Current Situation for Adolescents in Indonesia

UNICEF Indonesia report that, “In Indonesia, there are approximately 46 million adolescents, aged 10 to 19, which represent 18 per cent of the total Indonesian population”, (UNICEF undated), and that, “this group represents an important foundation for the nation's long term growth and development … and will be key to Indonesia achieving its global sustainable development goals” (“IndonesianYouth – a foundation for the future,” 2017). Recent reports commissioned by UNICEF Indonesia predict six major trends which will impact on Indonesia’s achievement of the Sustainable Development Goals (Economist Intelligence Unit 2016 a; Economist Intelligence Unit 2016 b; Indonesia Ministry of National Development Planning and the United Nations Children’s Fund 2017), • • • • • •

Growing economic inequality Limited access to quality healthcare and education Increasing youth unemployment Rise in non-communicable diseases Threat to health posed by pollution Increasing vulnerability to climate change.

Reports point to a growing “inequality dynamic” in Indonesia which results in a high rate of out-ofschool adolescents and which will significantly impact on the potential to achieve SDG 8 on decent work and economic growth. The link between adolescent development, education and employment, will be critical to future programming design to assure the potential for human development (Economist Intelligence Unit 2016 a). At the same time, HIV infections amongst adolescents in Indonesia are increasing, in part due to limited access to HIV testing among adolescents compared to adults. Ministry of Health Indonesia and UNICEF (2016) report that, “Between 2011 and 2015, new reported HIV infections in Indonesia increased annually by 13.1 per cent among adolescents (aged 15 to 19)”. Noting that adolescence is also a critical period for interventions to prevent malnutrition, and that, “in Indonesia, many girls get married and give birth while their own bodies are still growing and © Oxford Policy Management Ltd, November 2017

4


maturing, causing increased nutritional demands and competition for nutrients between the foetus and it’s still developing young mother”, GAIN (2014) reports that early intervention is required to ensure pre-conception health and wellbeing of girls to break the cycle of intergenerational transmission of undernutrition and its affects. WHO (2015) report that in Central Java, East Java and Bali more than 77 per cent of adolescents have inadequate knowledge, in terms of adolescent reproductive development. However, access to sexual and reproductive health services for adolescents remains a challenge, in part because of conflicting values and norms around sexuality (Indonesia Planned Parenthood Association 2015). Although adolescents are sexually active before they are married, unmarried young people [adolescents] are frequently denied SRH services because of specific laws and regulations and dominant cultural and religious norms (Ibid.). Youth Friendly Sexual and Reproductive Health Services are being developed in some areas through partnership with public health centres (Puskesmas), however challenges remain including, “human resources, methods, funding and supporting facilities and infrastructures” (Ibid.). Data on adolescent experience of violence and child marriage suggests that 12 per cent of women aged 20-25 were married before their 18th birthday and that child marriage rates are disproportionately high for girls living in poor families and in rural compared to urban areas (Indonesia Ministry of National Development Planning and the United Nations Children’s Fund 2017). Girls who marry are six time less likely to complete senior secondary school. Recent research notes that 55 per cent of female subjects involved in the process of marriage dispensation (whereby religious and civil courts can be petitioned to lower the age of marriage), were aged 14-15 years whilst for boys more than three-quarters were aged 17-18 (UNICEF and 18+ Coalition un-dated). The WHO global school-based student health survey (2015) indicates that 20 per cent of students aged 13-15 report experiencing bullying at school. In the same age group, almost 25 percent report having been in a physical fight in the previous 12 months and 30 per cent say they were seriously injured on more than one occasion in the previous 12 months. Whilst disaggregated rates indicate that reports are higher for males than females, they are significantly high in both sexes to cause alarm. The Indonesian policy and legislative environment is structurally robust although gaps in implementation and enforcement remain. A recent Government of Indonesia report launched at the High Level Political forum on SDGs in July 20172 with baseline data for the SDGs in Indonesia recommends that for useful implementation to occur national action plans and draft national actions plans should be fully costed, with commitments in the national budget and with clear monitoring strategies (Indonesia Ministry of National Development Planning and the United Nations Children’s Fund 2017). This review identified emerging practice in adolescent programming which can be considered for further review as a promising practice (Sammon, Martinez, Jagmag and Wahyudi 2017 a and 2017 b). The programmes were considered under the five domains of the UNICEF Adolescent Development Framework. Whilst the domains provided a structural framework for the review it is understood that many programmes are crosscutting and can be considered as having contributed to improved outcomes across more than one domain. For example, WASH in Schools can relate to protection – prevention of bullying, health – reduced infections, and education – girls stay in school. For ease of reading and analysis, a ‘primary’ domain has been selected for the programme intervention. Health and wellbeing: programmes identified through the document review include those related to nutrition, HIV and AIDS and access to sexual and reproductive health. The adolescent friendly health services (PKPR) are provided by government through government mandated community health clinics (Puskesmas) throughout Indonesia. These tend to focus on adolescent maternal and child health, although Agustina and Prafiantini (2016 a) noted the continuing prevalence of 2

https://sustainabledevelopment.un.org/hlpf © Oxford Policy Management Ltd, November 2017

5


anaemia amongst adolescent girls in some districts, as a severe public health problem. Through the Child Friendly City/District initiative these PKPR should also offer sexual and reproductive health services (SRH) however the implementation of this initiative does not explicitly address access issues (Carvalho and Koteng 2014). A 2010-2014 HIV and Life-skills programme in Papua, reported that 91 per cent of schools have provided HIV education to students although the programme did not manage to produce significant and sustained changes in the lives of young people and the HIV epidemic among the targeted population group remained unchanged (UNICEF 2015 a). A current HIV related programme with young key populations in Bandung, integrated within the broader Education Sector System Strengthening programme, is emerging as an interesting model, although there is currently insufficient information on impact. As noted above, access to SRH services remains challenging, constrained by prevailing cultural, religious and social values. Education and learning: programmes identified through the document review include those related to access to secondary education and to WASH3 in Schools. The One Roof School Programme, (SATAP) combines primary and junior secondary education in one compound, aiming to increase access for students in remote rural areas. International Labour Organisation in 2011 and Pillay and Haribowo (2013) report inconclusive results in understanding if the model supports secondary education retention. Reports of support for strengthening the national curriculum by UNESCO and for a UNICEF supported Community Based Development Information System to track out of school children have not been verified by documentation. The WASH in Schools programme has included cross-sectoral approaches to menstrual hygiene management to improve girls’ health, to increase girls school attendance, and to address bullying and harassment of girls during menstruation. The programme design is embedded in formative research and initial reported results demonstrate positive outcomes. Protection: UNICEF Indonesia reports that they are involved in adolescent programming to i) reduce school bullying and to introduce positive discipline in schools, ii) around child marriage, and iii) in the juvenile justice sector, to support diversion and reintegration of children from prison. The pilot programme to prevent peer violence and bullying in Junior High Schools, has been identified as a promising practice internationally however implementation in Indonesia is in the earliest stages and the documentation on design and intended results is limited. Transition to work: The smooth transition from education to employment will be a key predictor of Indonesia’s future economic prosperity (Economist Intelligence Unit 2016 a.). Ensuring that ‘education’ is fit for purpose will require an examination of the intersection between future employers needs and current education sector goals, so that adolescents have a realistic expectation of future employment. UNICEF does not report that it is supporting any current programmes in the transition to work domain. During the field work for the case study reviews associated with this review, a Save the Children programme for adolescent employment, Skills to Success, was identified in Bandung and included as light-touch case study. Participation and engagement: This document review identified a range of tools being applied to encourage participation and engagement however the integration with programmatic interventions which had clear intended outcomes for adolescent development was less evident. Reality Check Approach (RCA), Human Centred Design (HDC), U-Report, and community digital storytelling are useful participative tools for gathering evidence and data on adolescents’ perceptions. The Adolescent Kit for Expression and Innovation provides a support to interventions using creative methods to achieve outcomes for adolescent development. These tools were suggested by programmers as promising practices in adolescent programming. Their value is not as standalone interventions but rather as tools to support participative achievement of programme outcomes. Their use is referenced where they have been applied in programmes considered within the scope of this review.

3

water, sanitation and hygiene WASH © Oxford Policy Management Ltd, November 2017

6


Key points •

• •

•

The context for adolescent programming in Indonesia is heterogeneous and complex. Inequalities persist and the underlying trends are predicted to continue. This will likely exacerbate the poor outlook for youth employment which may contribute to rising tensions. The links between education and consequent employment for economic growth stand-out as critical contextual factors for adolescent programming. This includes both ensuring that adolescents stay in school or other educational systems and that the quality of the learning programmes offered meets their needs. UNICEF and other stakeholders’ adolescent programming is designed against this background of inequality, high need and limited resources. Adolescent programming is a critical intervention area for UNICEF Indonesia. Significant key interventions are underway in programmatic silos - the links/integration across programme areas is not well-defined through a strategic plan for adolescent programming which details intended outcomes. Much of the available foundational evidence presented in support of specific adolescent programmes is anecdotal rather than evidence-based, which is a critical criterion for measurement of best practice. Although a significant number of documents were provided, these did not provide corroboration of the programme interventions subsequently described during key informant interviews.

1.6

Adolescent Perceptions of Priority Concerns

Whilst adolescents were included in focus group discussions as part of the case study data collection methodology, as an additional data collection method, and to solicit the views of as many adolescents as possible, a poll was administered using the U-Report4 digital platform. This expands the scope of the enquiry and assures participation and engagement with as wide a cohort of adolescents as possible. UNICEF launched U-Report in Indonesia in 2015 as an, “innovative new platform that gives young people the chance to speak up on issues that affect their lives”. This Twitter, Facebook and SMS based polling mechanism enables young people aged 14-24 to share their opinions on topics ranging from education to violence to health to governance. For the purposes of this review, the UReport targeted adolescents aged 14-19 years. The responses are analysed and shared with key partners, including government to contribute to policy debate, programme design and in this case to ensure adolescents are involved in research. UNICEF Indonesia advises that, “the goal of U-Report is youth participation and the data measures perception; it is an indication of youth experience. Since this is anonymous data and we do not control who signs up or answers the polls we cannot claim that it is objective or representative data.” (Written communication, October 2017). The inclusion of a U-Report poll and the involvement of youth in the design of the U-report methodology explicitly for this assignment, ensures the continued involvement of adolescents in the review. With the support of UNICEF Indonesia, a participatory workshop with youth representatives to design a U-Report poll of adolescents, was held 4th September 2017. They identified three follow-on questions which they felt best served the data collection needs for this review, Box 3. More than 2,360 adolescents responded to the poll, slightly more girls (53 per cent) than boys (47 per cent).

4

https://www.facebook.com/UReportindonesia/ and https://indonesia.ureport.in and https://twitter.com/ureport_id?lang=en © Oxford Policy Management Ltd, November 2017

7


Box 3 U-Report Poll Questions

Source: UNICEF Indonesia September 2017

Of these, • • • •

30% said their biggest problem is bullying (F=27% M=32%) 21% each said education and other - unspecified (F=22% M=20%) 17% said popularity and identity (F=18% M=15%), and 12% access to employment (F=10% M=14%).

Respondents said that they would aim to solve these problems by, • • • •

57 % joining a community activity (F=52% M=63%) 24% talking with a friend or teacher (F=26% M=21%) 5% spending time on social media because it’s easy to access (F=5% M=4%) 14% prayer, meditation, ‘letting it go’, or other (F=17% M=12%)

Adolescents asked to define why they chose specific method for problem resolution said they joined community activity because they found it useful, spent time on social media because it is easy to access, and talked to others because they feel listened to, Box 4.


8


Box 4 U-Report Respondents - Methods for Problem Resolution

Source: UNICEF Indonesia September 2017

Since each of the three in-depth case studies have been found to have a protective element, specifically around bullying prevention and incidence reduction, and all provide a community space for discussion and resolution of these problems, this suggests that current programme design is to some extent correctly targeting the needs of adolescents.


9


Methodology

2

The key objective of the primary data collection was to present evidence on where selected programmes sit on the promising practice continuum based on their design, planning and practice. The review team conducted three in-depth case studies of programmes for adolescent wellbeing and six light-touch case studies to consider the factors that enable successful programme delivery. The three in-depth case studies involved primary data collection from key stakeholders – adolescent beneficiaries, programme planners and implementers, parents, community members, and representatives of civil society, NGO and government institutions to provide their perceptions of whether the programme meets objective indicators of promising practice. This was analysed together with a review of programme documents. The six light-touch studies were based on key informant interviews and focus group discussions with programme implementors and, where these were made available, a review of programme documents. For the in-depth case study, the review team has employed an instrumental case study design. Cases were selected purposively based on the promising practice matrix. As noted by Grandy (2010), “An instrumental case study is the study of a case (e.g., person, specific group, occupation, department, organization) to provide insight into an issue, redraw generalizations, or build theory. In instrumental case research the case facilitates understanding of something else.” Given the contextualised nature of each case study it is not possible to compare results across interventions. To explain further, these case studies are not statistically generalisable, nor are they conducted to review the complexity of the case but to shed more light on the specific case itself. Information from these cases answer broader questions in the matrix – and allow for transferability of findings to inform future programming.

2.1

The Promising Practices Framework

The 11 criteria for review of a programme as a promising practice for adolescent wellbeing are described in Table 1. They include those that, •

have a solid evidence-based design,  

•

have developed a theory of change on how outcomes will be achieved,  

•

have sufficient and necessary documentation to be evaluated,  

•

are considered by stakeholders as appropriate for adolescents,  

•

have a monitoring and evaluation system in place,  

•

address cultural and gender specific issues,  

•

have developed mechanisms to identify and reach the most vulnerable populations  

•

have generated evidence of positive impact for beneficiaries,  

• • •

have the potential for replication and scaling up, can be seen to encourage participation of and engagement with adolescents, and demonstrate flexible and adaptive approaches.


10


Table 1: The Promising Practice Framework Revised Criteria Promising Practice in Adolescent Programming - Analysis Matrix UNICEF Adolescent Country Tracker Domains

No.

Promising Practice Criteria. (*indicates essential criterion)

1*

Evidence-based programme design

2

A theory of change

3

Documentation

4

Accepted practice

5*

Monitoring and evaluation

Area of Inquiry

1

2

3

4

5

Education Participation Protecti Transition and and on to Work learning engagement Source: programme documents; KII; FGD; other

Health and well-being

How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? How do stakeholders demostrate general acceptance of the programme as appropriate for use with children? Is there an effective monitoring and evaluation plan and evidence of it's execution?


11


6

Cultural competency and partnerships

7

Reaches most vulnerable and marginalized

8

Evidence of positive outcomes and/or impact

9*

Sustainable and replicable

10*

Involved and empowered youth

11

Innovative

How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? How does the programme identify and reach the poorest adolescents, adolescent girls, adolescents with disabilities, adolescents affected by HIV, adolescents affected by violence, other marginalized populations? Has the programme been subject to an external independent study demonstrating positive outcomes? Is the evaluation available for review? How do programme participants perceive the benefits of the programme? How did the progragramme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures How did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? How was the programme design flexible when required to


12


change and adapt to new information and challenges?

To make a distinction between those practices which are definitively a promising practice and those which with additional forthcoming evidence or with a course correct could in the future be considered a promising practice, a ranking is provided to describe the continuum from undetermined practice to emerging practice to promising practice. Programmes are considered on this continuum towards promising practice, which allows for variable results to be reported.

Promising Practice – meets most of the criteria for promising practice Emerging Practice – meets many of the criteria but there is insufficient information to determine if it currently meets most of the essential criteria Undetermined Practice – does not yet meet promising or emerging practice criteria or there is insufficient information to assess if it will meet promising or emerging practice criteria.

The case study review applies a score of 1 for “YES” meets criterion, and 0 for “NO” does not meet criterion. We suggested and used a dichotomous indicator (rather than, for example, a Likert scale) for these reasons, 1. The scale is used as a quick tool to score programmes. In order arrive at a definitive answer in a limited period with limited resources, a yes/no answer allows researchers to look at the evidence and respond quickly to the evidence they see, 2. The yes and no answer forces the researcher to provide an explanation for the scoring, as is provided in the current report. This allows for stakeholders participating in the programme to provide and triangulate evidence to these scores, and for the reader to understand the researchers’ logical reasons behind the scores provided.

For this review of programming for adolescent wellbeing four essential criteria to be considered for review as a promising practice were selected and validated by stakeholders during an initial workshop in Jakarta in July 2017. • • • •

# 1 Evidence-based programme design # 5 Monitoring and evaluation (M & E) # 9 Sustainable and replicable # 10 Empowered and involved adolescents.

These four essential criteria mostly concern programme design and process rather than actual results themselves. Which means even without evidence of tangible outcomes, some programmes could be considered “promising practice.” Promising practices are understood as the mid-point on a continuum from emerging practice to good and best practice. They are defined both positively against emerging practice– as making progress or demonstrating greater potential than a practice that is undocumented or does not represent good practice; as well as negatively – against good or best practices that meet all or most of the criteria of ideal practice. Most importantly in comparison with good/best practice a promising practice describes a programme for which robust evidence of programmatic effectiveness its ability to be sustained and/or replicated is emerging or not available. Evidence of © Oxford Policy Management Ltd, November 2017

13


positive outcomes or impact would be desirable to though not necessary for a promising practice. In addition, almost all promising practices considered for this report either did not have an impact evaluation, or had not reached the endline phase of their evaluations, and would not be considered for review under this category. The criteria in the promising practice matrix serve two purposes, 1. Preliminary review of programmes: To select programmes for review or further research, the promising practice matrix allows researchers and programme implementers easily score and sort through programmes based on literature review or preliminary findings. For this, it is essential to fix essential criteria and a benchmark to select programmes. In this review, four essential criteria (see above) were set for a programme to be considered for review and a benchmark was set of roughly 50 per cent or six criteria to qualify for in-depth review. 2. In-depth study and determination of promising practice: Once programmes have been selected, the promising practice criteria forms the basis of collecting and analysing data in depth. For this, it is essential to populate the matrix with detailed findings across all the criteria to determine the practice. It is possible, as in the case of this review that triangulating data with stakeholders and subjecting the practice to a thorough review may lead to a reduction in the final score. There were cases where the final score increased following the in-depth analysis, for example, the Sabang Education Grants score increased from nine to 10. There were also lighttouch studies where scores changed - The Indonesian Planned Parenthood Association increased from 0 to 5 once documentary evidence was provided for scoring. The rest of the scores have fallen from the first review.

2.2 1

Key Research Questions Evidence-based programme design

How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? • • • • 2

How does the program meet international commitments? Was the programme designed based on a researched/ documented need? How was it designed to address this need? Did the programme have a pilot to test the design before scaling up? On what way evidence from the pilot inform project design? Is this document publicly available? A theory of change

How does the programme specify intended outcomes and describe the activities that are related to those outcomes? • • • 3

Did the programme have a well-defined theory of change or other document method to plan outomes, outputsand activities and measure change in the program? Is thisdocument publicly available? How was the theory of change incorporated into programme design, implementation and evaluation? Documentation

Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it?


14


• • • • 4

Does the programme have an implementation plan? Is it available to the public? Is it clear, detailed and easy for non-programmatic staff to read?" Was the plan followed? To what degree? Was implementation and deviation in implementation documented? Is this documentation available? Accepted practice

How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? • • • 5

Does the programme have buy in from the government? How did it foster acceptance? Does the programme have buy in from the community and parents? How did it foster acceptance? Does the programme have buy in from civil society and other organisations working with adolescents? How did it foster acceptance? Monitoring and evaluation

Is there a monitoring and evaluation plan and evidence of its execution? • • • 6

Was M&E planned? To what degree was it executed? Did the programme use the M&E data to change and improve their strategy? How did they do so? Does the programme have an external and independent assessment (formative, summative, impact and/or process evaluation?) Cultural competency and partnerships

How does the programme consider the specific requirements of and involve adolescent boys and girls, of adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? • • • • • • • • • 7

How did the programme create an environment where adoelscents could participate? Did the program identify specific requirements of adolescentsh who might have different needs? How does the program include these requirements? How does the program ensure inclusive involvement of adolescents? Was the programme designed to involve the community? Does the programme work with the community, and build strengths where required? How does it do so? Where adequate skills were not present in the community, did the programme invest in capacity building, how did it do so? How was the community involved in practice? How did the programme create an environment where adolescents could participate? Reaches most vulnerable and marginalized

How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? • •

Was the programme designed to involve the most vulnerable and marginalised? How did the programme identify and involve the most vulnerable and marginalised? © Oxford Policy Management Ltd, November 2017

15


• • 8

How were they involved in practice? How did the programme create an environment where adolescents could participate? Evidence of positive outcomes and/or impact

Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? • • • • • • • 9

What are the programme's desired outcomes? To what degree has programme practice achieved it desired outcomes? To what degree have these outcomes been documented and verified? What is the level of confidence in / robustness of the data? Is the assessment done independently? Are evaluation reports available publicly? Are there any adverse outcomes of the programme? Sustainable and replicable

Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? 9.1 • • • • • 9.2 • • • 10

Sustainability Did the programme integrate with existing government or non-government or private institutional structures? How did it do so? How is the programme designed to integrate with networks and key stakeholders in the community? What measures did the programme undertake to ensure sustainability? Have these measures been adequate? What level of resources are required for the programme to be sustained? Are these resources available when the programme withdraws support? What are the long-term effects (observable after the programme ends) achieved by the programme? Replicability How have programmatic practice been replicated? How have programmatic practice been adapted? Has the programme proved to be effective in different contexts (geography, society, etc.?) Involved and empowered adolescents

Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? • • • • 11

Was the programme designed to involve adolescents? Where adolescents required additional skills to participate, did the program invest in capacity building, and how did it do so? How were adolescents involved in practice? How did the programme create an environment where adolescents could participate? Innovative


16


Was the programme’s design flexible when required to change and adapt to new information and challenges? • • •

How did the programme recognise and solve challenges? Did the programme use the M&E data to change and improve their strategy? How did they do so? In what way did the programme demonstrate flexibility?

2.3

Respondents

Respondents for the in-depth case studies were identified in a two-step process. UNICEF programme staff provided the review team with contacts details of implementing partners on the ground. These partners, including government, were contacted prior to the fieldwork. They were then requested to identify adolescents, parents and community members that the programme team could speak with. In Sabang, the UNICEF office facilitated the interviews for the review team. Table 2: Number of interviews and FGDs per in-depth study Level

Interviews

Focus Group Discussion

Provincial level District level Programme lead Programme team Adolescents Teacher / Health worker Parents Community level

1-2 1-2 1 N/A N/A 1 1-2 1-2

N/A N/A N/A 1 2 N/A

Total

6-9

4-5

1-2

Table 3: Respondents for in-depth case studies Level

List of respondents Programme director, chief of programme at national and/or provincial level involved in planning

Programme staff

Programme implementers at provincial and village level Programme M&E staff (if available) Programme volunteers Government representative district level

Government staff

Adolescent beneficiaries (and non-beneficiaries)

Any other government staff involved in planning, implementation or scaleup of the programme Respondents will be selected based on the programme, but will include boys and girls, vulnerable youth, such as adolescents who are not in educational institutions, respondents with disabilities. The team will especially attempt to interview adolescents who may not have participated in the intervention, to ensure a balanced review

Community members

Community leaders including religious leaders and community elders


17


Community health workers, teachers Other community members or volunteers that may have participated in the programme’s planning and implementation Parents and guardians

2.4

Parents and guardians of adolescents involved in the programme

Ethics

Ethical Review The primary data collection methodology was subject to a rigorous ethical review. The research study, protocol, interview and discussion guides as well as consent and assent forms were submitted to an ethical review committee for their approval before the fieldwork began. The review committee suggested minor revisions to the consent form and requested a justification for interviewing adolescents as young as 14-15. Upon submission of the justification and revisions, the final approval was granted. Research Training Researchers were trained, reflected on and practiced research based on the ethical guidelines developed by the review team, and included in the research protocol. This included guidance to ensure that expectations are not raised, confidentiality is maintained, and respondents are not forced to participate or encouraged to speak about subjects that may be difficult or traumatising for them. Reseachers were provided with in-depth guidance on how to maintain confidentiality, privacy, neutrality and respect towards respondents. Interview notes were anonymised before being coded and no respondent has been quoted or alluded to in the report in a manner that will identify them, unless explicit permission to ascribe comments has been given. Special attention was paid to informed consent and assent of the adolescents who were invited to participate in discussions. Protocol to safeguard research participants A protocol was in place to enable researchers to address a situation when an adolescent shared that they or others were at risk, or if as researchers, they had witnessed or heard evidence of incidents likely to cause serious harm. A set of practical guidelines were provided to researchers on what to do when an adolescent made a clear disclosure of abuse. An advisory committee with the UNICEF Child Protection Expert and the review team research manager and team leader was set up and on stand-by to deal with any disclosure reported to them by researchers. Team leaders were also provided a set of practical guidelines on what to do if their team member reported a disclosure of abuse, to enable them to take swift action.

2.5

Research Guide

Following approval by the ethical review committee, the research guide was translated into Bahasa Indonesian. This working document was used as the basis for the training of the field team. Based on lessons learned through the training and piloting and inputs from the research team the guide was adjusted before the commencement of the fieldwork. The guide included, • •

details about the research study, what it aims to achieve and how descriptions of the programmes selected for case study


18


• • • • •

basic concepts related to qualitative research, including what it means and the common methods utilised the ethical guidelines and protocol for the study descriptions of how the study will be undertaken, including how respondents will be selected as well as the protocol for interviewing, capturing the data and analysing it the promising practice analysis matrix in Bahasa and the templates for taking notes and recording debriefs the question guides and participatory tools that the team utilised in the field for key informant interviews and focus group discussions.

2.6

Training for Field Researchers

Researcher training took place in Jakarta over five days, from the 5th to the 9th of September 2017. This included one day to practice and test the feasibility and applicability of the research guide, tools and instruments in the field. The training was conducted by the Team Leader and Lead Researcher in English and Bahasa with the help of a qualitative training guide developed for this project. Trainings included role-playing and participatory learning to enable researchers to familiarise themselves with the research guides and practice the participatory activities before fieldwork begins. The training plan is presented in Table 4. Table 4: Training Agenda Day

Activity Introduction to the research study, the promising practice matrix and the selected programmes

Day 1 Discussion on risk posed to adolescent in Indonesia and introduction to qualitative research. Presentation of qualitative research practice and ethical guidelines for the study Day 2 Introduction to the research guides and participatory exercises

Day 3

Discussion and clarification on research instruments, including practice of the research guides and participatory exercises Presentation and discussion on note taking and field debriefs

Day 4

Pilot study (in the field)

Day 5

Fieldwork logistics, planning and wrap-up

2.7

Pretesting

On the fourth day of the training, the researchers were asked to practice using the reseach instruments in a school (junior high school) and community. At school, the researchers were divided into three groups to conduct FGDs with students at grade 9 and teachers using different instruments, including score-card, time line and strength-weakness analysis. In the community, nearby the school, the researchers were divided into two groups: one to facilitate the FGDs and take notes, and the other observe the process. The afternoon session of the pretesting days was to debrief and review the practice. Feedback to researchers was also provided for them to improve the process of FGDs and interview later in the fieldwork stage. © Oxford Policy Management Ltd, November 2017

19


2.8

Data Collection

Data was collected by two teams. Team composition was as follows, • • •

Aceh Team: International Team Leader + 2 national researchers researching the SEG, and two light-touch reviews. Bandung Team: National Team Leader + 4 national researchers. The Bandung team was split into two and cover two programmes – LOLIPOP and MHM and three light-touch reviews. An additional light-touch review was undertaken by the International Team Leader in Jakarta, with the UNICEF Team from Papua.

The responsibilities assigned to each team member were: • • •

The team leaders were the technical leads for the team during the fieldwork, and conducted some key informant interviews with the help of a lead interviewer. The lead interviewers were responsible for facilitating interviews and discussions with research participants The note-takers were responsible for taking notes during the interview.

Each team spent six days within each province/district to greet government officials at provincial and district level and to seek permission to begin the study. Relevant government officials were interviewed to understand their opinion of the programme. On the next day, interviews were conducted with the head of the programme and key programme staff. Group discussions with adolescents who have participated in the programme were carried out, however, in Bandung, the team faced some challenges in conducting FGDs with adolescents. Interviews were conducted with teachers (SEG and MHM) and health workers (LOLIPOP). Parents were intereviewed in all programmes. Discussions with community members were also conducted where possible. At the end of each day, the fieldwork was reviewed and the data collected was analysed. Researchers wrote up their notes in between the fieldwork. Several transcripts were written in Bahasa and were translated after the fieldwork was completed. The team leaders were responsible for synthesising the interviews with different respondents into final case study debriefs. There were debriefs for all in-depth and light-touch case studies.

2.9

Data Management

All interviews and discussions were documented digitally by the researchers. Where feasible, these were written in English. Interview transcripts were labelled using a unique code that helped identify each interview and discussion. Data was stored digitally and shared between the key research staff only. Interview transcripts in Bahasa were professionally translated.

2.10

Data Analysis

The research team was responsible for analysing the data from the case studies. For the in-depth case study, the lead researchers responsible for conducting the research were also responsible for reading, assimilating and writing up the case studies. Once initial drafts were complete, these were reviewed and discussed within the team. Revisions were made and the final analysis was submitted for a quality assurance review to an an internal reviewer.

2.11

Quality Assurance

The report was internally peer-reviewed by a senior technical staff member from OPM's social policy programme. © Oxford Policy Management Ltd, November 2017

20


Regarding data collection, the following quality control mechanisms were put in place: • • •

Tools and protocols have been reviewed by multiple team members to ensure peer examination from both thematic and methodological perspective. Instruments and tools were submitted to OPM’s Ethical Review Board for review and feedback provided was duly incorporated. The research team was comprehensively trained with a strong emphasis on ethics.


21


3

Results

Box 5 Overview of in-depth and light-touch case studies Ranking

Name of the Program

ID/TL

Score

Lead Agency

1

Sabang City Education Grant

ID

10/11

Sabang City Local Government

2

Menstrual Hygiene Management

ID

7/11

UNICEF Indonesia

3

Anaemia prevention in school-going adolescent girls through iron-folicacid supplementation

LT

5/11

Micronutrient Initiative

4

Skills to Succeed

LT

5/11

Save the Children

5

Support for closing the HIV prevention and treatments gaps for Young Key Populations in Indonesia LOLIPOP

ID

3/11

UNICEF Indonesia

0/11


0/11

Indonesia Planned Parenthood Association

#

Promising Practice

Emerging Practice

6

Undertermined Practice


LT

7

Youth Programme

LT

8

Strengthen Child Protection (Restorative Justice)

LT

3/11

Pusat Kajian Pendidikan dan Masyarakat

9

Life-skills Education

LT

0/11

UNICEF Indonesia

Three in-depth case-studies

1. Sabang City Education Grant (SEG), Sabang, Aceh* Lead Agency Sabang City local government

Ranking

Promising Practice

Headline The grant, is defined in robust secondary legislation keeps children in school up to grade 12 and reduces drop-out; also has the reported effect of making the difference between poor and non-poor children disappear. This ‘equity effect’ also appears to have a protective factor, and has contributed to a reported reduction in school bullying. Promising Practice Criteria Scoring 1 = YES meets criterion 0 = NO does not meet criterion


22

documented

accepted practice

M&E

cultural competency

reaches most vulnerable

outcomes

sustainable and replicable

empowered adolescents

innovative

1

theory of change

evidencebased design


1

1

1

1

1

1

1

1

0

1

2. Menstrual Hygiene Management (MHM), Bandung, West Java Lead Agency

Ranking

Headline

The programme is still at its early stages of implementation, however there are strong indications that it improves MHM knowledge, reduces days out-of-school for girls and moderates school bullying.

Emerging Practice

UNICEF

documented

accepted practice

M&E

cultural competency


outcomes



innovative

1

theory of change


Promising Practice Criteria Scoring 1 = YES meets criterion 0 = NO does not meet criterion

0

0

1

1

1

0

1

0

1

1

3. Support for closing the HIV prevention and treatments gaps for Young Key Populations in Indonesia (LOLIPOP), Bandung, West Java Lead Agency

UNICEF

Ranking

Headline

Undetermined Practice

The Programme ran for one year and thus it is not yet clear if it reduces the HIV rate amongst adolescents. This programme goal is ambitious, but the review did not find any evidence of indicators, targets or activities intended to achieve them. With these in place and an extended period of monitored implementation it may have potential to reduce the HIV rate among adolescents

M&E

cultural competency


outcomes



innovative

0

accepted practice

0

documented

evidencebased design theory of change


1

0

0

0

0

0

0

1

1

Six light-touch case studies 4. Anaemia prevention in school-going adolescent girls through iron-folic-acid supplementation, Bandung, West Java


23


Lead Agency

Ranking

Micronutrient Initiative NGO

Headline

A formative evaluation along with a quantitative baseline study in 2016 highlighted some success in IFA up-take. The mechanism for distribution of the IFA supplement is in place regulated and funded by government, Further investigation is warranted into the behaviour change component which was not sufficiently elaborated.

Emerging Practice

M&E

cultural competency


outcomes



innovative

0

accepted practice

1

documented



1

1

0

0

0

0

1

1

0

5. Skills to Success, Bandung, West Java

Lead Agency

Ranking

Save the Children NGO

Headline

The programme supports poor adolescents in and out of school to make productive and safe transitions to the labour market. It appears to have produced several positive outcomes; STC have documented individual case studies although no detailed M&E data was available.

Emerging Practice

M&E

cultural competency


outcomes



innovative

1

accepted practice

1

documentati on



1

0

0

0

0

1

0

1

0

6. The Bandung Street Children Programme, Bandung, West Java

Lead Agency

Ranking

Headline

Perkumpulan Sanggar Sahabat Anak Undetermined Friendly Place Practice for Children Association (PSSA) - NGO

The programme is reliant on external donor funding, operates in a complex environment and reports having positive impacts on the life of adolescent girls who are living and working on the street. Verification data was unavailable. The programme would benefit from a full review of its M&E system, including implementation of its child protection policy.

M&E

cultural competency


outcomes



innovative

0

accepted practice

0

documeted



0

0

0

0

0

0

0

0

0


24


7. Youth Programme, Banda Aceh, Aceh

Lead Agency

Ranking

Indonesia Planned Parenthood Association (PKBI) NGO

Headline

PKBI implement sophisticated programming, (especially the Sexual, Orientation, Gender, Identity, Expression component) in a complex operating environment, intending to improve outcomes for the most vulnerable and marginalised adolescents. Their continued presence and courage in maintaining a dialogue on often controversial issues, is acknowledged

Emerging Practice

M&E

cultural competency


outcomes



innovative

0

accepted practice

0

documented



0

0

0

0

0

0

0

0

0

8. Strengthening Child Protection (Restorative Justice), Banda Aceh, Aceh

Lead Agency Pusat Kajian Pendidikan dan Masyarakat NGO

Ranking

Headline


There is not enough evidence to identify the programme as a promising practice, the findings are of interest and may suggest the emergence of an innovative programme on community justice, which may warrant further investigation.

Promising Practice Criteria Scoring 1 = YES meets criterion 0 = NO does not meet criterion theory of change

documentati on

accepted practice

M&E

cultural competency


outcomes



innovative


0

0

1

0

0

0

0

0

1

1

0

9. Life-skills Education, three target districts, Papua

Lead Agency

UNICEF

Ranking

Headline


The review is constrained by lack of documentation and confusion amongst programmers regarding the intended outcomes. The key success of having the life-skills curriculum integrated into the curriculum for teacher training institutes has not been linked to reduction in HIV prevalence.


innovative

0


0


0

outcomes

M&E

0


accepted practice

0

cultural competency

documented

0

theory of change



0

0

0

0

0

25


3.1

Overview

We identified 19 programmes for review in the preliminary screening. Of these, three programmes met the criteria for in-depth assessment against the promising practice criteria established in this review. Finally, only one programme met the criteria for a promising practice – a government school grant programme for children in Sabang Island, in Aceh Province. One UNICEF-supported programme met the criteria for an emerging practice, the Menstrual Hygiene Management programme, in Bandung, West Java and one UNICEF-supported programme was an undetermined practice, the Support for closing the HIV prevention and treatments gaps for Young Key Populations in Indonesia (LOLIPOP) programme in Bandung, West Java. A further six programmes were subject to a light-touch review; two of these are considered emerging practice and four are considered undetermined practice. Several programmes did not meet the criteria because of insufficient documentation. Either the information was not available, or was not provided to the review team. The in-depth assessment against promising practice criteria ncluded a further review of programme documents, key informant interviews (KII) and focus group discussions (FGDs) with key stakeholders including programme staff, government staff, adolescent participants in the programme intervention, parents and guardians and community members. The methodology for the six light-touch reviews intended the data collection for these to be principally limited to a desk review with the views of programme staff considered where they were available. However, during data collection it became apparent that whilst programme staff were interested and available to discuss the programme the supporting documentation was not necessarily available. These case studies collected information from programme staff members but not from other stakeholder and/or beneficiaries. Therefore, the assessment of this criteria is based on information provided by programme staff and review of any available literature and has not been validated with other sources. It was intended that the case studies would be conducted in three separate geographic locations, however the initial screening and short-listing, applying the Promising Practices criteria, identified three programmes in two locations for in-depth case study. In agreement with UNICEF Indonesia primary data collection was conducted for two in-depth case studies in Bandung, West Java (urban, located close to the capital Jakarta) and for one in-depth case study in Sabang, Aceh (remote rural located in Western Indonesia). Five light-touch case studies were then purposively selected in the two locations. This allows for learning at two levels: to identify why programmes do or do not identify as promising practice as well as to observe the possible impact of co-location. This also allowed for the inclusion of NGO designed and delivered programmes. At the request of UNICEF, the current Life Skills Education programme (that emerged from the former HIV and Lifeskills Education programme 2010-2014) in a third location - Papua and West Papua (remote urban/rural mix located in Eastern Indonesia) - was included as the sixth light-touch case study. The review team met with programme staff in Jakarta and did not visit the location or meet with programme partners.


26


Box 6 Locations of Promising Practice Case Studies

3.2

Three In-depth Case Studies

For ease of reading, the individual case studies are presented here with headline findings only. The reader is referred to the case studies with detailed findings which are included as annexes. The three in-depth case studies presented here relate to (1) the Sabang Education Grant implemented by the Sabang City Administration, (2) the Menstrual Hygiene Management programme supported by UNICEF Indonesia and (3) the Support for closing the HIV prevention and treatments gaps for Young Key Populations in Indonesia programme (LOLIPOP), also supported by UNICEF Indonesia.


27


Sabang Education Grant Please see Annex A. 1 for detailed findings. Sabang Education Grant (SEG) Short Description of the Programme Sabang City is a second-level administrative sub-division in Indonesia, consisting of a main island (Weh Island/Pula Weh) and several smaller uninhabited islands off the northern tip of Sumatra, in Aceh province. The 2017 population is reported by local government as approximately 40,000. The Sabang Education Grant (SEG) is a universal annual cash transfer of 2 million IDR5 per child aged 7-18 years enrolled in a public school, private or integrated boarding school at elementary level, junior high school and senior high school in Sabang. Approximately 7,500 children and adolescents (the total eligible population in Sabang) are in receipt of the grant in the 2017/2018 school year. The SEG was designed, operationalised and has been fully funded by the local government since 2013. The impetus and mechanism for the programme is fully described in (UNICEF 2017a). Summary of findings The SEG is considered a PROMISING PRACTICE. It fully meets 10 of the 11 criteria and partially meets the the 11th. Adolescents in FGD overwhelmingly support the grant and see it as a positive way to help parents to buy school supplies and make sure children go to school. Similarly, teachers acknowledge the positive benefits in terms of enrolment and equity. They report that students are more active, that there is academic improvement as well as change in behaviour as children become more confident. Adolescents and teachers both report the effect of the grant in making the difference between poor and non-poor children disappear. This ‘equity affect’ appears to have a protective factor, and has contributed to a reported reduction in school bullying. The measure of success for the SEG is numbers of children enrolled in school at Primary, Junior High and Secondary levels. UNICEF (2017 pg. 13) report that based on an analysis of government data, “The recent figure for academic year 2015/2016 shows no school drop-out from primary and secondary schools in Sabang”. The net enrollment rate for 2015/2016 decreases progressively from primary to junior high and secondary, however this does not take account of children who may have moved to junior high and secondary school in Banda Aceh. The SEG does not include attendance, grade progression or attainment indicators in its monitoring plan since government report that enrollment is the purpose of the grant. Collection of this type of data is strongly encouraged. The current discussion on extension of the grant to all Sabang adolescents attending Secondary School in Banda Aceh should be considered carefully in the context of a child’s right to education versus a child’s right to live in a family, to prevent unnecessary separation. Consideration should be given to improving quality of education in Sabang schools which would negate the need for this move. The grant appears to have household and local economy benefits beyond the intended outcome of school enrollment. There is currently anecdotal evidence to suggest that this aspect of the SEG should be more thoroughly investigated. This is connected to the current discussions on frequency of payments reported by the local administration. There is some global evidence to 5

At current exchange rates of 1 USD: 13,000 IDR this equates to approximately USD 150.00 © Oxford Policy Management Ltd, November 2017

28


suggest that larger more flexible payments give beneficiaries greater agency and dignity and increases the likelihood that the beneficiary will invest in their own livelihood, however access to more frequent payments may result in improvements in food availability and health-care use. There are hard conditions included in the Mayors Decree No. 8/2016 regarding grant expenditure which appear to be creatively applied. There is a sense amongst some adolescents that when the money is not used directly for school purposes it is fraud and better controls are needed. This aspect of the grant requires further careful consideration prior to scale-up. The information regarding the grant provided to children and parents should include feedback on the wider programme benefits and consultation on proposed amendments. This means letting them know about the possible benefits beyond enrolment, for example local economy, protective factors (potential for reduction in bullying because of increased equality between rich and poor), and links to overall improved wellbeing. Data provided by Sabang City Administration suggests that 10 children with disabilities are enrolled in special schools. As crude calculation (see full case study) suggests that there may be 400 children with disabilities who should be in school. Since the SEG is available to every child enrolled in school this may indicate that poverty is not the only barrier for children with disabilities to enter education, and that increased efforts are warranted to understand and improve educational opportunities for these children.

Promising Practices Criteria *applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion 1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: The SEG demonstrates evidence-based programme design – it is intended to realise a child’s right to education (as defined in the UNCRC) and reflects similar regional cash transfer programmes in its mechanisms. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: The goal of the programme is to support realisation of a child’s right to education through provision of an annual cash transfer for the purchase of school equipment (uniform, shoes, books etc.). 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: The SEG is defined within a robust regulatory framework of national policy and legislation and local secondary legislation which details the mechanism for delivery. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: The SEG is widely and positively acknowledged by government stakeholders, direct beneficiaries and the community as a working mechanism for keeping children in school. 5. *Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution?


*Score 0/1

1 Score 0/1 1 Score 0/1


29


Headline: The governance of the SEG including M&E, is defined in a technical guideline attached to the Mayors Decree No. 8/2016. Members of the management team are responsible for school-based M&E which is verified by the Department of Education. 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: The SEG is universal for all children enrolled in school who have been resident in Sabang for more than 12 months, it is inclusive of male and female children and of children with disabilities. There is evidence that the grant has contributed to improved equity across poor and non-poor peers. There is a current discussion on the introduction of age related incremental increases. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The SEG is universal for all children enrolled in school who have been resident in Sabang for more than 12 months, it is inclusive of male and female children and of children with disabilities. However further action to improve inclusiveness may be warranted. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: The SEG is intended to realise a child’s right to education by reducing poverty related access barriers. The Sabang local government provided data showing that school enrollment is increasing and remains steady up to grade 12 with zero school drop-out reported. Documentation of the programme also notes significant reduction in school drop out (UNICEF 2017 a). The introduction of quality education indicators including attendance, grade progression and attainment may enhance the programme. 9. *Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: The process mechanism for the grant is in place and regulated by national legislation and local secondary legislation. The affordability of the programme has been demonstrated by Sabang over a sustained period of five years. 10. *Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: Adolescents were not involved in the programme design but are involved in programme delivery. The grant is designed to encourage the active participation of adolescents through disbursement of the grant into a personal bank account for each student and the proportional allocation for spending on personal items. Adolescents also reported that they are involved in the decision making around grant expenditure. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: The SEG has evolved over time, has risen to legislative and other challenges and appears to respond to the needs of adolescents’ and families.


1

Score 0/1

1

Score 0/1

1

Score 0/1

1

Score 0/1

1 Score 0/1

0

Score 0/1 1

30


Menstrual Hygiene Management Please see Annex A.2 for detailed findings Menstrual Hygiene Management (MHM), Bandung, West Java Short Description of the Programme The Menstrual Health Management (MHM) programme is in the early implementation stage. The process of preparing and testing the intervention began in 2015. Consequently, the information available to the review was limited. MHM was based on a formative study carried out in 2014 which identified insufficient knowledge and misconceptions about menstruation among adolescent boys and girls. This was followed by a consultative process with adolescents using Human Centered Design (HCD). Following this UNICEF developed a comic book to provide guidance to adolescent girls and boys to improve knowledge and practice of MHM. The intervention provides teachers with information about MHM and how to use the comic book with adolescents. The MHM intervention was designed as a national programme with UKS (national school health programme); it was piloted in Bandung and Biak and evaluated through a baseline and endline study. For this review, primary data was collected in Bandung only. Summary of findings The MHM programme is an EMERGING practice. It fully meets seven of the 11 criteria and is on track to become a promising practice for Indonesia and the region. To facilitate this, it is recommended that the programme design and implementation mechanism and resources are consolidated into an MHM manual which can be used by programmers who may not have the in-depth institutional knowledge. The conclusions of the end-line survey are generally positive. The programme improved students’ awareness and knowledge about menstruation, it changed their attitudes to become more positive about menstruation, it improved girls’ hygiene practices and reduced the number of days they were out of school due to menstruation, and contributed to a reduction in school bullying. Whilst the programme has been designed with and for government and components of the programme support scale-up, local anxieties noted in Bandung suggest there may be a requirement for further support from UNICEF to ensure the programme becomes fully embedded. The programme has been well received by stakeholders and endline data suggests positive results. It is suggested to investigate the maintenance of learning and behaviour change postintervention with the initial cohort. In view of the programmes acknowledged protective factors, the reported reduction in school bullying, it is suggested that UNICEF consider how this can be integrated with other schoolbullying reduction interventions. Similarly, there may be opportunities for linkages to health interventions, or expansion of the methodology through existing programmes targeting marginalised and vulnerable populations. It is understood that a detailed strategy for scale-up is in process. The inclusion of clearly articulated goal and outcomes, together with indicators and targets will contribute to the database for measuring achievement of the SDGs.


31


Promising Practices Criteria * applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion

1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: The MHM - Wash in Schools programme aims to address two of the Sustainable Development Goals (four and six) to improve education facilities to promote safe and effective learning and access to sanitation and hygiene for all. The programme considered regional research and conducted significant formative research to inform the programme design. The pilot testing phase has also been subject to end-line research which is intended to inform anticipated scale-up in 2018. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: The programme focused on both boys and girls and was designed to address both WASH and gender challenges identified through the research. The MHM education component aimed to increase knowledge amongst both boys and girls around menstruation (improve girls’ hygiene, reduce bullying, and reduce girls’ days out of school). The programme was linked to overall improvements in WASH facilities in schools. The programme would benefit from development of a distinct programme document which clearly elaborates the overall goal, intended outcomes, outputs and activities with detailed indicators and targets. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: There are several brochures and other materials describing the programme interventions and results of baseline and endline are available. Several handbooks are included in the MHM Basic Package. This review did not find a guidance document describing the programme and how to administer it. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: Government representatives, parents and teachers all accept and express appreciation for the programme’s activities and consider the programme essential to improving knowledge and practice amongst adolescents. The programme was designed with government (UKS) for national scale-up. The involvement of the Council of Islamic Scholars has contributed to the MHM programming becoming accepted practice 5. Monitoring and evaluation: Is there a monitoring and evaluation plan and evidence of its execution? Headline: A baseline survey was conducted to assess levels of preintervention knowledge. An end-line survey was conducted to assess the effectiveness of comic books after three months of its introduction (UNICEF (2016) and Myriad Research (2017). 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: To address misconceptions around menstruation, the programme worked with Indonesia Council of Islamic Scholars and prepared a handbook on “Menstrual and Health according to Islamic Teachings”. The endline data suggests that this has contributed to the © Oxford Policy Management Ltd, November 2017

*Score 0/1

1

Score 0/1

0

Score 0/1

0

Score 0/1

1

Score 0/1

1 Score 0/1

1 32


dispelling of myths surrounding menstruation (Myriad Research 2017). This endline also provides data on the experience of children with disabilities in MHM. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The programme targets school-going girls and boys at grade 5 and 6. It also included children with disabilities in one third of intervention schools. Although it is reported that scale-up plans will include out of school children in a scale-up phase there is currently insufficient evidence on their involvement. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: The conclusions of the end-line survey are generally positive. The programme improved students’ awareness and knowledge, it changed their attitudes to become more positive about menstruation, it improved girls’ hygiene practices and reduced the number of days they were out of school during to menstruation. Some of schools now have also improved WASH facilities, e.g. provision of sanitary pads and improved toilet and clean water. 9. Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: MHM was designed as a national programme with government partners, and an initial pilot/test phase. As noted above national level changes have been implemented to school health programme guidelines. Locally, government partners in Bandung were aware of the plans for sustainability and scale-up, although they expressed interest in expanding the MHM programme to other schools. Because of the early stage of implementation there is insufficient information to assess sustainability and replicability. 10. Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: Adolescents were involved in the programme design during programme preparation, through the HCD approach to developing the programme intervention, and in the development of the materials (comic books). The early results indicate that the improved knowledge has contributed to reduction in bullying which in turn contributes to empowerment, especially for girls. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: UNICEF report that the comic books have gone through several iterations. Similarly, the delivery mechanisms are evolving, from specific teaching through the curriculum to a more integrated approach.


Score 0/1

0

Score 0/1

1

Score 0/1

0

Score 0/1

1

Score 0/1

1

33


Linkages of Quality Care for Young Key Populations (LOLIPOP) Please see Annex A.3 for detailed findings. LOLIPOP - Linkages of Quality Care of Young Key Populations Short Description of the Programme The LOLIPOP (Linkages of Quality Care for Young Key Populations) programme in Bandung was implemented as a pilot by UNICEF Indonesia in 2015 to increase detection of HIV amongst adolescents and youth aged 15-24 years by promoting HIV testing among young key populations and increasing access to treatment and adherence to decrease new infections. The programme focused on five young key populations (YKPs): males who have sex with males (MSM), female sex workers (FSW), transgender (TG), injecting drug users (IDU) and young people living with HIV, however UNICEF reports that “since the context has a lot to do with access to testing the engagement leans heavily to the first four”, (written communication, October 2017). The programme worked with various stakeholders and utilised existing initiatives and networks in Bandung. There are 4 strategic interventions under the programme: 1) Increasing demand for HIV testing and treatment that includes creative use of prevention and promotional materials, using the peer educator methodology and providing better outreach to YKP; 2) Improving supply by creating adolescent friendly services at community health clinics; 3) Creating an enabling environment by providing supportive laws, policies and regulation; and 4) Generating strategic information with better data to inform decision-making, planning and monitoring.

Summary of findings The LOLIPOP programme is an UNDETERMINED PRACTICE. It fully meets three of the 11 criteria. The goal of LOLIPOP programme was to increase testing, treatment, adherence and decrease new HIV infections for YKPs. It used Kota Bandung as its demonstration site to pilot the approach. Bandung is selected because the city already has stakeholders/institutions that have been working in this area. There is baseline data but it is not clear how this informed programme design. The programme goal is ambitious, but this review did not find any evidence of indicators, targets or description of activities intended to achieve them. Since there are several stakeholders involved in programming with the YKP cohort in Bandung, attribution of results is difficult. Data on increased testing, treatment, adherence and decrease in new HIV infections was not available to this review. UNICEF (2017) report that the comprehensive model framework for YKP-friendly HIV programming, “is a results-based framework, meaning that the higher-level goals and objectives can remain in place, while the exact activities to achieve each objective can change according to changes in technologies, funding, and potential partner interest”. Whilst the flexibility described in the programme design is welcomed, it is difficult to understand how a programme can be sustained and scaled-up based only on a goal and set of objectives.


34


The focus of the programme is YKP aged 15-24, however stakeholders reported difficulties in reaching adolescents aged 15-19 during programme implementation. The review team faced similar difficulties in meeting with this group during field work (only able to meet with 3 adolescents aged 15-19). Most of the beneficiaries appear to be in the cohort over 19 up to age 30. The review team could not identify what specific activities were included to explicitly target adolescents. Quantitative data on involvement of beneficiaries in the adolescent age group was not available to the review team. In view of the strong evidence in support of investments in adolescents, it is recommended that a focus is placed on this age group in future programmes, with specific attention paid to needs and how to meet them. This is particularly important with YKP since “adolescence is a time when gender roles are consolidated, challenged and transformed”, (UNICEF 2017 b). Baseline was followed by a mid-term review within five months, this was not available to the review team and therefore there is insufficient information to make an assessment on progress or recommendations for changes to the activities. The endline data collection is planned for end of 2017. It is recommended that this undertakes a rigorous approach to identifying positive outcomes for YKP with attention to up-take of testing, treatment, adherence and decrease in HIV infection. The programme does not seem to address some cultural sensitivitities, for example by sufficient consultation with religious leaders The LOLIPOP is an undetermined practice. It is not yet clear if it decreases new HIV infections amongst YKP. However, the fact it only ran for one year, and the noted absence of indicators and targets for intended outcomes limits results reporting. Promising Practices Criteria *applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion

1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: LOLIPOP addresses a well-documented need for better services for adolescents with HIV. UNICEF (2016) report that, “Between 2011 and 2015, new reported HIV infections in Indonesia increased annually by 13.1 per cent among adolescents (aged 15 to 19)”. In addition, a baseline study conducted in Bandung in 2015 confirmed that concerns around the lack of privacy and the prevalence of stigma were key deterrents to young key populations (YKP) accessing healthcare. The baseline data was collected in mid-2015, after the programme was initiated and it is not clear how this was used to inform programme design.

*Score 0/1

0

UNICEF (2017, pg. 11) explains that it used the UNICEF MoRES framework (Monitoring Results for Equity Systems) in formulating the LOLIPOP programme, in which equity for young people in key populations requires good access to prevention, testing and treatment services. This reflects programme compliance with the UNCRC.


35


2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: The overall goal of LOLIPOP programme as described by UNICEF (2017) is “to increase testing, treatment, adherence and decrease new infections for YKP”. The objectives are primarily output related and outcomes for YKP relative to the overall goal are not clearly described. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: Several LOLIPOP programme documents detailing its components and strategic interventions as the key approaches for programme implementation, are described by respondents as available at a central level. However, primary data collection found that stakeholders interviewed did not always know of all the documents developed by LOLIPOP. Documents are reported as available but based on responses during KII and FGD the dissemination appears to have been problematic. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: Stakeholders including government health departments, and community members understand the importance of providing health services to adolescents with HIV. However, the programme was not entirely successful in changing the attitudes of health staff and community members who knew of and interacted with the programme, owing to the culturally sensitive nature of the programme content. Concern was expressed around the messages in communication materials which was perceived as encouraging sexual behaviour. 5. Monitoring and evaluation: Is there a monitoring and evaluation plan and evidence of its execution? Headline: There was a clear plan for baseline (2015) and endline data collection (reported as planned for end 2017) to be managed by UNPAD and Burnet Institute. There was also a plan for quarterly meeting for all stakeholders to meet and evaluate programme implementation during 2015. Government and NGO confirmed attendance at these meetings, however meeting minutes were not available. Form and content of the meetings has not been verified. Whilst available documentation describes the goal and objectives, the review has not seen a work plan/activity plan or details of indicators and targets. 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: To increase their involvement in the programme, training was conducted for Peer Educators (PE) to learn how to approach YKPs and invite them to check their health situation at the Puskesmas. The staff at Puskesmas also received training on how to provide adolescent friendly services. Some key informants pointed out the sensitivity of HIV issues in relation to religious belief. However, there was no information available to understand if the LOLIPOP methodology involves consultation with religious leaders.


Score 0/1

0 Score 0/1

1

Score 0/1

0

Score 0/1

0

Score 0/1

0

36


7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: LOLIPOP's main target groups - males who have sex with males (MSM), female sex workers (FSW), transgender (TG), and injecting drug users (IDU) are vulnerable and marginalised. Since they are hard to reach, the programme used Peer Educators and outreach workers to identify the population. Stakeholders reported difficulties in reaching adolescents aged 15-19. The review team faced similar difficulties in meeting with this group during field work (only able to meet with 3 adolescents aged 15-19). Whilst the programme targets a marginalised population there was no evidence of inclusiveness, e.g. involvement of YKP with disabilities or from ethnic minorities. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: An increase in HIV test take-up amongst YKP 2014-2015 is reported. It is not known if this data reflects targets. No information has been made available on increased treatment, adherence, and decrease in new HIV infections. UNICEF (2017) has reported on five key outputs. 9. Sustainable and replicable: Did the programme put in place plans for the programme to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: The programme was piloted in Bandung, where there was favourable support from government and several implementing agencies that had been working in this area and were willing to partner with UNICEF. This pilot took place over one year, and it key informants reported that 2016 was a ‘vacuum’, with minimal activity taking place. It is reported that the training modules have also been delivered in three additional cities (UNICEF 2017). Key informats reported that a local regulation to formalise HIV programming with a focus on adolescents is being introduced. 10. Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: Key informants note that in preparing detailed activities, a consultation meeting with adolescent was held in July 2015, a mid-term review conducted in December 2015 notes that young key populations and adolescents, were involved in planning activities and its implementation. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: UNICEF (2017) report that the comprehensive model framework for YKP- friendly HIV programming, “is a results-based framework, meaning that the higher-level goals and objectives can remain in place, while the exact activities to achieve each objective can change according to changes in technologies, funding, and potential partner interest”.

Score 0/1

0

Score 0/1

0 Score 0/1

0

Score 0/1

1 Score 0/1

1

The programme was designed to be flexible and to adapt to local conditions, and there is evidence to some extent to support this. Stakeholders mentioned that they met regularly to discuss issues/challenges and how to address them, including if they need to change the strategy.


37


3.3

Six Light-touch Case Studies

For ease of reading, the case studies are presented here with headline findings. The case study with detailed findings are included as annexes. The six light-touch case studies presented here relate to (1) Anaemia prevention in school-going adolescent girls through iron folic acid supplementation implemented by Micronutrient Initiative, (2) Skills to Success implemented by Save the Children, (3) The Bandung Street Children Programme implemented by PSSA, (4) The Youth Programme implemented by PKBI, (5) Community Youth Justice Programme implemented by PKPM, and (HIV and Life Skills Education implemented by UNICEF, (Table 6). Anaemia prevention in school-going adolescent girls through IFA supplementation Please see Annex A.4 for detailed findings. Anaemia prevention in school-going adolescent girls through IFA supplementation Short Description of the Programme In 2011 WHO recommended that for iron folic acid (IFA) supplementation to be a useful strategy to prevent anaemia among adolescent girls, programmes should be complemented by a wellinformed and effective Behavioural Change and Communication (BCC) strategy. In 2016 Micronutrient Initiative (MI) with support from UNICEF started a programme to provide weekly IFA supplementation to adolescent girls in schools in West Java and to develop a BCC strategy intended to promote dosing, adherence and coverage of IFA supplementation (Agustina and Prafiantini 2016 a; Agustina and Prafiantini 2016 b). The main activities implemented by the programme are the provision and distribution of IFA supplementation tablets to girls in school age (12 to 19 years old), and capacity building of health personnel and school teachers to provide information, advice and support to the adolescents involved in the programme. Summary of findings The Anaemia prevention in school-going adolescent girls through IFA supplementation programme is an EMERGING PRACTICE. It fully meets five of the 11 criteria. The programme responds to government priorities. The Ministry of Health in Indonesia follows the WHO recommendations to provide IFA supplementation to adolescent girls to prevent anaemia. Anaemia prevention and treatment programmes for girls are priorities for the Ministry of Health. There is also currently a nutrition development programme that is included in the Ministry of Health Strategic Plan 2015-2019. The programme is informed by findings of formative research and a quantitative baseline study (Agustina and Prafiantini 2016 a and Agustina and Prafiantini 2016 b). These highlighted successes and challenges, specifically around incremental coverage, and modification of adolescent, parents and other service providers’ perceptions on the purpose, use, dosage and intended outcomes of the IFA supplements. This contributed to the design of the BCC strategy. Further investigation is warranted into this behaviour change component which was not sufficiently elaborated. There may be more adolescent-friendly approaches to provision of information on anaemia prevention to adolescent girls including social media as well as other more traditional channels like TV and consultation with health workers during visits to health facilities.


38


The programme is implemented through government service providers in in hospitals and Puskesmas and in schools. The programme appears to be sustainable as the mechanism for distribution of the IFA supplementation is in place and regulated by the government, furthermore, the government through the MoH is funding the initiative. Whilst this review could find evidence of formal programme documents such as a well-defined theory of change and/or a monitoring and evaluation system, the programme appears to have well-articulated links between activities and outcomes. It has furthermore developed a curriculum, to train educators, teachers and medical staff. Promising Practices Criteria * applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion

1. *Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: The design of the IFA programme is based on the 2011 WHO recommendations for IFA and informed by two studies conducted in 2016. A qualitative formative research study carried out in two programme intervention districts and a quantitative baseline survey conducted in the same two districts, plus Bandung as a comparison district. The baseline survey’s findings show worse results in anaemia prevalence, intake rates, availability and coverage of IFA supplements in the intervention areas 2. A Theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: Programme staff described the results chain from activity to result. No programme documentation was available to support this. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: The programme has developed a comprehensive curriculum to train educators, teachers and medical staff at Puskesmas. The training curriculum and modules provide a clear description of the programme components and its administration. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: Based on findings from the formative research, adolescent girls, teachers, parents and service providers had erroneous conceptions of the use and value of using IFA supplementation to prevent anaemia among adolescent girls. Some respondents also thought the channels used to reach adolescent were not appropriate and instead the use of social media as opposed to more traditional channels was encouraged. 5. Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: This review did not come identify any explicit monitoring and evaluation plan. However, baseline data is available and there is a plan to collect post-intervention comparison data.


**Score 0/1

1

Score 0/1

0 Score 0/1

1 Score 0/1 1

Score 0/1 0

39


6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: The IFA programme is inclusive of all school-going adolescent girls. There is not enough information to determine whether it is inclusive of girls with disabilities or from specific religious or ethnic backgrounds. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The review did not come across any explicit mechanism to reach the poorest and most marginalised adolescents. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: Most the evidence collected through the baseline study suggest the outcomes and/or impact of the IFA supplementation programme could have been better. Some results suggest that majority of girls in implementation districts didn’t receive the supplementation and that knowledge and understanding regarding dosage, duration, and benefits of the intervention were limited. 9. *Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: The mechanism for distribution of the IFA supplementation is in place and regulated by the government, furthermore, the government through the MoH is funding the initiative. 10. *Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: The programme design has been informed by qualitative evidence collected from adolescent girls including their understanding of the causes and consequences of anaemia, their perception towards IFA supplementation and their opinion on the channels and appropriateness of the information delivered. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: The MI programme is still in its early stages of implementation (pilot phase) and therefore there is insufficient evidence on its capacity to adapt and adjust to challenges.


Score 0/1

0 Score 0/1

0 Score 0/1

0

Score 0/1

1

Score 0/1

1

Score 0/1 0

40


Skills to Success Please see Annex A.5 for detailed findings.

Skills to Success Short Description of the Programme Since 2012 Save the Children’s (StC) Skill to Succeed programme (STS) aims to empower adolescents and young people aged 16-24 in Bandung through the acquisition of skills and competencies to pursue work aspirations. STS is an extension of the youth-based education programme which integrates skills education, entrepreneurship and work placements to meet the transition to work needs of both in-school and out-of-school adolescents. The programme aims to help adolescents to obtain advanced training opportunities, access decent jobs, return to or stay in both formal and non-formal education and combine continuation of studies while working. The intended outcomes of the programme are; 1) key capabilities are strengthened, 2) availability and access to decent economic opportunities increased, and 3) Enabling environments fostered. The STS programme engages with a range of government and non-government district level partners, including the multi-stakeholders networks, local NGOs, teachers association and the private sector etc. to open up opportunities for internships and sustainable employment of adolescents and youth. Summary Findings The Skills to Success programme is an EMERGING PRACTICE. Based on the available information it fully meets five of the 11 criteria. It may benefit from a more in-depth review against the promising practice criteria. The design of the STS programme is informed by findings of a previous programme evaluation in which adolescents claimed the economic development component was not sufficiently dressed. StC report that it has been reaching poor adolescents in and out of school and has documented case studies of positive outcomes. It also has served as a platform for a range of stakeholders to collaborate by bringing together private and public sectors, government, and businesses to find decent and quality job opportunities and hopes for the future of adolescents. The programme Theory of Change, M&E mechanism, other relevant programme documentation and defined plan for future sustainability or/and replicability was not available to this review. The STS programme is an important intervention serving a population of adolescents at a critical juncture in their transition from childhood to adulthood.

Promising Practices Criteria * applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion

1. *Evidence-based programme design: How does programme design comply with international and national obligations and commitments on


**Score 0/1

41


human rights and gender equality; how is the programme design evidence based? Headline: The project design is informed by the findings of an evaluation of a previous StC programme to reduce exploitive child labour through education and economic empowerment (EXCEED). It is also guided by StC’s global work on adolescents’ transition to work, including the ‘Adolescents Skills for Successful Transitions’ position paper. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: Whilst a document outlining the programme’s theory of change, or explicit results chain/programme logic ws not available to this review, key respondents report that they are aware of programme documents guiding the implementation of activities to meet the main goal of facilitating adolescent transition to work. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: The STS has produced an informative brochure outlining the main purposes and components of the programme as well as training modules to increase adolescent skills. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: The STS programme has established good collaboration with district government officials, the local teachers’ forum, local NGOs and the private sector. There is insufficient information to assess the involvement and acceptance by community members, families and guardians and adolescents themselves. 5. *Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: An explicit monitoring and evaluation plan or evidence of its implementation was not available to this review. 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: The programme is inclusive of male and female adolescents in and out of school. There is, however, not enough information regarding adolescent with disabilities or from different ethnic or other marginalised groups. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The project target beneficiaries are youth at risk of being engaged in exploitative labour, including those out of school and those living in poor households. Sex disaggregated data was not available, nor other data indicating inclusiveness.


1

Score 0/1

1

Score 0/1

1 Score 0/1

0

Score 0/1 0 Score 0/1

0 Score 0/1

0

42


8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme?

Score 0/1

Headline: In 2015 lessons learned were documented highlighting stories of success. The report states that in addition to the educational and employment results, the building of self-confidence among adolescent was also perceived as a positive outcome. 9. *Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures?

1

Headline: Whilst the programme has implemented several measures to ensure sustainability, such the engagement of an interdisciplinary working group, it remains highly reliant on external funding. 10. *Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: Adolescents were consulted for the evaluation of the previous EXCEED programme which highlighted the weakness of the economic development component. This finding served to improve the design of the current STS programme. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: A certain degree of flexibility can be seen in the programme, however the review team had insufficient information to provide a definitive assessment.

Score 0/1

0

Score 0/1

1

Score 0/1 0

The Bandung Street Children Programme Please see Annex A.6 for detailed findings.

Short Description of the Programme The Bandung Street Children Programme (BSCP) is run by the Perkumpulan Sanggar Sahabat Anak (Friendly Place for Children Association) which was established in 2014. BSCP estimate that some 3,000 to 11,000 children live in the streets of Bandung, national figures go as high as 5 million boys and girls1. Most of these children and adolescents are at risk of exploitation, sexual abuse, trafficking, substance abuse etc. Girls are especially at risk of sexual exploitation, unwanted pregnancies, HIV infection and psychological distress which can have long-term consequences for their health and development. The BSCP programme focuses on providing a safe space for adolescent girls living on the street with inadequate or no parental or adult care. Through the “Art in a Box” activities, programme outreach workers gather and engage children living in the streets. This activity provides a space to create and develop artistic talent, as well as to deliver important health and protection messages. It also helps to identify girls in need for further basic services and/or to be provided with the opportunity of living in the “home for girls” where they are under the supervision of responsible adults, receive formal education, regular health check-ups, and support to transition into work and adulthood © Oxford Policy Management Ltd, November 2017

43


Summary of findings The BSCP programme is an UNDETERMINED PRACTICE. It does not fuly meet any of the 11 criteria. This is in part due to the limited availability of documentary evidence. The BSCP programme is a donor supported service which aims to fill the gap in statutory service provision for vulnerable children and adolescents living on the streets in a specific location. The programme operates in a complex environment and reports having positive impacts on the life of adolescent girls who are living and working on the street. The girls admitted to the “Home for girls” are offered protection and care and provided with basic health and education services as well as opportunities for development, growth, and transition to work. Programme staff report having re-located to a less central environment to prevent girls resuming their life on the street. This poses an ethical and programmatic challenge for BSCP. However, the organisation appears to be responding positively by recognising the value of a family strengthening approach. Although the programme reaches a great number of children living in the streets of Bandung through the “Art in a Box” component. the home for girl does not have the capacity to admit all. The programme appears focus on admitting the poorest and most marginalised girls, the the home for girls. There is little evidence of the programmes evidence-based design, theory of change and/or a monitoring and evaluation plan against which to measure results. There is also no information on the inclusion of adolescents in the design or implementation phases. The programme does not appear sustainability or replicable but demonstrates clear intention to provide the best care they can to beneficiaries with limited resources. The BSCP references its local affiliations, compliance with local legislation and advocacy with local government for continued financial stability. This review was unable to meet adolescent girls involved in the programme, however it recommended that this cohort is explicitly included in future research on adolescent wellbeing. Similarly, BSCP may benefit from a full review of its monitoring and evaluation systems, and explicitly those which involve M&E of its child protection policy. Promising Practices Criteria *applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion 12. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based?

*Score 0/1

Headline: There is insufficient information to provide an assessment of this criterion. 13. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes?

0 Score 0/1

Headline: Programme staff describe the goal of the programme as, “to break the life cycle of adolescent girls living in the streets” by providing them with education, care, support and shelter. Outreach workers identify children though implementation of “Art in a Box” activities. 14. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it?


0 Score 0/1

44


Headline: There is no evidence of formal programme documentation, the programme has a detailed web-page and issues a newsletter. 15. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: The BSCP programme has good collaboration with sub-district government and providers of basic services such as health and certification of births. 16. *Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: The review of the BSCP did not come across any explicit monitoring and evaluation plan or evidence of its implementation. 17. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: The programme through its component of "home for girls” aims to be inclusive of all adolescent girls regardless of religion or ethnicity. There is not enough information regarding adolescent with disabilities. 18. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The BSCP focuses on reaching girls most at risk and in need of support, including those at risk of serious child protection violations. 19. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: The review did not come across any explicit formal independent study demonstrating positive outcomes and/or impact other than anecdotal individual stories. 20. *Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: There is not enough information to discern whether the programme has sustainability or replicability plans in place. The programme is a primarily donor driven service provider filling a gap in statutory service provision. 21. *Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: The review did not come across any explicit engagement of adolescent on the design of activities. 22. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: The BSCP is a donor driven programme which is required to respond flexibly on a regular basis, and which reports flexibility and adaptability in responding to the needs of adolescent girls.


0 Score 0/1 0 Score 0/1 0 Score 0/1


0 Score 0/1

0 Score 0/1 0 Score 0/1 0

45


Youth Programme. Indonesia Planned Parenhood Association Please see Annex A.7 for detailed findings.

Youth Programme. Indonesia Planned Parenhood Association (PKBI) Short Description of the Programme The goals of the PKBI national strategy 2010-2020 are i) Developing models and standards of quality sexual and reproductive health services to meet the needs of the community, ii) Empower people to fight for sexual and reproductive rights for themselves and others, iii) Developing STI and HIV / AIDS Prevention and Control Prevention and iv) Advocate at all levels of the organization to decision-makers to ensure the fulfilment of sexual and reproductive rights and health6. The Executive Director of PKBI Aceh reports that their Youth Programme has four components, i) Sexual, Orientation, Gender, Identity, Expression (SOGIE), ii) Youth Centre iii) Young Voice, and iv) Youth Forum. The information available to the research team included the PKBI 2015 Annual Report and Key Informant Interview transcript with the Executive Director of PKBI Aceh. Summary of findings The Youth Programme is an UNDETERMINED PRACTICE. It does not meet fully the 11 criteria. The programme partially meets many of the criteria however there was insufficient information provided to make a full determination. Aspects of the programme suggest there may be potential to move towards promising practice and the progamme may warrant a fuller review against the criteria. The discourse on Aceh takes place within the post-tsunami and post-conflict context and against the background of the enforcement of some provisions of Islamic criminal law. This contributes to complexities in the operating environment for NGOs, particularly those with a rights-based agenda. It is within this context that PKBI Aceh are working to implement sophisticated programming which is intended to improve outcomes for the most vulnerable and marginalised adolescents. Given the limited information available to this light-touch review it is not possible to conclusively report the PKBI Aceh Youth Programme as a Promising Practice in Adolescent Programming. Components of the programme are integrated and accepted by local government and communities. The components which target the most marginalised LGBT adolesents are less accepted. Their continued presence, and courage in maintaining a dialogue on often controversial issues, is acknowledged. Future support on programme implementation should consider the promising practice criteria as the foundation of good programme design. Promising Practices Criteria *applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion

1. *Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: The PKBI Aceh Youth Programme is in line with the overall goals of the national organisations strategy and with a human-rights based 6

*Score 0/1

0

http://pkbi.or.id/tentang-kami/renstra-2010-2020/


46


approach. However, there was insufficient information about specific programmes to make a full assessment of this criterion. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: Intended outcomes are linked to the organisations national strategy. Progress is constrained by funding. There was insufficient information about specific programmes to make a full assessment of this criteria. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: There was insufficient information to assess this criterion. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: Youth Centres are a long-standing initiative which operate on volunteer support and can be considered accepted practice locally. Young Voice is a community programme which is reported as being implemented with the support of teachers, parents and planners. Youth Forums is nascent due to limited donor funding. Initially supported by government through the Social Affairs Office in Banda Aceh, the SOGIE programme is reported as struggling with acceptance due to local religious conservatism. 5. Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution?

Score 0/1

0 Score 0/1

0 Score 0/1

0

Score 0/1

Headline: There was insufficient information to assess this criterion 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities?

0

Headline: There is some evidence of cultural competency and partnerships. However, there was not sufficient information to fully assess this criterion. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations?

0

Headline: The overall concept of the Youth Programme, evidenced in its several components, is one of social inclusion and participation. There is insufficient evidence of indicators and targets for a conclusive finding that the intention is achieved. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? how do programme participants perceive the benefits of the programme? Headline: Positive impacts on adolescent lives are reported by the organisations Executive Director although no independent study has been conducted, and this review did not have the capacity to directly contact beneficiaries.


Score 0/1

0

Score 0/1

0

47


9. *Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: There was insufficient information to assess this criterion, however unreliable programme funding is reported by PKBI Aceh as affecting development. 10. *Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: The overall philosophy of the Youth Programme is one of participation and engagement leading to the empowerment of adolescent boys and girls. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges?

Score 0/1

0

Score 0/1

0

Score 0/1

Headline: The continued existence of the programmes as reported, in a complex context is evidence of some level of flexibility and innovation.

0

Strengthening Child Protection (Restorative Justice) Please see Annex A.8 for detailed findings. Strengthening Child Protection (Restorative Justice) in Aceh Besar and Banda Aceh Short Description of the Programme The programme is implemented by the NGO Pusat Kajian Pendidikan dan Masyarakat (PKPM – Centre for Education and Community Studies) in collaboration with the Restorative Justice Working Group, supported by UNICEF. It aims to use customary law, which is in line with existing Law on Criminal Justice System for Children, and the non-formal justice system, (rather than through police and court system) in diverting children in conflict with law. They are assisting villages to (a) to formulate a local regulation (called reusam); and (b) to record the cases and local judgements and sentencing. The information was provided by programme staff during a focus group discussion, however no programme documents were shared. Summary Findings The Strengthening Child Protection (Restorative Justice) progrmme is an UNDETERMINED PRACTICE. It meets three of the 11 criteria. Whilst there is not enough evidence to identify the programme as a promising practice, the findings are of interest and may suggest the emergence of an innovative programme on community justice (and wider child protection). This programme may warrant a more in-depth evaluation. Promising Practices Criteria * applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion

1. *Evidence-based programme design: How does programme design comply with international and national obligations and commitments on


**Score 0/1

48


human rights and gender equality; how is the programme design evidence based? Headline: The programme is based on local Regulation No. 8/2008 concerning Development of Life and Customary Law, modeled the Justice System Law (No. 11/2012), especially for the article on diversion. There was insufficient information additional to assess this criterion. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: There was insufficient information to assess this criterion. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: The programme is defined within a robust regulatory framework of national policy and legislation and local secondary legislation. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: PKPM report that they have introduced the customary justice mechanism for mediation into four villages, with the cooperation of community leaders. There was no case information available to verify this; PKPM did not describe any successful interventions and the extent to which the community accept the programme is unconfirmed. 5. *Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: PKPM programme staff report that they monitor the number of registered cases and that these cases have decreased. The reason for the decrease is not explained. PKPM did not provide sufficient information to verify. 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: The programme is embedded in community leadership and customary law and is reported by programme staff as accepted in 4 villages. There is no information regarding the involvement of marginalised adolescents. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: This programme targets children in conflict with the law. There is insufficient information regarding integrated mechanisms for working with other marginalised groups, or specific differences in procedures for work with girls and boys. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme?


0 Score 0/1

0 Score 0/1

1 Score 0/1 0

Score 0/1 0

Score 0/1

0

Score 0/1

0

Score 0/1

49


Headline: The programme has not been subject to an independent evaluation however the programme staff report that because of community mediation the number of cases referred to the customary court has reduced. 9. *Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: PKPM has scaled the programme from four to six villages in 2017. In the initial four villages, the programme for child protection (community mediation and other child protection issues) is included in the village budget. 10. *Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: PKPM programme staff report the involvement of adolescents in the village child forum and in the drafting of the reusam. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: There is insufficient information to assess this criterion.

0

Score 0/1

1

Score 0/1

1 Score 0/1 0

Life-skills Education Please see annex A. 9 for detailed findings. Life-skills Education Short Description of the Programme The current Life-skills Education Programme in Papua is a UNICEF Programme emerging from the HIV and Life-skills programme which ran from 2010-2014. An end of programme evaluation showed that 91 per cent of schools in the target area provided HIV education to students although the programme did not manage to produce significant and sustained changes in the lives of young people and noted that the HIV epidemic among the targeted population group remained unchanged (Burnett Institute and Survey Meter 2014). The significant result currently reported is that the local education department has accepted the life skills education programme as teaching programme with accredited teaching hours, which means that schools can implement within their approved curriculum. In association with a local university this programme is now being rolled out to schools. The light-touch case study is based on a focus group discussion with UNICEF staff in Jakarta and a review of documents related to the initial programme. Summary Findings The Life-skills Education programme is an UNDETERMINED PRACTICE. It does not fully meet any of the 11 criteria. This may be in part because of the limited data available to the review team.


50


The assessment of the current Life-skills programme as a promising practice is significantly constrained by the lack of documentation related to the programme. It is not possible to know if the detailed recommendations of the 2014 evaluation have been implemented, and if not why not. For example, due to significant changes in the operating environment. Further, the discussion with the programme staff elicited information which suggests there is confusion at grass-roots level about the intended outcomes of the programme. For example, some referred to it as a programme for HIV prevention, whilst others considered it as a programme for building adolescents confidence. This suggests that results will be limited, and where they are reported cannot necessarily be attributed to this intervention. There is some evidence to suggest that the success of having the life-skills curriculum integrated into the curriculum for teacher training institutes is considered the optimum result. The next phase should link this achievement to the reduction in HIV prevalence in intervention areas (or another agreed outcome). As a starting point the programme design could be reviewed to clarify the overall goal, and to create coherence across the education and health sector components. This will contribute to development of clear indicators and targets. Promising Practices Criteria *applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion

1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: The evidence base for the redesigned programme is difficult to determine because there is insufficient information regarding the current programme intended outcomes, outputs and activities. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: Staff directly involved in programme implementation were unable to articulate the intended programme outcomes or the intended results chain. The perceptions of intended outcome varied between, ‘reduction in HIV prevalence’, or, ‘helping young people to be more confident’, to, ‘helping teachers to teach life-skills better’. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: There is insufficient information to make a full assessment of this criterion. UNICEF report that the Life-skills Curriculum is available and currently being delivered through the teacher training institutes. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: There was insufficient information to score this criteria in this programme phase. However, the 2014 evaluation noted the creation of supportive policy environment. UNICEF report that, “this has been integrated as a subject in “local content” in schools, registered in DAPODIK (National Ministry list of approved subjects), (written communication October 2017). 5.*Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: Whilst the 2010-2014 programme was subject to an evaluation and regular reports to donors, no documentary information on this current programme was made available. © Oxford Policy Management Ltd, November 2017

**Score 0/1

0 Score 0/1

0

Score 0/1

0 Score 0/1 0

Score 0/1 0

51


6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: There is insufficient information to assess if the recommendations of the 2014 evaluation related to inclusion do not appear to have been addressed. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The 2014 evaluation recommends addressing equity imbalances by focussing on reaching adolescents in more remote and rural locations and by strengthening links between education and health sectors. There is insufficient information to understand if this approach has been adopted. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? how do programme participants perceive the benefits of the programme? Headline: The current programme has not been subject to an independent evaluation. UNICEF report that the Life-skills curriculum has been adopted locally however the results for adolescents are not known. 9. *Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: The 2014 evaluation notes that sustainability rests on incorporation of the life-skills training into the teacher training curriculum. This has been a significant achievement of the programme. There is insufficient information to assess if the recommended supportive measures have been put in place or what changes the introduction of the life-skills curriculum has made in the lives of adolescents. 10. *Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: There is insufficient information to make an assessment on participation of adolescents in programme design or delivery. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: There is insufficient information to make an assessment on participation of adolescents in programme design or delivery.

3.4

Score 0/1

0

Score 0/1

0

Score 0/1

0

Score 0/1

0

Score 0/1

0 Score 0/1 0

Promising Practice Methodological Guide

The framework for planning and measuring promising practice in programming for adolescent wellbeing is a low-cost qualitative methodology that has been tested extensively in Indonesia, and can help to monitor the implementation of a programme and to clearly demonstrate why the chosen intervention is a promising practice. To enable this methodology to be applied to future programme, design, monitoring and evaluation, the criteria were developed into an accompanying guide to support programmers in the future to use the framework to establish the credibility of a programme (Annex B). Credibility with stakeholders is a critical asset for successful programming activity. It means that organisations can assure the people who are involved in and who directly benefit from programme interventions that their investments, of both time and resources, will achieve results, because you © Oxford Policy Management Ltd, November 2017

52


are using a tried and tested design. Stakeholders include clients, beneficiaries, or recipients of an intervention. Other stakeholders can include your managers and colleagues, donors, government and non-government organisations7 and private sector partners. This guide was developed to help programmers to select and to design and deliver credible programmes for adolescents. It provides a framework for planning effective strategies and interventions to make best use of limited resources in contexts of inequality and huge need and can contribute to the design of monitoring and evaluation plans. If a programme is already in the implementation phase, framework can be sued as a benchmarking tool. It can help programmers to see if a programme is on-track to be considered a promising practice, and if necessary to alter course. This framework was developed and tested in Indonesia to review adolescent programmes. However, the principles of the framework are considered universal and can be adapted for use in any programme context. It can be used as a as a point of reference, • • • • •

7

during programme design phase (supports planning for scalable and replicable design) for regular planned M&E and mid-term evaluation by steering/advisory group meetings by managers with programme staff (internal critical review) as a guide for decisions about what data needs to be collected.

Includes faith-based organisations © Oxford Policy Management Ltd, November 2017

53


4

Summary Conclusions

The focus on 10-19 year olds comes at a time when significant evidence is building to suggest that adolescence provides, “a second window of opportunity to influence developmental trajectories and make up for poor childhood experiences”, (UNICEF 2017a). UNICEF Indonesia report that, “In Indonesia, there are approximately 46 million adolescents, aged 10 to 19, which represent 18 per cent of the total Indonesian population”, (UNICEF undated), and that, “this group represents an important foundation for the nation's long term growth and development … and will be key to Indonesia achieving its global sustainable development goals” (“IndonesianYouth – a foundation for the future,” 2017). Overall, the review finds that the context for adolescent programming in Indonesia is heterogeneous and complex. Inequalities persist and the underlying trends are predicted to continue. This will likely exacerbate the poor outlook for youth employment which may contribute to increasing inequalities (Economist Intelligence Unit 2016a and 2016b). The links between education and employment for economic growth stand-out as critical contextual factors for adolescent programming. This includes both ensuring that adolescents stay in school or other educational systems and that the quality of the learning programmes offered meets their needs. At the same time, HIV infections amongst adolescents in Indonesia are increasing, in part due to limited access to HIV testing among adolescents compared to adults (UNICEF 2015 and UNICEF 2015 a; Ministry of Health and UNICEF 2016). Data on adolescent nutrition, experience of violence, child marriage and school bullying suggest continued vulnerability (GAIN 2014; Indonesia Ministry of National Development Planning and the United Nations Children’s Fund 2017; Rachmi, Li and Bauer 2017; UNICEF and 18+ coalition, undated). Of the programmes identified for case study review, one programme related specifically to maintaining adolescents in school up to Grade 12, and one related to transition to work. A separate programme worked on Life-skills education with yet another programming for LGBT adolescents, a key population for HIV risk. All the programmes were operational at different scales in different locations. UNICEF report in a written communication (October 2017) that there are two planned initiatives linking MHM and child marriage and the Adolescent Kit and child marriage. However, there was no evidence during this review of efforts to co-locate interventions, a strategy which could contribute to improved outcomes for overall wellbeing of adolescents. This dilution of effort has potential consequences for both effectiveness and efficiency. Adolescent programming is a critical intervention area for UNICEF Indonesia. Significant key interventions are underway in programmatic silos - the links/integration across programme areas is not well-defined through a strategic plan for adolescent programming which details intended outcomes. There is limited evidence of coherence across programme streams where areas of interest converge. For example, Menstrual Hygiene Management programme being implemented by UNICEF WASH section includes a component which shows early evidence for prevention of school bullying. This is not yet integrated with a separate school-bullying programme, implemented by UNICEF Child Protection section which is applying a different methodology in a different location. Similarly, an access to justice programme working with adolescents in prison, had not considered links to health sector response around HIV prevention. The current UNICEF Life-skills Educaton programme in Papua does not appear to have integrated the education and health sector components which was a key recommendation of the 2014 HIV and Life-skills Education programme evaluation. This dissonance may be in part related to an identified perception of programming for adolescent wellbeing as a separate and distinct programme stream, in addition to health, education, child protection and WASH. Several UNICEF programme staff did not perceive a programme as an adolescent programme even when it targeted adolescents, unless it was specifically designated such.


54


The case studies highlighted the challenges in equity focused programming. When programme design had an explicit mandate to meet the needs of a specific marginalised community, for example LGBT adolescents, it did this well. However, when this focus was not explicit, by default programmes tend to be exclusive rather than inclusive. Thus, LGBT adolescents are not visible in programmes which do not target them overtly. Similarly, adolescents who are potentially exposed to the most risk of rights violations are least served by programme interventions and include those living in rural and remote locations, those out-of-school, those with a minority ethnic background and those with a disability. The review was constrained by access to appropriate information, notwithstanding the copious documents provided to the initial document review. Much of the available foundational evidence presented in support of specific adolescent programmes is anecdotal rather than evidence-based, which is a critical consideration for measurement of promising practice. Although a significant number of documents were provided, often these did not provide corroboration of the programme interventions subsequently described during key informant interviews. Availability of financial data detailing programme cost, for start-up and on-going running costs was a significant gap, which contributed to limitations on assessing sustainability and replicability. Knowledge management across UNICEF and partners is identified as a critical strategic focus for advocacy for child rights. Without this, success cannot be acknowledged and sustainability and replicability, key criteria for promising practice, cannot be assured. Adolescents responding to the U-Report poll indicated that their biggest problem/concern is bullying, followed by education, popularity and identity, and access to employment. Most said they would try and solve the problem/deal with the concern by joining a community activity or talking to a friend or teacher. Since each of the three in-depth case studies have been found to have a protective element, specifically around bullying prevention and incidence reduction, and provide a community space for discussion and resolution of these problems, this suggests that current programme design is to some extent correctly targeting the needs of adolescents, as described by them. The appearance of the Sabang Education Grant as the sole promising practice consolidates findings from the document review. This programme fully meets 10 of 11 criteria to be considered a promising practice and partially meets the final 11th criterion. It also appears to confirm global evidence that staying in school improves adolescent wellbeing across all domains. This programme warrants further investigation to validate. The MHM programme is positioned as an emerging practice and anticipated promising practice. The current stage of implementation does not yet allow for a full assessment of outcomes; however, the programme design and initial results suggest that it will achieve promising practice status. The assessment of the LOLIPOP programme was constrained by limited data. Equally the current stage of implementation is an obstacle to it being considered a promising practice. Further investment in strategic planning, monitoring and data collection are required to move towards promising practice. Specific recommendations for each of the nine programmes which were involved in case study review are included in the summary conclusions for each case study. In a context of limited resources and significant need, consideration could be given to how future programming can be concentrated on achieving significant results. This approach, should focus on concerted investments in proven interventions which impact on overall adolescent wellbeing. For example, the effects of the Sebang Education Grant appear to extend beyond maintaining children in school to produce equity effects including protective factors. The grant also appears to have positive effects on household and local economy which may warrant further investigation. This approach should also be applied to a rigorous ‘weeding-out’ of programmes which do not have a clearly articulated results chain, with measurable indicators and targets. © Oxford Policy Management Ltd, November 2017

55


Whilst the constraints and challenges noted above can be related to external factors including donor requirements, to resolve these challenges it is suggested that UNICEF consider a formative evaluation of the combined cross-sectoral support provided to Indonesia through its programme strategy. This can take account of the newest evidence baseline for the SDGs as well as recent emerging evidence on HIV and access to quality education etc. and can consider how to improve coherence and understanding of adolescent programming across programme sectors. This formative evaluation should place an emphasis on results-based management and adaptive design with concern for outcomes, impact and sustainability. It should consider opportunities and challenges for programmatic coherence and integration relative to the current political and social context of Indonesia. It should consider if and how current programmes are diffused or concentrated and the effectiveness and efficiency of these approaches. Further it can provide a strategic learning opportunity for inclusion of cross-cutting themes, for example gender, adolescence, and disability. Alternatively, a series of targeted evaluations of key pilot inititatives showing improvements in adolescent well-being and generating lessons learned for evidence-based policy is recommended.


56


Annexes

5

Annex A.1

Sabang Education Grant In-depth Case Study

Annex A.2

Menstrual Hygiene Management In-depth Case Study

Annex A.3

Support for closing the HIV Prvention and treatment Gaps for Young Key Populations in Indonesia (LOLIPOP) In-depth Case Study

Annex A.4

Anaemia preention in school-going adolescent girls through Iron Folic Acid supplementation Light-touch Case Study

Annex A.5

Skills to Success Light-touch Case Study

Annex A.6

The Bandung Street Children Programme Light-touch Case Study

Annex A.7

Youth Programme Light-touch Case Study

Annex A.8

Community Youth Justice Light-touch Case Study

Annex A.9

HIV and Life Skills Education Light-touch Case Study

Annex B

How do you know what’s good for me? A guide for planning and measuring promising practices in programming for adolescent wellbeing


57


References / Bibliography Agustina, R. and Prafiantini, E. (2016 a) Formative research for an improved Iron Folic Acid supplementation program for school going adolescent girls in selected districts of West Java province in Indonesia. Univeritas Indonesia and Micronutrient Initiative: Jakarta Agustina, R. and Prafiantini, E. (2016 b) Report of a Baseline Survey. Quantitative study for the improved Iron Folic Acid supplementation program for school-going adolescent girls in selected districts of West Java province Indonesia. Univeritas Indonesia and Micronutrient Initiative: Jakarta Barkley JS, Kendrick KL, Codling K, Muslimatun S, Pachón H. Anaemia prevalence over time in Indonesia: estimates from the 1997, 2000, and 2008 Indonesia Family Life Surveys. Asia Pac J Clin Nutr. 2015;24(3):452-5. Burnet Institute, UNPAD and UNICEF (2015) Demonstration Site of Friendly HIV Services for Young Key Population in Bandung: Baseline Evaluation. Jakarta: UNICEF Indonesia Burnet Institute (2016) Demonstration site of Friendly HIV Services: LOLIPOP Mid Term Review Report. Based on notes of workshop prepared by UNPAD. Carvalho, M.A. and Koteng, Z. (2014) Systematic assessment of the Child Friendly City/District (CFC/D) initiative in Indonesia. UNICEF: Jakarta Economist Intelligence Unit (2016 a) Megatrends to 2030. UNICEF: Jakarta Economist Intelligence Unit (2016 b) Scenarios to 2030. UNICEF: Jakarta GAIN (2014) Landscape Report on Adolescent and Maternal Nutrition in Indonesia Retrieved 3rd July 2017 from http://www.gainhealth.org/wp-content/uploads/2016/08/Landscape-Report-onAdolescent-and-Maternal-Nutrition-in-Indonesia-ilovepdf-compressed-ilovepdf-compressedilovepdf-compressed-lower.pdf Grandy, G (2010) Instrumental Case Study in Mills, A. J., Durepos, G., & Wiebe, E. (Eds.). Encyclopaedia of Case Study Research: Vol 1 PP 474-475 Sage ILO. (2011). One Roof School Program and Its Impact on Child Labor: A Case Study in Sukabumi District. Indonesia. Indonesia Ministry of National Development Planning and the United Nations Children’s Fund (2017). SDG Baseline Report on Children in Indonesia. Jakarta: BAPPENAS and UNICEF. Indonesia Ministry of National Development Planning and the United Nations Children’s Fund (2017). SDG Baseline Report on Children in Indonesia. Jakarta: BAPPENAS and UNICEF. Indonesian Planned Parenthood Association (2015) Annual Report 2015. Achieving the Fulfilment of Sexual and Reproductive Health Rights. PKBI: Jakarta Indonesian Youth – A foundation for the future (2017) Retrieved from https://www.unicef.org/indonesia/children_2834.html LOLIPOP Facebook page (https://www.facebook.com/The-HIV-Lolipop-Programme-Indonesia1594913080773041/) Ministry of Health Indonesia and UNICEF (2016) Brief overview: HIV among Adolescent and Young Key Populations (Aged 15-19 and 20-24) in Indonesia. Ministry of Health and UNICEF: Jakarta Myriad Research (2017) End-line Survey on Menstrual Hygiene Management (MHM) Interventions in Schools in Bandung: Key Findings. Presentation.


58


Pillay, H. and Haribowo, H. (2013) Review of SATAP Schools. UNICEF: Jakarta Ramadhan, A.S. (2013) Youth Policies in Indonesia: Activating the Role of Youth. Retrieved from http://www.youthpolicy.org/national/Indonesia_2013_Youth_Policy_Review.pdf Reality Check Approach plus and UNICEF Indonesia (2016) Children and Their Families Perspectives and Experiences on Poverty and Social Protection. Jakarta: The Palladium Group and UNICEF Indonesia Sammon, E.M., Jagmag, M, Martinez, M. and Wahyudi, R. (3017 b) Inception Report. Overview of Promising Practices in Indonesia by UNICEF and other partners. Jakarta: UNICEF Indonesia Sammon, E.M., Martinez, M., Jagmag, M. and Wahyudi, R. (2017 a) Document Review of Promising Practices in Indonesia by UNICEF and other partners. Jakarta: UNICEF Indonesia Thome, K., Taylor, E.J., Mateusz, F., Davis, B. and Handa, S. (2016) The local economy impacts of social cash transfers: A comparative analysis of seven sub-Saharan countries. Rome: FAO UNDP (2016) The Impact of Cash Transfers on the Local Economy. Brazil: The International Policy Centre for Inclusive Growth UNICEF (2011) The State of the World’s Children. Adolescence -- An Age of Opportunity. UNICEF: New York UNICEF (2015 a) Averting New HIV infections among young people in Papua and West Papua: and education sector response. UNICEF: Jakarta UNICEF (2015) Support for Closing the Prevention and Treatment Gaps for Young Key Populations in Indonesia. Jakarta: UNICEF Indonesia UNICEF (2016) WASH in Schools in Indonesia: Incredible Opportunities, An Overview of the Current Situation with Recommendation for Progress. Jakarta: UNICEF Indonesia UNICEF (2017 b) Working with and for Adolescents UNICEF Programme Guidance for the Second Decade of a Child’s Life. Version 12. UNICEF: New York UNICEF (2017) All In to End Adolescent AIDS in Indonesia; UNICEF Report (draft) to MAC AIDS Foundation. Jakarta: UNICEF Indonesia UNICEF (2017a) Sabang Education Grant. Ensuring Free and 12-Year Compulsory Education For All Children. Final Report. UNICEF: Jakarta UNICEF and 18+ Coalition (undated) Research Brief. Revealing the Truth of Marriage Dispensation: An Analysis of Child Marriage Practice in Tuban, Bogor, And Mamuju. UNICEF: Jakarta UNICEF and INNOVESIA (2015) UNICEF Indonesia Innovation Human-Centered Design Activity for (Menstrual Hygiene Management) Concept Development UNICEF (undated) Unleashing Adolescents Potential. UNICEF: Jakarta WHO (2011) Guideline: Intermittent iron and folic acid supplementation in menstruating women. Geneva, World Health Organization. WHO (2015) Global school-based student health survey (GSHS). Indonesia. Retrieved from http://www.who.int/chp/gshs/indonesia/en/ [accessed 29th October 2017]


59


Annex A A.1

Detailed Case Studies

Sabang Education Grant In-depth Case Study

Name of the programme 5 x domains of the UNICEF Adolescent Country Tracker (ACT) Principle domain of ACT which applies to the programme

Sabang Education Grant (SEG)

Health and wellbeing

Education and learning

Transition to work

√

√

Participation and engagement

√

Subsidiary domains of ACT which apply to the programme Short description of the programme

Protection

Sabang City is a second-level administrative sub-division in Indonesia, consisting of a main island (Weh Island/Pula Weh) and several smaller uninhabited islands off the northern tip of Sumatra, in Aceh province. The 2017 population is reported by local government as approximately 40,000. The Sabang Education Grant (SEG) is a universal annual cash transfer of 2 million IDR8 per child aged 7-18 years enrolled in a public school, private or integrated boarding school at elementary level, junior high school and senior high school in Sabang. Approximately 7,500 children and adolescents (the total eligible population in Sabang) are in receipt of the grant in the 2017/2018 school year. The SEG was designed, operationalised and has been fully funded by the local government since 2013. The impetus and mechanism for the programme is fully described in the 2017 report by UNICEF. The purpose of this case study is to better understand the outcomes for adolescents, by reviewing the programme against specific promising practice criteria to identify the degree to which the plan, implementation and perceptions of change can be considered promising. * applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion

Good Idea Undetermined Practice

Emerging Practice

Promising Practice

Promising Practices Criteria

8

At current exchange rates of 1 USD: 13,000 IDR this equates to approximately USD 150.00 © Oxford Policy Management Ltd, November 2017

60


1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: The SEG demonstrates evidence-based programme design – it is intended to realise a child’s right to education (as defined in the UNCRC) and reflects similar regional cash transfer programmes in its systems. Detailed findings: The goal of the programme is described by local government key informants as “to enable realisation of a child’s right to education, as described in the UNCRC”. The programme is described as intended to address poverty related access to education. The design is based on local poverty data, local education data, existing social protection programmes in Indonesia and specifically the Aceh Jaya school based cash transfer programme introduced in 2006. The programme is inclusive of both boys and girls in equal numbers9 and children with disabilities at both primary, junior high school and secondary school. The Mayor, who initiated the programme in 2012, is a former member of the provincial parliament and has had access to the government’s wider social protection initiatives, which is also reported as having informed the design. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: The goal of the programme is to support realisation of a child’s right to education through provision of an annual cash transfer for the purchase of school equipment (uniform, shoes, books etc.). Detailed findings: Key informants from local government reported that the goal of the SEG is to support realisation of a child’s right to education. As described by UNICEF (2017 pg.16) the objectives are described in the Mayor’s Decree No.8/2016 on technical guidelines for the management of the educational grant programme in Sabang, a special/frontier region, for 12 (twelve) years compulsory education for students, • • •

“ensure free and compulsory 12 (twelve) year education programs for all students, improve self-confidence and learning motivation among students in the learning and interaction process, and improve student achievement through various educational activities that foster the spirit of learning and competition”.

**Score 0/1

1

Score 0/1

1

The inputs, outputs and outcome pathway are described in the Mayor’s Decree and by key informants to this review, Box 1. The allowable expenditure of the annual cash transfer of IDR 2 million is specified in the Mayors Decree No.8/2016 and includes purchase of school uniform, stationary, extra-curricular activities and other learning supports. There is an explicit prohibition on the grant being used for other purposes (UNICEF 2017 pg. 16). Key informants report that 10 % of the transfer can be used by children and adolescents for daily expenditures (school snacks, school social fund10 etc.).

9

2016-2017 Sabang local government data on enrollment The school social fund is a collection by pupils which can contribute towards a celebration, towards sending condolences, etc., whilst participation in the fund is voluntary, non-participation is a stigmatising indicator of poverty 10


61


The indicators for success are reported by key informants as % increase in school enrolment, especially at grades VII-XII and % decrease in school dropout. Comparison of school enrollment grade progression data provided by the local Department of Education indicates minor fluctuations in numbers, Table 1 Box 1. Input, output, outcome pathway

This may be explained by children either leaving school or moving to another school in Banda Aceh, or for those at vocational secondary school, leaving to take up employment. Table 1. School enrollment and grade progression Year 10 2015 Year 11 2016 SMA/MA Secondary 439 422 school/ Madrasa Aliyah SMK Vocational 87 68 secondary school

Year 12 2017 413

57

UNICEF (2017 pg. 13) report that based on an analysis of government data, “The recent figure for academic year 2015/2016 shows no school drop-out from primary and secondary schools in Sabang”. The net enrollment rate for 2015/2016 decreases progressively from primary to junior high and secondary, however this does not take account of children who may have moved to junior high and secondary school in Banda Aceh. The indicators relate specifically to school enrollment. School attendance, grade progression and learning attainment are not included. Key informant reports that through making school enrollment a reality for all children they have fulfilled their obligations. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: The SEG is defined within a robust regulatory framework of national policy and legislation and local secondary legislation which details the mechanism for delivery. Detailed findings: The SEG is permissible and defined by specific national and local legislation. Specifically, the Mayor’s Decree No.8/2016 on technical guidelines for the management of the educational grant programme in Sabang, a special/frontier region, for 12 (twelve) years compulsory education for students, describes how the grant shall be administered.

Score 0/1

1

UNICEF (2017) have recorded and documented “the overall mechanism of the Sabang Education Grant to understand its process (from planning to reporting), challenges, and to identify opportunities for improvement”. The full regulatory framework for the SEG is discussed at Chapter 2.5 (Ibid).


62


The grant has also been subject to a Reality Check Approach review (Reality Check approach plus and UNICEF Indonesia 2016) which, “explores and provides a deeper understanding of children’s experience of poverty and their experience of the current nationwide and district-run cash transfer programmes”. Whilst this study was not exclusive to but included Sabang amongst other programmes, the overall conclusion was that in areas where social protection (cash transfer interventions) were available, “People feel that access to education facilities has improved” (Ibid pg. xiv). Programme documentation which speaks directly to the Sabang constituency was less evident. Whilst some adolescent FGD respondents referred to the availability of written materials provided for their parents’ information during ‘socialisation’ meetings in school, these documents were not available to this review. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: The SEG is widely acknowledged by government stakeholders, direct beneficiaries and the community as a working mechanism for keeping children in school. Detailed findings: A total of 50 key informants and focus group respondents (including 33 adolescents) provided consistently positive reports on the benefits of the programme. Government officials and teachers noted that school enrollment had increased at junior high and secondary levels. Adolescents noted that some children who had previously dropped out of school had returned to secondary school to continue their studies following the introduction of the SEG.

Score 0/1

“As a government employee, my salary is not very high and I have 4 children so the grant is very useful to make sure my children have everything they need to go to school”. (KII Sabang local government) One adolescent noted that the community know about and accept the programme because, “if my neighbour sees me on the street he will ask why I am not at school, and will tell me to go to school because I get a grant for being there”, (FGD respondent Junior High School) Adolescent respondents in FGDs noted that they had been in receipt of the grant for four years and that it was important for them that it continue because it helps to remove a perceived burden from their parents to pay for school costs. Respondents provided anecdotal evidence of positive household and local economy impacts. In some cases, these were connected to opinions of inappropriate use of the grant, “my friend says one of his workers used the school grant to buy a motorbike instead of school books, and that he was using it to get to a second job - I told him this was a good thing because his children were in school and the grant was now helping him to benefit his family economically and then he understood” (KII UNICEF)

1

In a separate case the family of one student beneficiary were reported as having used the SEG to invest in equipment to set up an ironing business. Adolescents also reported use of the grant for items outside the scope of the allowable expenses detailed in the Mayors Decree No. 8/2016. However, respondents also noted that expenditure on uniforms and school equipment took precedence and children were in school wearing appropriate uniform and with the required equipment. © Oxford Policy Management Ltd, November 2017

63


5. Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: The governance of the SEG including M&E, is defined in a technical guideline attached to the Mayors Decree No. 8/2016. Members of the management team are responsible for school-based M&E which is verified by the Department of Education. Detailed findings: Fiscal management, monitoring and evaluation is conducted through the established government channels including budgetary reporting and audit. The SEG has also established a Management Team within the Department for Education whose responsibilities include data verification, school-based monitoring, complaints handling, and reporting (UNICEF 2017 pg. 32).

Score 0/1

The enrolment of students is based on verification of a required set of documents by the homeroom teacher supervised by the school principal. This includes evidence of 12 months’ local residence, which condition is required since students from Banda Aceh have been moving to Sabang to enrol in school. The local Department of Education provided full enrollment data to this review for the period 2013-2017. Although there are conditionality’s described in the Mayors Decree No.8/2016 these appear to be ‘soft’. School-based monitoring requires teachers to signoff on a student’s adherence to the conditionality of the grant for the payment to be released. In effect this amounts to teachers checking that students have the correct uniform and present well at school with appropriate and required school equipment. The teacher usually deals with any challenges by calling the parents in to address the concern, so that it is usually not elevated to the Department of Education for imposition of sanctions.

1

The local government report that, “the SEG is solely to support a child’s right to education, and since we have data showing that children are in school we do not need to evaluate it, it is already achieving what we wanted it to achieve”, (KII Sabang local government). Thus, enrollment is the key indicator. School attendance, grade progression and attainment are not included. Approximately 1 % of transfers were not processed for the 2016/2017 school year. In the case of 80 students’ parents reported to the Department of Education that their child’s details had not been transferred. This anomaly has been corrected. 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: The SEG is universal for all children enrolled in school who have been resident in Sabang for more than 12 months, it is inclusive of male and female children and of children with disabilities. Detailed findings: School enrollment data indicates there are 9 % (n=68) fewer girls than boys enrolled in Junior high school SMP) and 24 % (n=45) more girls enrolled in Madrasa Tsanawiah (MTS) at grades 7-9. The situation reverses for girls in Secondary (SMA) and Madrasa Aliyah (MA) where there are 9 % (55) more girls than boys enrolled in grades 10-12.


Score 0/1

1

64


There are 9 adolescents with disabilities enrolled in the Junior High School for children with disabilities (SMPLB) at grades 7-9, at least one of whom has an intellectual disability. Eight of these adolescents were included in an FGD and reported receipt of the grant. Teachers noted that there are more children with disabilities living in the community and not attending school. This suggests that poverty is not the principle challenge related to school enrollment for this cohort. Additionally, two adolescents anonymously selfidentified as having a disability during FGDs. Adolescents and teachers both report the effect of the grant in making the difference between poor and non-poor children disappear. This ‘equity effect’ appears to have a protective factor, “My friends used to bully me because my shoes were broken and I couldn’t afford to buy new ones, I am happy and have more confidence going to school now because we are all the same, I have a good uniform and good shoes and the same pocket money”, (FGD adolescents Secondary school). Teachers report a similar effect, “bullying from the past does not exist, they are no longer competing with each other and are feeling more togetherness…students are more active, there is academic improvement as well as change in behaviour as children become more confident. it helps that they also have pocket money from the grant”, (FGD teachers Secondary school) Students reported having more expenses at Secondary school level, for example uniforms and shoes are costlier for adolescents than for primary school age children. Girls also reported that their uniforms are more expensive than boys. They suggested that an incremental increase in the grant for Junior High School and Secondary School students would be valued. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The SEG is universal for all children enrolled in school who have been resident in Sabang for more than 12 months, it is inclusive of male and female children and of children with disabilities. However further action to improve inclusiveness may be warranted. There is evidence that the grant has contributed to improved equity across poor and non-poor peers. There is a current discussion on the introduction of age related incremental increases. Detailed findings: The SEG is a universal grant for all children enrolled in school. It includes both boys and girls with slightly more boys than girls at Junior high school and more girls than boys in grades 10-12. Children migrating to Sabang must provide evidence of 12 months’ residency to be enrolled into the programme. The local government describe the SEG as a grant to eliminate poverty related access to education. The universality is justified because, “sometimes it’s difficult to choose who is poor and who is not poor and this can also cause social difficulty. With this programme children are targeted just because they are children and it doesn’t matter if they are rich or poor; there are other programmes targeting the poor using poverty data and you can still get these grants even if you are getting the education grant”, (KKI Sabang local government).


Score 0/1

1

65


In one school visited the school Director reported that 50 % of the 437 children came from low income families. Adolescents involved in FGDs told this review that children whom they knew has already dropped-out of school had returned when the grant was initiated. The programme is inclusive of children with disabilities. However, fewer than 10 children are enrolled in the Junior High School for children with disabilities and in receipt of the grant, and only two of 33 adolescents involved in FGDs self-identified as having a disability. A crude calculation based solely on the number of children enrolled in Sabang schools in 2017 (n=8,134) and taking the lowest prevalence rate of 5 %, Box 2, suggests that there may be 400 children with disabilities who should be in school. Box 2. Estimating disability prevalence Whilst disability prevalence is difficult to estimate, the United States 2016 Disability Compendium reports a national rate of 5.4 % amongst 5-17 year olds. This is in line with the Global Burden of Disease estimates for 2004, cited in the 2011 World Disability Report of 5.2 % for males aged 0-14 and 5.0 % for females in the same age range (WHO and World Bank 2011 pge.30). Since the SEG is available to every child enrolled in school this may indicate that poverty is not the only barrier for children with disabilities to enter education, and that increased efforts are warranted to understand and improve educational opportunities for these children. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: The SEG is intended to realise a child’s right to education by reducing poverty related access barriers. The Sabang local government provided data showing that school enrollment is increasing and remains steady up to grade 12 with zero school drop-out reported. Documentation of the programme also notes significant reduction in school drop out (UNICEF 2017 a). The introduction of quality education indicators including attendance, grade progression and attainment may enhance the programme. Detailed findings: There is evidence of positive outcomes for children. The findings of this review concur with those of UNICEF (2017) in that the SEG has proven effective in removing poverty related access barriers to education. There is further emerging evidence that the grant generates equity related protective factors especially for students from poor families and that there are opportunities to increase inclusiveness for children with disabilities. Whilst an independent evaluation of the SEG has yet to be conducted both UNICEF (2017) and Reality Check Approach plus and UNICEF Indonesia (2016) have investigated the programme to generate documentation and evidence of improved outcomes for children directly related to the grant.

Score 0/1

1

The injection of 1.25 million USD11 per year into the community of Sabang is likely to have had significant effect both at household and local economy level. The aggregate effects of supporting investments by poor people can contribute to equity. During the last ten years, there has been significant research and evidence generation on the benefits of cash transfers to the local economy (Thome and others 2016; UNDP 2015). There is currently

11

8,134 students enrolled in 2017/2018 school year at approximately USD 150 per student © Oxford Policy Management Ltd, November 2017

66


anecdotal evidence to suggest that this aspect of the SEG should be more thoroughly investigated. The SEG is perceived as beneficial by adolescents, teachers and the community. Adolescents take it seriously and report that it motivates them to study harder, “we all think this is a good programme and is having a positive influence”, and “the most important thing about the grant is that it gives us the motivation to stay in school”, and “the grant reduces the burden on our parents, it makes us more motivated to go to school”, (FGD adolescents Junior High School and Secondary School). Teachers report that students are more active, that there is academic improvement as well as change in behaviour as children become more confident. The impact on child marriage has not been measured. What is known is that more girls than boys are staying in school up to grade 12 and that these girls are not married. There has been a reported increase in perceptions of equity, “we are all the same now”, and reduction in reported bullying, “I am no longer mocked or humiliated because of my poor school uniform”. The perceived success has contributed to the local government recent decision to establish a universal child grant for children aged 0-6. 9. *Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: The process mechanism for the grant is in place and regulated by national legislation and local secondary legislation. The affordability of the programme has been demonstrated by Sabang over a sustained period of five years. Detailed findings: The process mechanism for the grant is in place and regulated by national legislation and local secondary legislation. The grant has been offered consistently since 2013 to all children who are in school, and there are plans to add a universal child grant for 0-6 year olds. Although the provincial government experiences some challenges assigning the expenditure to the appropriate budget line, it appears the foundations are in place and only minor legislative amendments are necessary for national scale-up. Full budget replicability for national scale-up is not known, however initial indications from UNICEF social policy team suggest it is affordable. A new regulation has made the provincial education office in Banda Aceh responsible for secondary schools and the Sabang local government has maintained open communications to ensure the grant remains in place under their management, “we were concerned that they would want to take the grant over too but so far it hasn’t happened because we make sure to consult with them regularly”. (KII Sabang local government)

Score 0/1

1

One adolescent respondent suggested that the grant was in place and operational due to the efforts of the former mayor, and that it may be withdrawn under the new incumbent, “the former mayor was very successful in having this programme especially for poor families, it’s 50/50 if the new mayor will keep it going, we think he might want to stop it and take the money so everyone can have free electricity instead”. (FGD Adolescents Secondary School). The veracity of this statement could not be confirmed. Other respondents noted that the demand and support for the grant would continue, © Oxford Policy Management Ltd, November 2017

67


“It needs to continue because poor students especially find it very helpful” (FGD Adolescents Secondary School) “If it doesn’t continue parents will march to the Mayor’s office and demand for it to go on”. (FGD Adolescents Secondary School) “We should not lose this grant, even if it requires a big budget”. (FGD Adolescents Junior High School) 10. Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: Adolescents were not involved during the design process; however, the grant is designed to encourage their active participation through disbursement into a personal bank account for each student and the proportional allocation for spending on personal items. Detailed findings: The programme was designed by local government and a process of ‘socialisation’ applied which involved teachers and parents. The understanding of the grant mechanism has consequently filtered down to the children and adolescents. The adolescents understand the grant is to enable them to stay in school, and to, “pay back the government through our hard work and achievement”, (KII Adolescents Secondary School).

Score 0/1

The grant was designed to be paid into individual bank accounts, for the younger age group these are joint accounts with parents for the older age group these are indivdual accounts in the name of the adolescent. During FGDs adolescents reported that they are involved in the decision making around expenditure of the grant. They reported that although the local government does not ask their opinion they are given space to self-manage the funds, and are, “allowed to make decisions about how to spend the money on what we need”. (FGD Adolescents Secondary School)

0

Adolescent girls reported in FGDs that some girls are buying make-up, clothes and phone credit which increases confidence and helps them to “feel better”. One girl reported that, “Having a phone and credit for the internet connection helps me with school work; learning from the internet is a really useful support for my school work”. (FGD Adolescents Secondary School). The adolescents involved in the FGDs did ask for more information about the grant, “we should have more information because the money is really for our use and our parents are only supposed to guide us in how to spend it”. (FGD Adolescents Secondary School) “The government should explain more to adolescents about this programme and ask our opinions about this grant. We think students should be enthusiastic about learning with this grant and not abuse it for other purposes”. (FGD Adolescents Secondary School) In a discussion with adolescent high school boys they agreed that the grant gives them an opportunity to learn how to manage money. Both adolescent and teacher respondents repeatedly noted that it empowered young people by eradicating the social stigma of poverty; they consistently reported that the difference between rich and poor children was no longer evident and noted the positive benefits of very poor families receiving more than one grant.


68


In one FGD the students proposed that the fund for awarding grants to outstanding students (which ended when the SEG was introduced) should be reinstated as a motivational tool. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: The SEG has evolved over time, has risen to legislative and other challenges and appears to respond to the needs of adolescents’ and families. Detailed findings: The SEG appears to have been a well-designed programme from the beginning. It has evolved over time to cope with contextual challenges and opportunities including concerns by the provincial government regarding budgetary allocations and the shift in management of secondary schools to provincial government. Adolescents noted in their FGDs the increased costs associated with age, for example cost of uniforms and shoes are higher for 16 year olds than 7 year olds, and suggested that there should be an increase in the amount received for Junior high school and Secondary school students. The local government also acknowledged this important issue and reported that they are considering including incremental increases for students in Junior high and Secondary school. This suggests that there is an open communication mechanism where issues are discussed and taken on board. During the 206/2017 school year 80 eligible children (1 per cent of the total) did not receive the grant because they were not transferred in the administrative system. They have remained in school on the understanding that the payment is guaranteed in the following year which allows parents to borrow against future income. In the meantime, the local government are putting in place measures to prevent this happening in the future.

Score 0/1

1

Evidence of the programme evolving is also seen in the eligibility programme for those Sabang students attending Secondary School in Banda Aceh. This currently allows students attending the “best” Secondary School to receive the grant. Local government report they are considering expanding this to all children moving to Banda Aceh at secondary level. Discussions are also being held regarding the regularity of the payment; introduction of the child grant for children aged 0-6 may involve a monthly payment, which perhaps may also be applied to the SEG. Local government reported during KII that they were not in favour of monthly payments which they considered administratively heavy and noted that parents preferred a lump sum. However, that the discussion is taking place points to flexibility in this innovative programme design. Summary conclusion The SEG is considered a PROMISING PRACTICE. It fully meets 10 of the 11 criteria and partially meets the the 11th. Adolescents in FGD overwhelmingly support the grant and see it as a positive way to help parents to buy school supplies and make sure children go to school. Similarly, teachers acknowledge the positive benefits in terms of enrolment and equity. They report that students are more active, that there is academic improvement as well as change in behaviour as children become more confident.


69


Adolescents and teachers both report the effect of the grant in making the difference between poor and non-poor children disappear. This ‘equity affect’ appears to have a protective factor, and has contributed to a reported reduction in school bullying. The measure of success for the SEG is numbers of children enrolled in school at Primary, Junior High and Secondary levels. UNICEF (2017 pg. 13) report that based on an analysis of government data, “The recent figure for academic year 2015/2016 shows no school drop-out from primary and secondary schools in Sabang”. The net enrollment rate for 2015/2016 decreases progressively from primary to junior high and secondary, however this does not take account of children who may have moved to junior high and secondary school in Banda Aceh. The SEG does not include attendance, grade progression or attainment indicators in its monitoring plan since government report that enrollment is the purpose of the grant. Collection of this type of data is strongly encouraged. The current discussion on extension of the grant to all Sabang adolescents attending Secondary School in Banda Aceh should be considered carefully in the context of a child’s right to education versus a child’s right to live in a family, to prevent unnecessary separation. Consideration should be given to improving quality of education in Sabang schools which would negate the need for this move. The grant appears to have household and local economy benefits beyond the intended outcome of school enrollment. There is currently anecdotal evidence to suggest that this aspect of the SEG should be more thoroughly investigated. This is connected to the current discussions on frequency of payments reported by the local administration. There is some global evidence to suggest that larger more flexible payments give beneficiaries greater agency and dignity and increases the likelihood that the beneficiary will invest in their own livelihood, however access to more frequent payments may result in improvements in food availability and health-care use. There are hard conditions included in the Mayors Decree No. 8/2016 regarding grant expenditure which appear to be creatively applied. There is a sense amongst some adolescents that when the money is not used directly for school purposes it is fraud and better controls are needed. This aspect of the grant requires further careful consideration prior to scale-up. The information regarding the grant provided to children and parents should include feedback on the wider programme benefits and consultation on proposed amendments. This means letting them know about the possible benefits beyond enrolment, for example local economy, protective factors (potential for reduction in bullying because of increased equality between rich and poor), and links to overall improved wellbeing. Data provided by Sabang City Administration suggests that 10 children with disabilities are enrolled in special schools. As crude calculation (see full case study) suggests that there may be 400 children with disabilities who should be in school. Since the SEG is available to every child enrolled in school this may indicate that poverty is not the only barrier for children with disabilities to enter education, and that increased efforts are warranted to understand and improve educational opportunities for these children.


70


# and details of KII and FGD conducted # FGD/KII participa description M F nts FGD 4 Secondary school (SMA12) -teachers 0 4 Junior high school for children with FGD 5 2 3 disabilities (SMPLB) -teachers Junior high school for children with FGD 9 8 1 disabilities (SMPLB) - adolescents Secondary school (SMA) FGD 8 0 8 adolescents FGD 4 Junior high school (SMP) - teachers 0 4 Secondary school (SMA) FGD 7 0 7 adolescents Junior high school (SMA) FGD 5 0 5 adolescents Junior high school (SMP) – FGD 5 5 0 adolescents BAPPEDA (Regional Development KII 2 Planning Agency) and Education 2 0 Department KII 1 UNICEF 1 0 References: Reality Check Approach plus and UNICEF Indonesia (2016) Children and Their Families Perspectives and Experiences on Poverty and Social Protection. Jakarta: The Palladium Group and UNICEF Indonesia Sammon, E.M., Martinez, M., Jagmag, M. and Wahyudi, R. (2017) Document Review of Promising Practices in Indonesia by UNICEF and other partners. Jakarta: UNICEF Indonesia Thome, K., Taylor, E.J., Mateusz, F., Davis, B. and Handa, S. (2016) The local economy impacts of social cash transfers: A comparative analysis of seven sub-Saharan countries. Rome: FAO UNDP (2016) The Impact of Cash Transfers on the Local Economy. Brazil: The International Policy Centre for Inclusive Growth UNICEF (2017) Sabang Education Grant. Ensuring free and 12-year compulsory education for all children. Jakarta: UNICEF Indonesia

12

The Indonesian acronyms for the school level © Oxford Policy Management Ltd, November 2017

71


A.2

MHM In-depth Case Study

Name of the programme

5 x domains of the UNICEF Adolescent Country Tracker (ACT) Principle domain of ACT which applies to the programme Subsidiary domains of ACT which apply to the programme Short description of the programme

Menstrual Hygiene Management – WASH in Schools



Protection

Transition to work


√ √

√

The Menstrual Health Management (MHM): Wash in Schools Programme is in the early implementation stage. The process of preparing and testing the intervention began in 2015. Consequently, the information available to the review was limited. MHM was based on a formative study carried out in 2014 which identified insufficient knowledge and misconceptions about menstruation among adolescent boys and girls. This was followed by a consultative process with adolescents using Human Centered Design (HCD). Following this UNICEF developed a comic book to provide guidance to adolescent girls and boys to improve knowledge and practice of MHM. The intervention provides teachers with information about MHM and how to use the comic book with adolescents. The MHM intervention was designed as a national programme with UKS (national school health programme); it was piloted in Bandung and Biak and evaluated through a baseline and endline study. For this review, OPM has collected primary data in Bandung only. * applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion;


Emerging Practice

Promising Practice

Promising Practices Criteria 1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: The MHM - Wash in Schools programme aims to address two of the Sustainable Development Goals (four and six) to improve education facilities to promote safe and effective learning and access to sanitation and hygiene for all. The programme considered regional research and conducted significant formative research to inform the programme design. The pilot testing phase has also been subject to end-line research which is intended to inform anticipated scale-up in 2018. © Oxford Policy Management Ltd, November 2017

*Score 0/1

72


Detailed findings: Information about menstrual health practices in Indonesia was limited to the 2012 Indonesian Demographic Health Survey (IDHS) which found that almost a quarter of the women surveyed had not discussed menstruation before their first menses. Therefore, in 2014, UNICEF Indonesia commissioned a study in four provinces (Papua, East Java, South Sulawesi and NTT) to explore MHM knowledge, attitudes and practices amongst adolescents. Based on these findings the intervention itself was developed using Human-Centred Design (HCD) involving adolescents (UNICEF 2016). 1 The programme aims to improve adolescent wellbeing and decrease school drop-out rates. It specifically speaks to the Sustainable Development Goals (SDG), specifically targets in SDG 4, “to build and upgrade education facilities that are child, disability and gender sensitive and provide safe, nonviolent, inclusive and effective learning environments for all”, and in SDG 6, “to achieve access to adequate and equitable sanitation and hygiene for all and end open defecation, paying special attention to the needs of women and girls and those in vulnerable situations”. During the review, stakeholders in Bandung, including adolescents, respondents in government, teachers at schools and implementing partners did not reference the associated research. However, government respondents and teachers agreed that adolescents at the outset of the programme intervention, did not have enough knowledge about menstruation hygiene and how to manage it. This is confirmed by the endline research (Myriad Research 2017). 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: The programme focused on both boys and girls and was designed to address both WASH and gender challenges identified through the research. The MHM education component aimed to increase knowledge amongst both boys and girls around menstruation (improve girls’ hygiene, reduce bullying, and reduce girls’ days out of school). The programme was linked to overall improvements in WASH facilities in schools. The programme would benefit from development of a distinct programme document which clearly elaborates the overall goal, intended outcomes, outputs and activities with detailed indicators and targets. Detailed findings: The programme identified the challenges through formative research, engaged with boys and girls to develop innovative communication materials, worked to up-date the national school health programme guidelines to include MHM, created resources to promote respectful relationships and address bullying and harassment of girls, partnered with the Council of Islamic Scholars to produce guidance on menstruation from an Islamic perspective and has a strategy in place to scale up from pilot stage to all 34 provinces.

Score 0/1

0

During KII and FGD stakeholders reported that that they knew that the programme is expected to provide knowledge about menstruation to adolescent girls and boys as well as increase the quality of facilities in schools. The UNICEF programme team provided several documents to the review team, including PowerPoint presentations, describing baseline and end-line


73


research, describing the intervention as part of an overall WASH programme for schools and flyers communicating results. At this current stage of implementation, the programme would benefit from development of a distinct programme document which clearly elaborates the overall goal, intended outcomes, outputs and activities with detailed indicators and targets. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: UNICEF (2016) describes the programme, its genesis, implementation and results from the pilot. The End-line Survey on Menstrual Hygiene Management (MHM) Interventions in Schools in Bandung: Key Findings provides detailed background to the methodology employed and the results (Myriad Research 2017). Several brochures and other materials describing the programme interventions and results are available. Several handbooks are included in the MHM Basic Package. Detailed findings: UNICEF (2016) describes the programme, its genesis, implementation and results from the pilot. The End-line Survey on Menstrual Hygiene Management (MHM) Interventions in Schools in Bandung: Key Findings provides detailed background to the methodology employed and the results (Myriad Research 2017). This means that at school level the teachers did not know what to do after they received the comic books. The UNICEF (2016) is the report after the pilot, not before the intervention as a guidance for implementing partner to deliver the programme. There is limited info on the programme component that describes how to implement. Key stakeholders in Bandung, did not demonstrate awareness of the programme documentation. In one school visited for the review, teachers noted that the comic books had been delivered, but that they did not know how they should use them. UNICEF suggested that the schools selected by local partners for the review had not been involved in the intervention, but were follow-up schools involved in the scale-up. This acknowledgement of receipt of the comic books was the first step towards further planned inputs to support teachers to deliver the educational messages.

Score 0/1

0

The UNICEF programme staff had a clear and comprehensive overview of what they intend to achieve and how they intend to achieve it. Many materials are available, including the comic books, a Handbook on Menstruation for Moslem Females (Parents) and an MHM Guideline for Teachers and Parents, various MHM infographics for a social media campaign, a Menstruation is Normal video, and a policy brief and basic advocacy kit for decision makers and senior government officers. However, this knowledge is retained by a small group and could be lost if programme staff leave. To guard against interruptions in programme implementation, in the case of loss of institutional memory, the review team recommend development of a specific document that puts together the current materials, describes the programme components and how to administer it. This will contribute to scale-up, within Indonesia and regionally. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: Government representatives, parents and teachers all accept and express appreciation for the programme’s activities and consider the © Oxford Policy Management Ltd, November 2017

Score 0/1 1

74


programme essential to improving knowledge and practice amongst adolescents. The programme was designed with government (UKS) for national scale-up. The involvement of the Council of Islamic Scholars has contributed to the MHM programming becoming accepted practice Detailed findings: Stakeholders accept and appreciated the knowledge provided by the programme, and that key messages were delivered to both parents and students. Teachers perception of the MHM comic is also positively rated (Myriad Research 2017). Some stakeholders reported that they had initial concerns regarding the appropriateness of the intervention for primary school children, however, over time they have come to accept that the programme is relevant for younger age groups highly relevant for older (adolescent) students. They also like the presentation of comic books that can be read easily by students. The involvement of the Council of Islamic Scholars in developing the materials has contributed to the MHM programming becoming accepted practice. 5. Monitoring and evaluation: Is there a monitoring and evaluation plan and evidence of its execution? Headline: A baseline survey was conducted to assess levels of preintervention knowledge. An end-line survey was conducted to assess the effectiveness of comic books after three months of its introduction (UNICEF (2016) and Myriad Research (2017). Detailed findings: Baseline and endline data is available for the pilot programme. The baseline and end-line collected data around MHM knowledge, attitude and practice, including data on school attendance and health indicators. The end-line also collected data to evaluate the effect of comic books after distribution in schools. The end-line highlights that the main sources of knowledge about menstruation for girls are from their mother, teachers, and sister. There is a significant increase in knowledge about menstruation between baseline and end-line. For example, 96 per cent of girls perceive that menstruation is a normal process in the end-line, compared to 77 per cent at baseline (Myriad Research 2017).

Score 0/1

1

It is recommended that a detailed M&E plan be developed for the planned scale-up phase. 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: To address misconceptions around menstruation, the programme worked with Indonesia Council of Islamic Scholars and prepared a handbook on “Menstrual and Health according to Islamic Teachings”. The endline data suggests that this has contributed to the dispelling of myths surrounding menstruation (Myriad Research 2017). This endline also provides data on the experience of children with disabilities in MHM. Detailed findings: To address misconceptions around menstruation, the programme worked with Indonesia Council of Islamic Scholars and © Oxford Policy Management Ltd, November 2017

Score 0/1

75


prepared a handbook on “Menstrual Health according to Islamic Teachings”. The endline data suggests that this has contributed to the dispelling of myths surrounding menstruation (Myriad Research 2017). This endline also provides data on the experience of children with disabilities in MHM.

1

In Bandung, discussions with teachers, parents, and those on the school management committee, the review found that there was limited knowledge about the MHM programme, the comic books, and the MUI handbook. Teachers who were trained by the programme were aware of the programme’s details, but this information had not been shared widely with other teachers or parents or the school committee. While the programme had solicited feedback from teachers, they said that they did not receive information about how their feedback was used by the programme. However, in principle, they considered the programme to be a much-needed intervention in school to increase knowledge for both girls and boys alike. The MHM education curriculum was designed and delivered to both boys and girls and to children with disabilities in one third of schools. Myriad Research (2017) report that most teachers in the 22 schools surveyed include the MHM content as part of the core curriculum, and that almost half of all teachers in the intervention schools had received training. The same report notes that MHM education was provided to children with disabilities in more than one quarter of the intervention schools. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The programme targets school-going girls and boys at grade 5 and 6. It also included children with disabilities in one third of intervention schools. Although it is reported that scale-up plans will include out of school children in a scale-up phase there is currently insufficient evidence on their involvement. Detailed findings: The pilot programme was designed to be implemented in schools, and as a result did not reach out of school adolescents. UNICEF report that out of school children have been incorporated into proposals for funding the next phase of MHM in 2018.

Score 0/1

0

Myriad 2017 report that MHM education for students with disabilities was provided in 29 per cent of intervention schools. UNICEF also report the inclusion of equity considerations when developing the tools; during the test phase the requirement for an additional tool for children with lower literacy levels was identified and addressed. Key informant implementing partners, reported to the review team that MHM materials are still mostly given to female students since the programme is in the early stages. These same respondents did not consider male students as priority beneficiaries for the programme, because sharing materials on menstruation are in some cases considered a taboo for boys. However, Myriad Research (2017) note that both, “girl’s


76


and boy’s knowledge of menstruation improved significantly after the intervention”. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: The conclusions of the end-line survey are generally positive. The programme improved students’ awareness and knowledge, it changed their attitudes to become more positive about menstruation, it improved girls’ hygiene practices and reduced the number of days they were out of school during to menstruation. Some of schools now have also improved WASH facilities, e.g. provision of sanitary pads and improved toilet and clean water. Detailed findings: Key findings from the end-line research are generally positive. Students now have improved knowledge on menstruation compared to the baseline; for example, 89 per cent of girls knew how to use re-usable pads in the end-line compared to only 75 per cent of girls in the baseline (Myriad Research 2017). UNICEF reports that, “adequate knowledge of menstruation increased by 16 per cent amongst girls and by 28 per cent amongst boys”, and that, “Utilising innovative communication tools increased the percentage of boys who felt it was wrong to bully menstruating girls by 34 per cent”, (UNICEF WASH Flyer, undated).

Score 0/1

1

Myriad Research (2017) report that boys’ attitudes towards girls who are menstruating had improved. This supports UNICEF’s findings on a noted reduction in school bullying (UNICEF WASH Flyer, undated). Teachers interviewed as part of this review did not know about the impact in terms of student knowledge around menstruation or about the programme monitoring system or any evaluation about the programme. UNICEF have worked with government to update the Ministry of Health UKS (national school health programme) guidelines to include MHM and are working to included MHM messaging in the reproductive health guidance. It is suggested to investigate the maintenance of learning and behaviour change post-intervention with the initial cohort. 9. Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: MHM was designed as a national programme with government partners, and an initial pilot/test phase. As noted above national level changes have been implemented to school health programme guidelines. Locally, government partners in Bandung were aware of the plans for sustainability and scale-up, although they expressed interest in expanding the MHM programme to other schools. Because of the early stage of implementation there is insufficient information to assess sustainability and replicability. Detailed findings: In Bandung, implementing partners reported anxiety that the government would not be able to sustain the programme due to limited interest and because they did not have enough money. They believed local © Oxford Policy Management Ltd, November 2017

Score 0/1

0 77


government budget (APBD) would not be able to cover the costs of the programme or be willing to take on additional allocations towards MHM. Government informants although interested in continuation of the programme, agreed that although they could reproduce the MHM products (comic books), they were not sure that they have the resources to send teams to schools to train teachers. At the school level, teachers agreed that even though the MHM programme was important, they could not continue the programme independently, as they didn’t have the funding to do so. MHM was designed as a national programme with government partners, and has completed an initial pilot/test phase. National level changes have been implemented to school health programme guidelines, which will support scale-up. One of the implementing partners in Banding indicated to the review team that they have found it difficult to solicit cooperation with the government because MHM is not a priority for local government. Whilst UNICEF report plans for scale-up in partnership with UKS to 34 provinces, this review found that the sustainability of the intervention was not guaranteed. Further support for implementation in the pilot areas may be required. The funding for scale-up in 2018 has not been sourced. 10. *Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: Adolescents were involved in the programme design during programme preparation, through the HCD approach to developing the programme intervention, and in the development of the materials (comic books). The early results indicate that the improved knowledge has contributed to reduction in bullying which in turn contributes to empowerment, especially for girls. Detailed findings: While documents received from UNICEF clearly state that the design of the programme and the finalisation of the comic books was based on adolescents’ experiences about menstruation and that the comic books were finalised based on their input before implementation began, the responses of stakeholders interviewed in Bandung on the degree of involvement were varied. This may indicate that the stakeholders interviewed were not part of the design cohort, rather than suggest children were not involved in the design.

Score 0/1

1

Implementing partners say that while adolescents were involved in delivering MHM materials, they were not involved in the programme planning and design. Teachers in one school visited by the review team said that programme implementers sought feedback on the comic books from students and teachers. Teachers in another school said that students did not participate in the design of the project. A third said that while teachers did collect feedback from students on behalf of the implementing team, they did not hear back on what came of the feedback. UNICEF report that materials were pre-tested in different schools to those involved in the baseline and endline surveys to reduce potential for bias and contamination of results. © Oxford Policy Management Ltd, November 2017

78


For ethical reasons this review did not include adolescents under 14 and therefore this information has not been verified with beneficiaries. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: UNICEF report that the comic books have gone through several iterations. Similarly, the delivery mechanisms are evolving, from specific teaching through the curriculum to a more integrated approach. Detailed findings: Changes made to the design of comic books were made based on input received during the trial phase. For example, adjustments were made to the profile picture of the children, to make the children look older, and an additional comic book was developed, to meet the needs of adolescents with lower literacy levels. There were also changes in the comic book content for urban (with more text) and rural (with less texts/narrative). UNICEF (2016) reports that, “the formative research indicated that menstruation has been taught during science, religious affairs, and physical education however the new 2013 curriculum stipulates that all schools adopt an integrated approach for all subjects. With this new movement, the allocation of the MHM comic book will not be able to be taught as part of one individual subject but instead teachers will be encouraged to determine at what stage they will be able to integrate the MHM comic book into their lesson plan”.

Score 0/1

1

Since the programme is only just emerging from the test phase, further changes are anticipated based on the findings of the endline research. Summary Findings The MHM programme is an EMERGING practice. It fully meets seven of the 11 criteria and is on track to become a promising practice for Indonesia and the region. To facilitate this, it is recommended that the programme design and implementation mechanism and resources are consolidated into an MHM manual which can be used by programmers who may not have the in-depth institutional knowledge. The conclusions of the end-line survey are generally positive. The programme improved students’ awareness and knowledge about menstruation, it changed their attitudes to become more positive about menstruation, it improved girls’ hygiene practices and reduced the number of days they were out of school during to menstruation, and contributed to a reduction in school bullying. Whilst the programme has been designed with and for government and components of the programme support scale-up, local anxieties in Bandung suggest there may be a requirement for further support from UNICEF to ensure the programme becomes fully embedded. The programme has been well received by stakeholders and endline data suggests positive results. It is suggested to investigate the maintenance of learning and behaviour change postintervention with the initial cohort. In view of the programmes acknowledged protective factors, the reported reduction in school bullying, it is suggested that UNICEF consider how this can be integrated with other schoolbullying reduction interventions. Similarly, there may be opportunities for linkages to health interventions, or expansion of the methodology through existing programmes targeting marginalised and vulnerable populations. © Oxford Policy Management Ltd, November 2017

79


It is understood that a detailed strategy for scale-up is in process. The inclusion of clearly articulated goal and outcomes, together with indicators and targets will contribute to the database for measuring achievement of the SDGs. # and details of KII and FGD conducted # FGD/KII description participants KII-1 1 GIZ field staff District Health Office KII-2 1 staff KII-3 1 Puskesmas staff KII-4 1 Head of TP-UKS FGD-1

8

FGD-2

4

FGD-3 FGD-4

8 15

FGD-5

6

FGD-6

9

FGD-7

8

District government Parents and school committee Teachers Teachers Parents and school committee Teachers Parents and school committee

M

F

1 1 1 1 2

6 4

2

6

1

5

4

5

References: Myriad Research (2017) End-line Survey on Menstrual Hygiene Management (MHM) Interventions in Schools in Bandung: Key Findings. Presentation. UNICEF (2016) WASH in Schools in Indonesia: Incredible Opportunities, An Overview of the Current Situation with Recommendation for Progress. Jakarta: UNICEF Indonesia UNICEF and INNOVESIA (2015) UNICEF Indonesia Innovation Human-Centered Design Activity for (Menstrual Hygiene Management) Concept Development


80


A.3

LOLIPOP In-depth Case Study

Name of the programme 5 x domains of the UNICEF Adolescent Country Tracker (ACT) Principle domain of ACT which applies to the programme Subsidiary domains of ACT which apply to the programme Short description of the programme

LOLIPOP - The Linkages Quality Care of Young Key Population Health and wellbeing √


Protectio n

Transitio n to Work


√ The LOLIPOP (Linkages of Quality Care for Young Key Populations) programme in Bandung was implemented as a pilot by UNICEF Indonesia in 2015 to increase detection of HIV amongst adolescents and youth aged 15-24 years by promoting HIV testing among young key populations and increasing access to treatment and adherence to decrease new infections. The programme focused on five young key populations (YKPs): males who have sex with males (MSM), female sex workers (FSW), transgender (TG), injecting drug users (IDU) and young people living with HIV, however UNICEF reports that “since the context has a lot to do with access to testing the engagement leans heavily to the first four”, (written communication, October 2017). The programme worked with various stakeholders and utilised existing initiatives and networks in Bandung. There are 4 strategic interventions under the programme: 1) Increasing demand for HIV testing and treatment that includes creative use of prevention and promotional materials, using the peer educator methodology and providing better outreach to YKP; 2) Improving supply by creating adolescent friendly services at community health clinics; 3) Creating an enabling environment by providing supportive laws, policies and regulation; and 4) Generating strategic information with better data to inform decision-making, planning and monitoring. * applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion


Emerging Practice

Promising Practice

Promising Practices Criteria 1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: LOLIPOP addresses a well-documented need for better services for adolescents with HIV. UNICEF (2016) report that, “Between 2011 and © Oxford Policy Management Ltd, November 2017

*Score 0/1

81


2015, new reported HIV infections in Indonesia increased annually by 13.1 per cent among adolescents (aged 15 to 19)”. In addition, a baseline study conducted in Bandung in 2015 confirmed that concerns around the lack of privacy and the prevalence of stigma were key deterrents to young key populations (YKP) accessing healthcare. The baseline data was collected in mid-2015, after the programme was initiated and it is not clear how this was used to inform programme design. UNICEF (2017, pg. 11) explains that it used the UNICEF MoRES framework (Monitoring Results for Equity Systems) in formulating the LOLIPOP programme, in which equity for young people in key populations requires good access to prevention, testing and treatment services. This reflects programme compliance with the UNCRC. Detailed findings: LOLIPOP responds specifically to the IBBS 2011 data from the Ministry of Health that reports adolescents and youth aged 15-24 were at risk of contracting HIV while they were still at young age. IBBS 2011 data findings highlight that “young people are experimenting with sexuality but are not equipped with adequate knowledge to protect themselves or others from HIV”, (LOLIPOP programme document 2015, no author, page 2).

0

In mid-2015, a baseline for Bandung was prepared jointly by Burnett Institute and University of Padjajaran (UNPAD) for UNICEF. The baseline consisted of a survey of 466 YKPs, health service staff (4 staff per service point), 9 Puskesmas, 2 private hospitals, 2 public hospitals, 1 NGO-based clinic, in-depth interviews of 10 YKPs, FGD with YKPs and secondary data collection. The study found that the main barriers to HIV testing for YKP mostly related to concerns regarding confidentiality and privacy, disclosure of HIV-positive status and stigma and discrimination. UNICEF (2017) reports that, “the baseline study has been informed back to the stakeholders in Bandung to further develop LOLIPOP”, but no details are provided on consequent changes or improvements to programme design. Primary data collection with government respondents at district level responsible for HIV prevention and adolescents in Bandung revealed a perception of high prevalence of HIV cases in adolescents. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline:The overall goal of LOLIPOP programme as described by UNICEF (2017) is “to increase testing, treatment, adherence and decrease new infections for YKP”. The objectives are primarily output related and outcomes for YKP relative to the overall goal are not clearly described. Detailed findings: The programme design (i) promotes behaviour change amongst YKPs to encourage HIV testing (increase demand), and (ii) at the same time prepares the health service points to provide YKP-friendly services (improve supply). It aims to (iii) create an enabling environment through legislative change, social change initiatives and improved The overall goal of LOLIPOP programme as described by UNICEF (2017) is “to increase testing, treatment, adherence and decrease new infections for YKP”. The objectives are primarily output related and outcomes for YKP


Score 0/1

0

82


relative to the overall goal are not clearly describedsocial protection initiatives, and to (iv) generate strategic information. The planned activities, indicators and targets related to these four specific objectives, or how these are linked to the overall goal, were not evident in the programme documentation provided. The programme started in early 2015 (some respondents said late 2014) with a stakeholder meeting (health office, 13 health service points, local NGOs/CSOs) to discuss issues and the needs of adolescents. One key informant described how LOLIPOP is different from other HIV and AIDS programmes, (e.g. Global Fund) in that it does not have a target number of YKP who take a HIV test, “We only measure the increase access of adolescent to health services…,” (KII, Bandung). Although it was not clear how this would be done. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: Several LOLIPOP programme documents detailing its components and strategic interventions as the key approaches for programme implementation, are described by respondents as available at a central level. However, primary data collection found that stakeholders interviewed did not always know of all the documents developed by LOLIPOP. Documents are reported as available but based on responses during KII and FGD the dissemination appears to have been problematic. Detailed findings: This review found that respondents could reference the documents related to the programme components and how to administer it. Respondents were familiar with specific documents such as the baseline study, training modules, brochures and the LOLIPOP website. Based on this, stakeholders described their understanding of their roles in the programme during interviews, • UNPAD said that they were asked to prepare the baseline, end-line and other evaluation materials • Local NGOs/CSOs/activists’ roles were to perform outreach to identify adolescents, act as their peer counsellor/educator and refer/take these adolescents to a health clinic. In addition to using an outreach worker to actively reach YKPs, the programme also provides a membership card with a point system every time an adolescent accesses health service • Local government, led by District Health Office and KPA (AIDS Commission), coordinate stakeholders for improving supply (health services) and creating an enabling environment, including the required regulations. Implementing agencies had access to the baseline survey results leaflets that provided brief details of the programme. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: Stakeholders including government health departments, and community members understand the importance of providing health services to adolescents with HIV. However, the programme was not entirely successful in changing the attitudes of health staff and community members who knew of and interacted with the programme, owing to the culturally sensitive nature of the programme content. Concern was expressed around


Score 0/1

1

Score 0/1

0

83


the messages in communication materials which was perceived as encouraging sexual behaviour. Detailed findings: The programme was supported by the mayor who attended the launch of LOLIPOP in 2015, and the health dinas who supported this programme through approval to use 13 puskesmas (health clinics) to promote provision of YKP-friendly services to adolescents seeking HIV tests. Since health providers and NGOs participating in this programme had all worked on providing HIV services and preventative activities and programmes in the past, LOLIPOP was not introducing new practices – for example an initiative called Warga Peduli AIDS (a community care AIDS program) had been working the in the district to reduce stigma and discrimination since 2011. However, the focus on adolescents introduced new elements to existing health practice. The programme was not entirely successful in changing cultural perceptions around HIV, which continued to hinder adolescent access to healthcare. For example, adolescents and a representative from a district health office spoke of the differential treatment counselling staff at Puskesmas mete out to young people, which dissuades them from seeking care. Community members also reported challenges to acceptance of the programme’s message – which is viewed to be encouraging a negative lifestyle amongst adolescents. The programme’s messaging, such as the headline on the LOLIPOP website that says “saya bangga menjadi….” (I am proud to be…), continues to be viewed unfavourably by representatives within the KPA. 5. Monitoring and evaluation: Is there a monitoring and evaluation plan and evidence of its execution? Headline: There was a clear plan for baseline (2015) and endline data collection (reported as planned for end 2017) to be managed by UNPAD and Burnet Institute. There was also a plan for quarterly meeting for all stakeholders to meet and evaluate programme implementation during 2015. Government and NGO confirmed attendance at these meetings, however meeting minutes were not available. Therefore, is has not been possible to verify form and content of the meetings has not been verified. Whilst available documentation describes the goal and objectives, the review has not seen a work plan/activity plan or details of indicators and targets. Detailed findings: Based on information from UNPAD, the baseline for LOLIPOP was conducted in August 2015 and a mid-term review conducted in December 2015. The mid-term review was not available to this review. Stakeholders report that the baseline was conducted post programme implementation, due to administrative issues with contracting. This review is unaware of how the baseline was used to identify indicators and targets for intended programme outcomes.

Score 0/1

0

Key informants report that quarterly stakeholder meetings were planned and conducted during which the progress of the programme was reviewed. These meetings also contributed to the development of training modules for the outreach worker, health worker and peer educator. Stakeholders interviewed for this review told researchers that said that there was no documentation detailing the discussion at these meetings. KPA provided one report and presentation of the M&E meeting in December 2016.


84


Stakeholders also reported that coordination between all parties was an issue. Even though they have regular quarterly coordination meeting, they did not prepare joint documentation/report on the progress of activities. So, it seems that, as one respondent said, “we implemented activities separately and we don’t know what others are doing.” 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: To increase their involvement in the programme, training was conducted for Peer Educators (PE) to learn how to approach YKPs and invite them to check their health situation at the Puskesmas. The staff at Puskesmas also received training on how to provide adolescent friendly services. Some key informants pointed out the sensitivity of HIV issues in relation to religious belief. However, there was no information available to understand if the LOLIPOP methodology involves consultation with religious leaders. Detailed findings: Some key informants pointed out the sensitivity of HIV issues in relation to religious belief. They noted that from the perspective of religious leaders, adolescents should be, “pulled away from the environments that harm their future”. However, there was no information available to understand if the LOLIPOP methodology involves consultation with religious leaders.

Score 0/1

Health workers noted that their responsibility is to provide health related advice to YKPs. They acknowledge that religious advice is somehow needed, but they do not consider it part of their role. The programme did not incorporate into its implementation, existing structures such as Warga Peduli AIDS (a community care AIDS program) that had been working the in the district to reduce stigma and discrimination since 2011. Developing links with existing community structures in the community is likely to increase the acceptance of the programme.

0

UNICEF (2017) highlights that there are still many challenges for YKP work, including: •

the need to involve broader stakeholders to have detail discussion about YKP so that YKP could be a concern and be part of important indicator in HIV prevention activities; • the need for a different strategy to YKP group. Even though they are the key populations that can / have been reached, the way they were being approached must be different because of the youth issues/needs; and • the Discussions on YKP not only done by LOLIPOP partners (e.g.: MoH, UNICEF, NAC, Fokus Muda) but should also be done among other stakeholders, this is due to cross issue that also needs to be discussed, and then strategy development for the future.” 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: LOLIPOP's main target groups - males who have sex with males (MSM), female sex workers (FSW), transgender (TG), and injecting drug users (IDU) are vulnerable and marginalised. Since they are hard to reach, © Oxford Policy Management Ltd, November 2017

Score 0/1

0

85


the programme used PE and outreach workers to identify the population. Stakeholders reported difficulties in reaching adolescents aged 15-19. The review team faced similar difficulties in meeting with this group during field work (only able to meet with 3 adolescents aged 15-19). Whilst the programme targets a marginalised population there was no evidence of inclusiveness, e.g. involvement of YKP with disabilities or from ethnic minorities. Detailed findings: To identify this hard to reach population, the programme used PE who were formerly part of YKPs and who have a good understanding of how to take care of themselves and avoid HIV risks. The PE were trained by the programme on effective communication, including how to persuade YKPs to access health service and take the HIV test. KPA (Bandung Aids Commission) notes that the programme should target all adolescents who are sexually active, not only YKP. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? how do programme participants perceive the benefits of the programme? Headline: An increase in HIV test take-up amongst YKP 2014-2015 is reported. It is not known if this data reflects targets. No information has been made available on increased treatment, adherence, and decrease in new HIV infections. UNICEF (2017) has reported on five key outputs. Detailed findings: UNICEF (2017) reports that facility based data was analysed to compare number of tests taken before (2014) and after (2015) LOLIPOP was introduced. The analysis suggested an increase in HIV testing among young key population in Bandung, from 141 to 257 among FSW, 146 to 176 among MSM, 7 to 11 among TG, and 14 to 25 among PWID. There are no reports on the programme goals of increased treatment up-take, adherence or decrease in HIV infection amongst YKP. UNICEF reports key outputs of the programme as, 1. Baseline data is available on YKP in Bandung, including practices and policies affecting access 2. A digital platform to increase demand is operational (most visitors are 18-24 years old) 3. YKP Leadership Training module delivered to peer educators and outreach workers 4. Development and distribution of 1,200 copies of Information Education and Communication (IEC) materials 5. Training of 36 health workers on YKP-friendly services 6. Development of a comprehensive model framework (although this was not evident in the 2017 report) 7. Coordination and advocacy 8. Expansion of training to three additional cities

Score 0/1

0

The three adolescent respondents to this review said that they benefited from increased knowledge of HIV risks and how to stay healthy. The health staff said the programme increased confidence amongst YKP to access health services. Several key informants noted that they had new knowledge and skills because of the training. Government respondents report that the programme drew the health departments attention to the requirement for outreach but noted that healthcare workers are overburdened, and taking out time to attend to YKP © Oxford Policy Management Ltd, November 2017

86


adolescents’ may not be a priority. In addition, while the programme has been implemented with the Health Office through the Health Service Sector (Puskesmas) and the P2PL Sector, two key offices - Promkes Section (Health Promotion) and Family Health and Nutrition Section are not involved, which is may affect continued efficient delivery of the programme. 9. Sustainable and replicable: Did the programme put in place plans for the programme to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: The programme was piloted in Bandung – where there was favourable support from government and several implementing agencies that had been working in this area and were willing to partner with UNICEF. This pilot took place over one year, and it key informants reported that 2016 was a ‘vacuum’, with minimal activity taking place. It is reported that the training modules have also been delivered in three additional cities (UNICEF 2017). Key informants reported that a local regulation to formalise HIV programming with a focus on adolescents has been introduced. Detailed findings: LOLIPOP has received significant support from the government in Bandung, which suggests a strong possibility of its structures being sustainable. The review found that the AIDS commission and district health office are committed to provide budget to support LOLIPOP activities in the future. However as noted above, the government also points to the concerns over health staff workload and capacity to offer YKP-friendly services. Adolescents who participated in the programme have reported increased awareness of positive health behaviours and were keen to continue its activities in the future. However, based on information from respondents, limited activity occurred during 2016 due to lack of leadership.

Score 0/1

0

Bandung seems to have several new initiatives, • The district health office and KPA report that Bandung is preparing a local regulation to formalise the implementation of HIV and AIDS prevention with a specific focus on adolescent. This will be issued through a Mayoral decree and include a strategic plan. • Bandung is also preparing a mobile counselling service, called KEKASIH (Kendaran Konseling Silih Asih)13, cars with psychiatrist and counselling officer which will tour the city of Bandung for ease of access. They report that this service will be offered to adults and adolescents. • The MoH together with UNICEF has prepared a concept note for adolescent specific targets to the Global Fund New Funding Model (GFNFM). • Puzzle Club through Fokus Muda has received GF-NFM funding from end 2016 and continues its activities during 2017. Despite the above initiatives and learning from lack of facilitation in 2016, the programme still requires a leadership who can bring together related parties, i.e. the government and civil society (two parties who play a key role in the program), to work together for the successful implementation of the programme. 10. Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery?

13

Score 0/1

Literally translated: vehicle caring counselling. The Mayor of Bandung has just launched this service last week. © Oxford Policy Management Ltd, November 2017

87


Detailed findings: In preparing detail activities of the programme, a consultation meeting with adolescent was held in July 2015 (based on KII and data from the LOLIPOP Facebook page). A mid-term review of the programme conducted in December 2015 notes that “Planning exercises, involving many project partners, commenced in December 2014 with a series of Inception Workshops in Bandung with local young key populations and local health staff” (Burnett Institute 2016). The mid-term review itself included a ‘Participatory Informal Discussion with Adolescents from Key Populations (10 participants)”, (Ibid).

1

The LOLIPOP logo and IEC (Information, Education and Communication) materials, such as brochures and leaflets, were developed in consultation with YKP, and up-dated to reflect YKP comments and concerns. The target population is YKP in the 15-24-year age group; the current data does not give an indication of adolescent specific involvement (15-19). UNICEF (2017) report, for example that most of the visitors to the digital platform are over 18. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: UNICEF (2017) report that the comprehensive model framework for YKP- friendly HIV programming, “is a results-based framework, meaning that the higher-level goals and objectives can remain in place, while the exact activities to achieve each objective can change according to changes in technologies, funding, and potential partner interest”. The programme was designed to be flexible and to adapt to local conditions, and there is evidence to some extent to support this. Stakeholders mentioned that they met regularly to discuss issues/challenges and how to address them, including if they need to change the strategy. Detailed findings: The minutes of the December 2016 stakeholder meeting include a discussion on alternative strategies to address issues in the field. One challenge that is still being discussed by respondent is how to address the age consent policy that is limiting access for YKPs aged below 18 years old to take an HIV test.

Score 0/1

1

At the service point, Puskesmas Garuda created an innovative approach by providing a special counselling room and decorate it specifically so that adolescents can feel comfortable when they want to discuss their issue. UNICEF (2017) records that “By end of 2015 the ICT strategy used by LOLIPOP was evaluated, suggesting specific recommendations for improvement. The result of the evaluation was discussed and a workshop was conducted to adjust the ICT platform following the evaluation result. UNICEF (2017) also report that, “A revised strategy is currently being designed with YKP CSO in Bandung leading the initiative.” This revised strategy was not available to this review. This issue was also highlighted during FGDs, that there were reflective questions (conveyed by


88


NGOs/activists) on whether the communication strategy was effective enough, such as LOLIPOP card and gifts as well as online consultation. Whilst the flexibility described in the programme design is welcomed, it is difficult to understand how it is a ‘programme which can be sustained and scaled-up’ based only on a goal and set of objectives. Summary conclusion The LOLIPOP programme is an UNDETERMINED PRACTICE. It fully meets three of the 11 criteria. The goal of LOLIPOP programme was to increase testing, treatment, adherence and decrease new HIV infections for YKPs. It used Kota Bandung as its demonstration site to pilot the approach. Bandung is selected because the city already has stakeholders/institutions that have been working in this area. There is baseline data but it is not clear how this informed programme design. The programme goal is ambitious, but this review did not find any evidence of indicators, targets or description of activities intended to achieve them. Since there are several stakeholders involved in programming with the YKP cohort in Bandung, attribution of results is difficult. Data on increased testing, treatment, adherence and decrease in new HIV infections was not available to this review. UNICEF (2017) report that the comprehensive model framework for YKP-friendly HIV programming, “is a results-based framework, meaning that the higher-level goals and objectives can remain in place, while the exact activities to achieve each objective can change according to changes in technologies, funding, and potential partner interest”. Whilst the flexibility described in the programme design is welcomed, it is difficult to understand how a programme can be sustained and scaled-up based only on a goal and set of objectives. The focus of the programme is YKP aged 15-24, however stakeholders reported difficulties in reaching adolescents aged 15-19 during programme implementation. The review team faced similar difficulties in meeting with this group during field work (only able to meet with 3 adolescents aged 15-19). Most of the beneficiaries appear to be in the cohort over 19 up to age 30. The review team could not identify what specific activities were included to explicitly target adolescents. Quantitative data on involvement of beneficiaries in the adolescent age group was not available to the review team. In view of the strong evidence in support of investments in adolescents, it is recommended that a focus is placed on this age group in future programmes, with specific attention paid to needs and how to meet them. This is particularly important with YKP since “adolescence is a time when gender roles are consolidated, challenged and transformed”, (UNICEF 2017 a). Baseline was followed by a mid-term review within five months, this was not available to the review team and therefore there is insufficient information to make an assessment on progress or recommendations for changes to the activities. The endline data collection is planned for end of 2017. It is recommended that this undertakes a rigorous approach to identifying positive outcomes for YKP with attention to up-take of testing, treatment, adherence and decrease in HIV infection.


89


The programme does not seem to address the cultural sensitivitities, for example by sufficient consultation with religious leaders The LOLIPOP is an undetermined practice. It is not yet clear if it decreases new HIV infections amongst YKP. However, the fact it only ran for one year, and the noted absence of indicators and targets for intended outcomes limits results reporting. # and details of KII and FGD conducted FGD/KII # participants description M F Media and Publication KII-1 1 Coordinator, Puzzle 1 club KII-2 1 Head of KPA Bandung 0 1 KII-3 1 UNPAD 0 1 Head of Health Prevention and KII-4 1 0 1 Curative, District Health Office Garuda Health Clinic KII-5 1 0 1 Staff FGD-1 &2 9 (aged (2 FGDs with 16, 18, same 9 Adolescent YKPs 0 19, 20, respondent 22, 26) using different instrument) FGD-3 5 Program Staff 1 4 FGD-4 & 5 (2 FGDs with same 3 YKP: Transgender 0 3 (aged 22, 24, 33) respondent using different instrument) FGD-6 6 Community 2 4 References: Directorate of Direct Communicable Disease Control and Prevention, MoH RI, Jan 2017 UNICEF (2017) All In to End Adolescent AIDS in Indonesia; UNICEF Report (draft) to MAC AIDS Foundation. Jakarta: UNICEF Indonesia UNICEF (2017a) Working with and for Adolescents. UNICEF Programme Guidance for the Second Decade of a Child’s Life. UNICEF: Ney York UNICEF (2015) Support for Closing the Prevention and Treatment Gaps for Young Key Populations in Indonesia. Jakarta: UNICEF Indonesia Burnet Institute, UNPAD and UNICEF (2015) Demonstration Site of Friendly HIV Services for Young Key Population in Bandung: Baseline Evaluation. Jakarta: UNICEF Indonesia Burnet Institute (2016) Demonstration site of Friendly HIV Services: LOLIPOP Mid Term Review Report. Based on notes of workshop prepared by UNPAD.


90


LOLIPOP Facebook page (https://www.facebook.com/The-HIV-Lolipop-Programme-Indonesia1594913080773041/) Minutes of Monitoring & Evaluation Meeting (December 2016)


91


A.4

IFA Light-touch Case Study


Micronutrient Initiative - Iron Folic Acid (IFA) supplementation programme for school going adolescents’ girls in selected districts of West Java Province in Indonesia. Health and Education wellbeing and learning

Protection

Transition to Participation and work engagement

√

√

√

In 2011 WHO recommended that for iron folic acid (IFA) supplementation to be a useful strategy to prevent anaemia among adolescent girls, programmes should be complemented by a well-informed and effective Behavioural Change and Communication (BCC) strategy. In 2016 Micronutrient Initiative (MI) with support from UNICEF started a programme to provide weekly IFA supplementation to adolescent girls in schools in West Java and to develop a BCC strategy intended to promote dosing, adherence and coverage of IFA supplementation (Agustina and Prafiantini 2016 a; Agustina and Prafiantini 2016 b). The main activities implemented by the programme are the provision and distribution of IFA supplementation tablets to girls in school age (12 to 19 years old), the capacity building of health personnel and school teachers to provide information, advice and support to the adolescents involved in the programme. * applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion

Good Idea – Undetermined Practice

Emerging Practice

Promising Practice

Promising Practices Criteria 1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: The design of the IFA programme is based on the 2011 WHO recommendations for IFA and informed by two studies conducted in 2016. A qualitative formative research study carried out in two programme intervention districts and a quantitative baseline survey conducted in the same two districts, plus Bandung as a comparison district. The baseline survey’s findings show worse results in anaemia prevalence, intake rates, availability and coverage of IFA supplements in the intervention areas Detailed findings: The IFA programme demonstrates evidence-based programming design. © Oxford Policy Management Ltd, November 2017

**Score 0/1

92


In 2016, the Micronutrient Initiative together with the Directorate of Research and Community Services of the University of Indonesia conducted two main studies: A qualitative formative research study conducted in two programme intervention districts (Puwakarta and Cimahi) and a quantitative baseline survey conducted in the same two districts, plus Bandung as a comparison district.

1

The findings of the baseline survey provide information regarding the prevalence of anaemia, habitual intake, availability and coverage of IFA supplementation. Significantly, the results in the intervention area were consistently worse than in the comparison area, although no analysis or discussion on this disparity has been provided to-date. The study found that the prevalence of anaemia among the adolescent girls aged 12-19 in the intervention area was 52 per cent, which was significantly higher than the prevalence in the comparison area at 38 per cent. (Agustina and Prafiantini 2016 a & b). Additionally, both studies informed the design of a Behavioural Change Communications (BCC) strategy to improve awareness regarding dosage, adherence, and benefits of IFA supplements, and estimate the coverage of weekly IFA supplementation among school going adolescent girls. Furthermore, the programme is rooted in evidence produced by other studies conducted in West Java province reporting high levels of anaemia among adolescent girls (i.e. the Indonesian Family Life Survey 2008 and the Basic Health Research 2013). 2. A Theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes?

Score 0/1

Headline: Programme staff described the results chain from activity to result. No programme documentation was available to support this. Detailed findings: The goal of the IFA programme is to support realisation of a child’s right to adequate nutrition. More concretely, the programme focuses on school going adolescents. While anaemia affects half a billion women of reproductive age and 38 per cent of pregnant women worldwide14, adolescent girls are at special risk of being affected. According to key informants, one of the objectives of the programme is to “prepare healthy girls who eventually will be a mother one day, if the girls have issues of anaemia, it will affect the health of the child later”.

0

According to key informants, MI started its activities in 2016 with an advocacy campaign conducted at the province level and using materials developed by the Ministry of Health. Advocacy efforts were extended to district level in a multi-sectoral effort to involve representatives from the health and education sectors as well as the religious affairs regional office. Consequently, IM started the distribution of IFA supplementation tables among school-going adolescent’s girls. Additionally, information and awareness sessions and trainings for teachers and health officers at 14

https://www.nutritionintl.org/right-start/ © Oxford Policy Management Ltd, November 2017

93


Puskesmas to improve their understanding of anaemia and IFA supplementation and to be more confident in supporting and advising adolescents’ girls were conducted. These information and awareness raising seasons fed into the No document that explicitly outlines the causal pathways for input, output and outcome was provided to this review, however this was well described by key informants. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: The programme has developed a comprehensive curriculum to train educators, teachers and medical staff at Puskesmas. The training curriculum and modules provide a clear description of the programme components and its administration. Detailed findings: In accordance with WHO 2011 recommendations, anemia prevention efforts for girls are focused on promotion and prevention activities, including the provision of IFA tables to school age adolescents. The Ministry of Health has strengthened this efforts by issuing a Circular letter15 regarding Guidelines for the provision of IFA tables to young women (translated from Bahasa)

Score 0/1

1

In order to support this guidance issued by the MoH, it is expected that educators and health personnel in contact with adolescents would increase their understanding and capacity to provide advice and support to the girls receiving the tables. For this purpose, MI together with the KKRI has developed a comprehensive and detailed curriculum to train educators, teachers as well as medical staff at Puskesmas that are involved in the provision of AIF supplements to adolescent girls in schools. The training module and curriculum provide clear explanation on the components of the programme and its administration. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents?

Score 0/1

Headline: Based on findings from the formative research, adolescent girls, teachers, parents and service providers had erroneous conceptions of the use and value of using IFA supplementation to prevent anaemia among adolescent girls. Some respondents also thought the channels used to reach adolescent were not appropriate and instead the use of social media as opposed to more traditional channels was encouraged. Detailed findings: One respondent noted that the IFA programme collaborates and has good acceptance from the following stakeholders including: 1 • • •

Religious leaders including Madrasah officials, Teachers and other education personnel in the schools where adolescents are enrolled Sanitary personnel in the Puskesmas.

The 2016 formative evaluation highlights the perceptions of adolescent girls, teachers and other key informants (service providers, parents etc.) towards the IFA supplementation programme. Based on the main observations 15

Circular Letter HK Number. 03. 03 / V / 0595/2016 © Oxford Policy Management Ltd, November 2017

94


outlined below, most of the respondents didn’t fully understand the purpose of the supplementation and/or its appropriateness for adolescents and even had some erroneous conceptions. •

• •

Adolescent girls: none of the informants mentioned IFA supplementation as a mean to prevent anaemia, but instead consider it as a treatment for girls that are already anaemic. Furthermore, some girls reported to understand IFA supplementation as negative for their health. Teachers: similarly, most of the teachers interviewed during the evaluation, said they considered IFA as a treatment for anaemia rather than a preventive measure. Other informants: most of the service providers (midwifes and cadres) said that IFA supplementation could prevent anaemia symptoms, however, most parents reported to be afraid of its side effects.

5. Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: This review did not identify any explicit monitoring and evaluation plan. However, baseline data is available and there is a plan to collect postintervention comparison data. Detailed findings: The Circular letter HK Number. 03. 03 / V / 0595/2016 regarding the guidelines for provision of Iron Folic Acid tables to young women is reported to include an M&E plan.

0

However, the review did not come across any explicit monitoring and evaluation plan or evidence of its implementation for this specific programme. Furthermore, key informants did not report that they were aware of its existence. The collection of qualitative and quantitative data at baseline suggests there may be a plan to collect post intervention comparison data. This was explicitly referenced by one key informant who remarked that a follow-up study is planned for March 2018. 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: The IFA programme is inclusive of all school-going adolescent girls. There is not enough information to determine whether it is inclusive of girls with disabilities or from specific religious or ethnic backgrounds. Detailed findings: KII respondents noted that the programme is inclusive of all adolescent girls regardless of religion or ethnicity. It is not clear whether the programme has special provisions to include girls with psychical or intellectual disabilities. One of the key informants reported that IFA supplement is not given to boys. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations?

Score 0/1

0

Score 0/1

Headline: The review did not come across any explicit mechanism to reach the poorest and most marginalised adolescents. © Oxford Policy Management Ltd, November 2017

95


Detailed findings: The review did not come across any explicit mechanism to reach the poorest and most marginalised adolescents. Furthermore, since the programme is targeted exclusively to girls in school, it is likely that it does not cover some of the poorest girls which very often are not enrolled in formal education. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: Most the evidence collected through the baseline study suggest the outcomes and/or impact of the IFA supplementation programme could have been better. Some results suggest that majority of girls in implementation districts did not receive the supplementation and that knowledge and understanding regarding dosage, duration, and benefits of the intervention were limited. Detailed findings: The programme has been subjected to a set of qualitative research and a quantitative baseline study. According to the results of this study, only 19 out of 340 adolescents’ girls in both areas (treatment and control) received IFA supplementation. This means that most adolescent’s girls had not received any IFA treatment.

0

Score 0/1

0

The same study suggests that there is a need for increased knowledge and understanding about IFA dosage, duration, benefits as well as other associated relevant information about anaemia. Additionally, some erroneous perceptions of the side effects of IFA supplementation were reported, i.e.: “it could affect the visceral organs because it’s chemical substances”. One key informant reported that there is a plan to conduct an end-line evaluation at the end of the programme conducted by IM (March 2018). 9. Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: The mechanism for distribution of the IFA supplementation is in place and regulated by the government, furthermore, the government through the MoH is funding the initiative. Detailed findings: In line with WHO 2011 recommendations, anaemia prevention efforts for girls are focused on promotion and prevention activities, namely; increased consumption of iron-rich foods, supplementation of IFA tables and increased fortified food consumption.

Score 0/1

1

Anaemia prevention and prevention programmes for girls are priorities for the Ministry of Health. There is also currently a nutrition development programme that is included in the Ministry of Health Strategic Plan 20152019. School institutions are the main target of this nutrition coaching programme. In collaboration with the Health Office and Puskesmas, the school is the gateway for prevention of anaemia in adolescent girls aged 12-18 who are in high school. Furthermore, KII respondents corroborated that this programme is already part of the government target on community nutrition both at provincial and district levels.


96


In 2016, the purchase and distribution of IFA tables was funded by the Indonesian government, through the MoH. In 2017, district administration is also taking part. 10. Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: The programme design was informed by qualitative evidence collected from adolescent girls including their understanding of the causes and consequences of anaemia, their perception towards IFA supplementation and their opinion on the channels and appropriateness of the information delivered. Detailed findings: The programme design has been informed by qualitative evidence collected from adolescent girls, teachers, midwives, cadres, parents and other relevant stakeholders. Adolescents girls provided information on a range of issues as per their understanding of symptoms and consequences of anaemia, their perception towards IFA supplementation and other animal source foods, and what are the most effective and accessible channels of information. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: The programme is still in its early stages of implementation (pilot phase) and therefore there is insufficient evidence on its capacity to adapt and adjust to challenges. Detailed findings: The programme is still in its early stages of implementation (pilot phase) and therefore there is insufficient evidence on its capacity to adapt and adjust to challenges.

Score 0/1

1

Score 0/1

0

Summary Findings The Anaemia prevention in school-going adolescent girls through IFA supplementation programme is an EMERGING PRACTICE. It fully meets five of the 11 criteria. The programme responds to government priorities. The Ministry of Health in Indonesia follows the WHO recommendations to provide IFA supplementation to adolescent girls to prevent anaemia. Anaemia prevention and prevention programmes for girls are priorities for the Ministry of Health. There is also currently a nutrition development programme that is included in the Ministry of Health Strategic Plan 2015-2019. The programme is informed by findings of formative research and a quantitative baseline study (Agustina and Prafiantini 2016 a and Agustina and Prafiantini 2016 b). These highlighted successes and challenges, specifically around incremental coverage, and modification of adolescent, parents and other service providers’ perceptions on the purpose, use, dosage and intended outcomes of the IFA supplements. This contributed to the design of the BCC strategy. Further investigation is warranted into this behaviour change component which was not sufficiently elaborated. There may be more adolescent-friendly approaches to provision of information on anaemia prevention to adolescent girls including social media as well as other more traditional channels like TV and consultation with health workers during visits to health facilities. The programme is implemented through government service providers in in hospitals and Puskesmas and in schools. The programme appears to be sustainable as the mechanism for


97


distribution of the IFA supplementation is in place and regulated by the government, furthermore, the government through the MoH is funding the initiative. Whilst this review could find evidence of formal programme documents such as a well-defined theory of change and/or a monitoring and evaluation system, the programme appears to have well-articulated links between activities and outcomes. It has furthermore developed a curriculum, to train educators, teachers and medical staff. # and details of KII and FGD conducted FGD/KII # participants description M F IFA supplementation extender in KII 1 0 1 Purkwakarta District IFA supplementation extender for KII 1 1 0 West Java Province. References: Barkley JS, Kendrick KL, Codling K, Muslimatun S, Pachón H. Anaemia prevalence over time in Indonesia: estimates from the 1997, 2000, and 2008 Indonesia Family Life Surveys. Asia Pac J Clin Nutr. 2015;24(3):452-5. Sammon, E.M., Martinez, M., Jagmag, M. and Wahyudi, R. (2017) Document Review of Promising Practices in Indonesia by UNICEF and other partners. Jakarta: UNICEF Indonesia WHO (2011). Weekly Iron and Folic Acid Supplementation Programmers for Women of reproductive age. Agustina, R. and Prafiantini, E. (2016 b). Formative research for the improved Iron Folic Acid supplementation programme for school going adolescent girls in selected districts of West Java province in Indonesia. Univeritas Indonesia and Micronutrient Initiative: Jakarta Basic Health Research (RISKESDAS 2013)


98


A.5

Skills to Succeed Light-touch Case Study Skills to Succeed




Protection

Transition to work


√ √

Since 2012 Save the Children’s (StC) Skill to Succeed programme (STS) aims to empower adolescents and young people aged 16-24 in Bandung through the acquisition of skills and competencies to pursue work aspirations. STS is an extension of the youth-based education programme which integrates skills education, entrepreneurship and work placements to meet the transition to work needs of both in-school and out-of-school adolescents. The intended outcomes of the programme are; 1) key capabilities are strengthened, 2) availability and access to decent economic opportunities increased, and 3) Enabling environments fostered. The specific strategies for the delivery of the programme outcomes are, 1. Assessing labor market conditions and needs, 2. Inform and support the training of adolescents, 3. Build strong working relationships, and, 4. Advocate for long-term change. The STS programme engages with a range of government and nongovernment district level partners, including the multi-stakeholders networks, local NGOs, teachers association and the private sector etc. to open up opportunities for internships and sustainable employment of adolescents and youth. ** applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion

Good Idea – Undetermined Criteria

Emerging Practice

Promising Practice

Promising Practices Criteria 1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? © Oxford Policy Management Ltd, November 2017

*Score 0/1

99


Headline: The project design is informed by the findings of an evaluation conducted on a previous programme implemented by StC to reduce exploitive child labour through education and economic empowerment (EXCEED). It is also guided by the StC’s global work on adolescent’s transition to work, including the ‘Adolescents Skills for Successful Transitions’ position paper. Detailed findings: The project design is based on findings and knowledge gathered from a previous programme implemented by Save the Children between 2009 and 2013 called EXCEED “Exploitive Child Labour through Education and Economic Development”, which aimed to prevent children and adolescents in several provinces in Indonesia (West Java, Lampung, Yogyakarta, East Java and West Kalimantan) from becoming involved in the worst forms of child labour, including plantation work, child sex work, domestic work etc. StC reported that an internal evaluation, of the EXCEED programme (not made available to this review team) showed that the economic development component of the programme had not been sufficiently addressed, and its goals remained unachieved. Based on this, StC created a new programme, Skill to Succeed (STS) to enhance the economic development element and strengthen future employment outcomes for adolescents in Indonesia. Key informants reported that the STS programme is rooted in the “Adolescents Skills for Successful Transitions” position paper which presents StC global expertise in working with adolescents and youth on livelihoods and Education for Youth Empowerment (EYE). This paper describes the foundations for working with adolescents to increase their capabilities (i.e. skills, attitudes, knowledge) and opportunities for obtaining decent and productive work. The paper focusses on gender-sensitive and inclusive programming. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: Whilst this review has not found a document outlining the programme’s theory of change, or explicit results chain/programme logic, key respondents report that they are aware of programme documents guiding the implementation of activities to meet the main goal of facilitating adolescent transition to work. Detailed findings: Key Informants from StC staff reported that, while they have internal documents such as work plans and activity schedules, they are not aware of the Theory of Change. Respondents are however aware of the main purpose of the programme, which is to help adolescents and young people (16 to 24 years old) to be better prepared to transition into the workforce. They also described some of the activities intended to achieve this.

1

Score 0/1

1

STS partners with local NGOs for five main activities, • • • •

Identification of training institutions, Enrolment of young people (aged 16-24) for skill training. There are two options available: skills to become an employee, or to become an entrepreneur, Provision of counselling services, Provision of life skills services to facilitate entering the workforce


100


•

Facilitation of coordination meetings with stakeholders, including government and private sector employers to assess the programme progress.

3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: The STS has produced an informative brochure outlining the main purposes and components of the programme as well as training modules to increase adolescent skills. Detailed findings: The background and approach to the implementation of the STS activities are detailed in a 2015 report, “There Is Still Another Way; Lessons Learned Documentation of Skills To Succeed Programme” (Ertanto 2015) The programme has produced an informative brochure in Bahasa outlining the main purpose and components of the programme to be distributed to adolescents, teachers and other relevant audiences. Some other available programme documents are the training modules used to improve adolescents’ skills on career guidance, financial literacy, employability skills and entrepreneurship. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: The STS programme has established good collaboration with district government officials, the local teachers’ forum, local NGOs and the private sector. There is insufficient information to make an assessment of the involvement and acceptance by community members, families and guardians and adolescents themselves. Detailed findings: One respondent noted that the STS programme collaborates with several stakeholders including:

Score 0/1

1

Score 0/1

0

•

Government officials at the district level, the offices of Education and Manpower. • Three selected local NGOs partners, these are: KAP (Anti-poverty consortium) Indonesia and Bahtera Foundation in Bandung and APIK in Lampung. • The private sectors through the creation of a memorandum of understanding to provide youth apprenticeships or support entrepreneurial training. i.e.: the NHI Bandung hospitality college • The teachers’ forum. 5. Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: The review of the STS programme did not come across any explicit monitoring and evaluation plan or evidence of its implementation. Detailed findings: Respondents were not able to provide an answer on whether an M&E plan exists. Moreover, the review did not come across any explicit M&E plan or evidence of its implementation. 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: The programme is inclusive of male and female adolescent. There is, however, not enough information regarding adolescent with disabilities or from different ethnic or other marginalised groups. © Oxford Policy Management Ltd, November 2017

Score 0/1

0

Score 0/1

0

101


Detailed findings: The STS programme targets young people and adolescents (age 16 to 24) including those that are not enrolled in the formal education system. Reaching this group poses some methodological challenges that the programme seeks to overcome through partnerships with local community groups and organisations. The programme is inclusive of male and female adolescents. There is not enough information to discern whether the programme is addressing specific needs of children with disabilities and/or those belonging to different ethnic and religious groups. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The project target beneficiaries are youth at risk of being engaged in exploitative labour, including those out of school and those living in poor households. Sex disaggregated data was not available, nor other data indicating inclusiveness. Detailed findings: The project target beneficiaries are youth at risk of being engaged in exploitative labour, sibling of child labour, out of school youth and youth come from poorest community (Programme Skills to Succeed PowerPoint). Furthermore, it relies on local partners (KAP and Bahtera in Bandung) to identify and reach adolescents in poor households. Key informants report that it specifically focuses on providing support to adolescents during the transition from primary school to junior high school. They also note that this group has been shown to be at a higher risk of abandoning education and entering the labour market via precarious jobs with potential detrimental consequences for life. The STC programme aims to provide a “second chance for disadvantaged youth to try to return to their precious work or education” (KII StC Bandung). 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: In 2015 lessons learned were documented highlighting stories of success. The report states that in addition to the educational and employment results, the building of self-confidence among adolescent was also perceived as a positive outcome. Detailed findings: StC has collected and published two collections of success stories of STS beneficiaries, i.e.: “New Shirt from Shy Girls”, a collection of success stories of Skill to Succeed participants from Lampung and “Inspirational Stories of Skill to Succeed; The story of the 25-young people's journey to success”. These provide descriptions of how the programme has changed the lives of some of its beneficiaries producing long-term positive outcomes.

Score 0/1

0

Score 0/1

1

In Bandung, StC report that some adolescents said that after finishing high school they wanted to work in fashion, a business which characterises the tourism industry in the area, however they were few opportunities or information about how to pursue their ambitions. Through the STS programme they were supported through internships to gain experience and have now found jobs locally (Eartanto 2015) Key informants noted that in addition to the educational and employment outcomes, the STS programme also helps young participants to build their self-confidence and hopes for a better future. © Oxford Policy Management Ltd, November 2017

102


There is no indication of other external rigorous evaluation being conducted for the STS programme. 9. Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: Whilst the programme has implemented several measures to ensure sustainability, such the engagement of an interdisciplinary working group, it is still highly reliant on external funding. Detailed findings: One of the stated objectives/intended outcomes of the STS programme is to foster long-term change in the lives of young people. To achieve sustainability, the programme has established a regular crosssectoral technical working group consisting of StC, private sector, local government, training providers and NGOs. This inter-disciplinary platform also aims to mobilise additional public and private funds to reach a greater number of adolescents through engagement with vocational schools, training and work placements.

Score 0/1

0

Respondents believe sustainability is also strengthened by the wellstablished relations with the teachers’ forum at district level. Commitment to sustainability of programming with out of school children is more challenging due to the difficulty on finding suitable partnering institutions. The programme is currently dependent on external financial sources provided by StC and other donors and there is no indication on whether there are plans for the programme to be managed by the government and/or replicated elsewhere. 10. Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: Adolescents were consulted for the evaluation of the previous EXCEED programme which highlighted the weakness of the economic development component. This finding served to improve the design of the current STS programme. Detailed findings: Whilst there was not a specific direct involvement of adolescents in the design of the STS programme, the current programme was designed based on specific findings and evaluation of the previous EXCEED (Exploitative Child Labour through Education and Economic Development). Adolescents consulted during the evaluation highlighted the weakness of the economic development component. This finding served to inform the development of the current programme and therefore to incorporate the voices of adolescents in its design. Additionally, the programme is designed to encourage recipient’s feedback on the training modules and skills being delivered. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: A certain degree of flexibility can be seen in the programme, there is, however, insufficient information to provide a definitive assessment. Detailed findings: As noted, the STS programme was developed based on findings of a previous programme evaluation, which suggests an element of flexibility. However, there was insufficient information available on this programme to make a definitive assessment.

Score 0/1

1

Score 0/1

0

Summary conclusion © Oxford Policy Management Ltd, November 2017

103


The STS programme is an EMERGING PRACTICE. Based on the available information it fully meets five of the 11 criteria. It may benefit from a more in-depth review against the promising practice criteria. The design of the STS programme is informed by findings of a previous programme evaluation in which adolescents claimed the economic development component was not sufficiently dressed. StC report that it has been reaching poor adolescents in and out of school and has documented case studies of positive outcomes. It also has served as a platform for a range of stakeholders to collaborate by bringing together private and public sectors, government, and businesses to find decent and quality job opportunities and hopes for the future of adolescents. The programme Theory of Change, M&E mechanism, other relevant programme documentation and defined plan for future sustainability or/and replicability was not available to this review. The STS programme is an important intervention serving a population of adolescents at a critical juncture in their transition from childhood to adulthood. # and details of KII and FGD conducted # participants description M F FGD/KII KII 1 ASST Specialist 1 0 KII 1 STS Project Officer 0 1 References: Save the Children (no date) Adolescents Skills for Successful Transitions position paper Bambang Ertanto (2015) Masih Ada Jalan Lain: Dokumentasi Hikmah Programme Skill to Succeed (There is Still Another Way: Lesson Learned Documentation of Skill to Succeed Programme) Programme Skills to Succeed PowerPoint.


104


A.6

Bandung Street Children Programme (BSCP) Light-touch Case Study


Bandung Street Children Programme (BSCP) Health and wellbeing


Protection

Transition to work


√

√

√

The Bandung Street Children Programme (BSCP) is run by the Perkumpulan Sanggar Sahabat Anak (Friendly Place for Children Association) which was established in 2014. BSCP estimate that some 3,000 to 11,000 children live in the streets of Bandung, national figures go as high as 5 million boys and girls1. Most of these children and adolescents are at great risk of exploitation, sexual abuse, trafficking, substance abuse etc. Girls are especially at risk of sexual exploitation, unwanted pregnancies, HIV infection and psychological distress which can have longterm consequences for their health and development. The BSCP programme focuses on providing a safe space for adolescent girls living on the street under inadequate or no parental or adult care. Through the “Art in a Box” activities, programme outreach workers can gather and engage children living in the streets. This activity provides a space to create and develop artistic talent, as well as to deliver important health and protection messages. It also helps to identify girls in need for further basic services and/or to be provided with the opportunity of living in the “home for girls” where they are under the supervision of responsible adults, receive formal education, regular health check-ups, and support to transition into work and adulthood. 1 http://bandungstreetchildren.org/background.html ** applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion;


Emerging Practice


Promising Practice

105


Promising Practices Criteria

1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: There is insufficient information to provide an assessment of this criterion. Detailed findings: The BSCP is intended to provide protection, education, and care for girls living in the streets of Bandung reflecting international practices on the work with street children. The BSCP reference national figures on number of street children on their website, however respondents noted they are not aware of any baseline or background study informing the design and development of the programme. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: Programme staff describe the goal of the programme as, “to break the life cycle of adolescent girls living in the streets” by providing them with education, care, support and shelter. Outreach workers identify children though implementation of “Art in a Box” activities. Detailed findings: Respondents reported that there is not a detailed theory of change for the programme.

*Score 0/1

0

Score 0/1

However, one BSCP staff member interviewed demonstrated awareness and understanding of the main goal of the programme which is to “break the life cycle of children living in the streets by providing them with education, shelter and the necessary facilities to break the cycle” Through the implementation of “Art in a Box” activities, the BSCP outreach workers would present the materials and invite children to join the activities. The “Art in a Box” methodology provides a good opportunity for children to create and to develop artistic skills as well as to express emotions. It is also a good entry point to identify children in need of specific support. According to the BSCP website “These events (Art in a Box) are also a way to get to know more about the issues the street children are facing. What medical, social or psychological support do they need? Additionally, the events a provide a unique opportunity for outreach workers to convey useful information regarding health, nutrition, hygiene, sexually transmitted diseases, pregnancy and drug use among others.

0

Girls living on the streets are at higher risk of experiencing abuse, exploitation and/or violence. Some of them are referred to live in the Home for Girls which is the only facility in Bandung that is exclusively designed to host girls living on the streets and without (appropriate) adult care and supervision. Additionally, BSCP also supports parents by providing small amounts of cash intended to run small business. Staff reports this is not always easy as some parents lack the necessary entrepreneurial skills. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: There is no evidence of formal programme documentation, the programme has a detailed web-page and issues a newsletter.


Score 0/1

0 106


Detailed findings: There is no information regarding any formal programme documentation, other than a newsletter that is issued every 6 months and the information provided on the web page of the programme. In the web page, there is mention of training materials used to capacitate outreach workers, these are developed and implemented by SOS Children’s Village. However, their existence is a recognition of the requirement for standardisation by the BSCP programme. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: The BSCP programme has good collaboration with sub-district government and providers of basic services such as health and certification of births. Detailed findings: Key Informants report that the BSCP Programme enjoys good collaboration with government at the sub-district level, and that the girls in the programme are provided with some basic services such accessing free healthcare through the local Puskesmas, obtaining birth registration certificates, accessing local schools etc. There is insufficient information as to how the programme is regarded by community members, parents and guardians and adolescents themselves. 5.Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: The review of the BSCP did not come across any explicit monitoring and evaluation plan or evidence of its implementation. Detailed findings: The review did not come across any explicit monitoring and evaluation plan or evidence of its implementation. Furthermore, key informants did not report that they were aware of its existence. The BSCP website describes the governance mechanism including the Executive Board, Financial Control Commission and Annual Member Meeting. 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: The programme through its component of "home for girls” aims to be inclusive of all adolescent girls regardless of religion or ethnicity. There is not enough information regarding adolescent with disabilities. Detailed findings: The programme through its component of "home for girls” is inclusive of all adolescent girls regardless of religion or ethnicity. It is not clear whether the programme has special provisions to include girls with psychical or intellectual disabilities. One of the key informants reported that "Art in a Box" also includes boys. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The BSCP focuses on reaching girls most at risk and in need of support, including those at risk of serious child protection violations. Detailed findings: The BSCP focuses on reaching girls most at risk and in need of support, including those at risk of serious child protection violations. This review has not, however, being able to collect enough information to assess whether the programme is inclusive of children with disabilities. The programme reports that it identifies girls living on the streets through a team of outreach workers who make an initial assessment of the child situation and circumstances. Understanding the importance of adequate parental care, © Oxford Policy Management Ltd, November 2017

Score 0/1

0

Score 0/1

0

Score 0/1

0

Score 0/1

0

107


and restoring family life, efforts are also directed to identifying parents/caregivers to present the programme, get their verbal consent and to work with them towards social and economic improvements. One of the respondents stated, “When we help the children, the social and economic side of the parents needs also to be supported, because when the children want to get back to the parents, if the parents are not ready, then the children will go back to the street”. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: The review did not come across any explicit formal independent study demonstrating positive outcomes and/or impact other than anecdotal individual stories. Detailed findings: The review did not come across any explicit formal independent study demonstrating positive outcomes and/or impact. However, there are some examples of individual positive outcomes for some of the girls that have lived in the house. 9. Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: There is not enough information to discern whether the programme has sustainability or replicability plans in place. The programme is a primarily donor driven service provider filling a gap in statutory service provision. Detailed findings: There is not enough information to discern whether the programme has sustainability or replicability plans in place. Respondents noted that funds are currently being sought from various private organizations such the Global Grant of Rotary, as well as from public sources “district grant”. 10. Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: The review did not come across any explicit engagement of adolescent on the design of activities. Detailed findings: See above. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: The BSCP is a donor driven programme which is required to respond flexibly on a regular basis, and which reports flexibility and adaptability in responding to the needs of adolescent girls. Detailed findings: The programme staff report flexibility in responding to adolescent-specific needs. However, there was insufficient information to make a full assessment of this criterion.

Score 0/1

0

Score 0/1

0

Score 0/1

0

0

Summary conclusion The BSCP programme is an UNDETERMINED PRACTICE. It does not fuly meet any of the 11 criteria. This is in part due to the limited availability of documentary evidence. The BSCP programme is a donor supported service which aims to fill the gap in statutory service provision for vulnerable children and adolescents living on the streets in a specific location. The programme operates in a complex environment and reports having positive impacts on the life of © Oxford Policy Management Ltd, November 2017

108


adolescent girls who are living and working on the street. The girls admitted to the “Home for girls” are offered protection and care and provided with basic health and education services as well as opportunities for development, growth, and transition to work. Programme staff report having re-located to a less central environment to prevent girls resuming their life on the street. This poses an ethical and programmatic challenge for BSCP. However, the organisation appears to be responding positively by recognising the value of a family strengthening approach. Although the programme reaches a great number of children living in the streets of Bandung through the “Art in a Box” component. the home for girl does not have the capacity to admit all. The programme appears focus on admitting the poorest and most marginalised girls, the the home for girls. There is little evidence of the programmes evidence-based design, theory of change and/or a monitoring and evaluation plan against which to measure results. There is also no information on the inclusion of adolescents in the design or implementation phases. The programme does not appear sustainability or replicable but demonstrates clear intention to provide the best care they can to beneficiaries with limited resources. The BSCP references its local affiliations, compliance with local legislation and advocacy with local government for continued financial stability. This review was unable to meet adolescent girls involved in the programme, however it recommended that this cohort is explicitly included in future research on adolescent wellbeing. Similarly, BSCP may benefit from a full review of its monitoring and evaluation systems, and explicitly those which involve M&E of its child protection policy. # and details of KII FGD/KII # participants KII

1

description Vice Chairmand of the Perkumpulan Sanggar Sahabat Anak BSCP programme staff member

KII 1 References: http://bandungstreetchildren.org/background.html


M

F

1

0

0

1

109


A.7

Youth Program. Indonesia Planned Parenthood Association (PKBI) Light-touch Case Study

Name of programme

the

5 x domains of the UNICEF Adolescent Country Tracker (ACT) Principle domain of ACT which applies to the programme Subsidiary domains of ACT which apply to the programme Short description of the programme

Youth Program. Indonesia Planned Parenthood Association (PKBI) Health and Education wellbeing and learning

Protection

Transition to work

Participation and engagement √

√

√

The goals of the PKBI national strategy 2010-2020 are i) Developing models and standards of quality sexual and reproductive health services to meet the needs of the community, ii) Empower people to fight for sexual and reproductive rights for themselves and others, iii) Developing STI and HIV / AIDS Prevention and Control Prevention and iv) Advocate at all levels of the organization to decision-makers to ensure the fulfilment of sexual and reproductive rights and health16. The Executive Director of PKBI Aceh reports that their Youth Programme has four components, i) Sexual, Orientation, Gender, Identity, Expression (SOGIE), ii) Youth Centre iii) Young Voice, and iv) Youth Forum. The information available to the research team included the PKBI 2015 Annual Report and Key Informant Interview transcript with the Executive Director of PKBI Aceh. * applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion


Emerging Practice

0Promising Practice

Promising Practices Criteria 1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: The PKBI Aceh Youth Programme is in line with the overall goals of the national organisations strategy and with a human-rights based approach. However, there was insufficient information about specific programmes to make a full assessment of this criterion. Detailed findings: PKBI Indonesia is part of the International Planned Parenthood Federation to deliver sexual and reproductive health services17. PKBI Aceh, A branch of the national organisation described four activities in their ‘youth programme’ which are inclusive of adolescents. However, there was insufficient information to assess the evidence base for local programme design. 16

http://pkbi.or.id/tentang-kami/renstra-2010-2020/

17

http://www.ippf.org/about-us/member-associations/indonesia © Oxford Policy Management Ltd, November 2017

*Score 0/1

0

110


The SOGIE programme was initiated in 2015, to work with lesbian, gay, bisexual and transgender adolescents (LGBT) and provide a safe space for expression in typically conservative communities, and to improve access to HIV testing. This is reported as having emerged from the national ‘Program Peduli’ funded by the Australian Government Department of Foreign Affairs and Trade and led by the Coordinating Ministry for Human Development and Culture to promote social inclusion to reduce poverty among marginalised people in Indonesia. The Youth Centre is a long-running programme which has evolved since it’s initiation in the 1990’s to build youth capacity at community level through basic training and information dissemination on general health, gender issues, substance abuse, HIV, and nutrition. In areas where volunteer interest is high, specially trained counsellors are available in the youth Centres. Young Voice is a partner of PKBI in implementing a peer educator programme for children with disabilities, which has been piloted in two schools. It is reported as involving concern sexual and reproductive health education for children with disabilities and their parents. Youth Forums encourage adolescents to contribute their ideas to local development plans, however this activity appears nascent due to funding issues. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: Intended outcomes are linked to the organisations national strategy. Progress is constrained by funding. There was insufficient information about specific programmes to make a full assessment of this criteria. Detailed findings: The concept of the Youth Forum is described in the 2015 Annual Report, along with a draft guidance for the forum. The SOGIE programme appears to be an offshoot of the ‘Program Peduli Develop an Inclusive Indonesia’ (PKBI 2015 pg. 34), intended to, “promote social inclusion, improvement and economic empowerment by fulfilling access to basic rights services and human rights advocacy for marginalized people”. This programme is reported as having provided support to transgender organisations in Aceh. Achievements are reported across the domains of Access to Public Services, Social Acceptance, and Policy however none are specifically ascribed to PKBI Aceh (PKBI 2015, pg. 38).

Score 0/1

0

Detail on the Youth Centres, Youth Voices and Youth Forums are not available. No theory of change for the PKBI Aceh Youth Programme was defined, intended outcomes and activities to achieve these outcomes were insufficiently described to reach an assessment of this criteria. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: There was insufficient information to assess this criterion. Detailed findings: As above 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: Initially supported by government through the Social Affairs Office in Banda Aceh, the SOGIE programme is reported as struggling with acceptance © Oxford Policy Management Ltd, November 2017

Score 0/1

0 Score 0/1

111


due to local religious conservatism. Youth Centres are a long-standing initiative which operate on volunteer support and can be considered accepted practice locally. Young Voice is a community programme which is reported as having been implemented with the support of teachers, parents and planners. Youth Forums is nascent due to limited donor funding. Detailed findings: Stigma and discrimination affecting LGBT adolescents is exacerbated by the prevailing religious conservatism in the province. PKBI Aceh report that they continue to advocate for acceptance and local funding for the SOGIE programme, and where possible network with local and national human rights organisations. Where possible PKBI conduct routine meetings with parents and families, some of whom have become role-models for sharing their experience of acceptance with other families.

0

The Youth Centre which was initiated in the 1990’s is delivered through voluntary participation and appears to be an embedded and accepted practice locally, due to its long running character. Young Voice has been piloted in two schools and has involved meetings with parents. It’s acceptability by the wider community is not known. Youth Forums are linked to a national programme and are currently established in 7 localities in Aceh. This programme’s development is also reported as subject to prevailing local conservative perspectives. These reports and conclusions, based on a review of the most recent annual report of the national organisation, and the KII with the PKBI Aceh Executive Director, have not been confirmed with local stakeholders or through documentary evidence. 5. Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: There was insufficient information to assess this criterion Detailed findings: As above 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: There is some evidence of cultural competency and partnerships however there is insufficient information to fully assess this criterion. Detailed findings: Youth Voice is reported as targeting adolescents with disabilities. The Youth Centres require that 25 per cent of board members are adolescents. The Executive Director of PKBI reported that this means adolescents engage directly in programme planning, especially the youth Centre programme. PKBI have also encouraged adolescent involvement and youth centre recruitment through the development of a brochure disseminated through its networks.

Score 0/1 0

Score 0/1

0

The SOGIE programme targets extremely vulnerable and marginalised adolescents (LGBT). The philosophy of Youth Forums is one of participation and engagement with adolescents, however the level of involvement of marginalised and vulnerable adolescents in these forums is not known. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents


Score 0/1

112


affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The overall concept of the Youth Programme, evidenced in its several components, is one of social inclusion and participation. Detailed findings: The several components of the PKBI Youth Programme clearly intend to reach vulnerable and marginalised populations. Youth Voice, for children with disabilities, and SOGIE for LGBT adolescents target some of the hardest to reach young people, and are particularly courageous enterprises given the local context. There is insufficient evidence of indicators and targets for a conclusive finding that the intention is achieved. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? how do programme participants perceive the benefits of the programme? Headline: Positive impacts on adolescent lives are reported by the organisations Executive Director although no independent study has been conducted, and this review did not have the capacity to directly contact beneficiaries. Detailed findings: The PKBI Excecutive Director reported changes in the lives of adolescents and their families. For example support to obtain an ID card which helped an adolescent to establish a small business; adolescents are less isolated and more involved socially in their communities; they have improved self-confidence to negotiate relationships; they can conceptualise a positive future for themselves [where previously they could not]; parents are more accepting of LGBT adolescents and are less violent towards them. 9. Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: There was insufficient information to assess this criterion, however unreliable programme funding is reported by PKBI Aceh as affecting development. Detailed findings: Whilst Youth Centres are reported as largely self-sufficient, relying primarily on volunteer involvement, their continued existence will be linked to funding for training and capacity building of future volunteers. It is not known if this is within the scope of local government/community budgets. PKBI Aceh reported that unreliability of external funding was linked to slow programme expansion. 10. Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: The overall philosophy of the Youth Programme is one of participation and engagement leading to the empowerment of adolescent boys and girls. However, there is insufficient information to assess the reality in practice. Detailed findings: Youth Centres require 25% of their board members to be adolescents with 50:50 girl/boy ratio. The Executive Director reported that this means adolescents are engaged in programme planning, especially in the development of the Youth Centre programme. The Youth Centre also has the authority to submit proposals to funding agencies and in 2014 received oneyear funding from the regional government budget (APBA). However, this review did not have the capacity to verify the actual participation of adolescents.

0

Score 0/1

0

Score 0/1

0

Score 0/1

0

Youth Voice is described as a peer education programme on sexual and reproductive health for adolescents with disabilities. Similarly, the ‘peer’ component was not fully reported on, rather the description given suggested


113


that in this current stage the programme relies on work with teachers and parents. SOGIE is intended to provide a ‘safe-space’ for extremely marginalised and vulnerable adolescents. However, it is reported as being subject to considerable restrictions in the conservative context of its operations, which may have implications for the outcome of empowerment. Youth Forums are explicitly intended to empower adolescents and youth, however there was insufficient evidence to assess the results. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: The continued existence of the programmes as reported, in a complex context is evidence of some level of flexibility and innovation. Detailed findings: As above.

Score 0/1

0

Summary discussion: The Youth Programme is an UNDETERMINED PRACTICE. It does not meet fully the 11 criteria. The programme partially meets many of the criteria however there was insufficient information provided to make a full determination. Aspects of the programme suggest there may be potential to move towards promising practice and the progamme may warrant a fuller review against the criteria. The discourse on Aceh takes place within the post-tsunami and post-conflict context and against the background of the enforcement of some provisions of Islamic criminal law. This contributes to complexities in the operating environment for NGOs, particularly those with a rights-based agenda. It is within this context that PKBI Aceh are working to implement sophisticated programming which is intended to improve outcomes for the most vulnerable and marginalised adolescents. Given the limited information available to this light-touch review it is not possible to conclusively report the PKBI Aceh Youth Programme as a Promising Practice in Adolescent Programming. Components of the programme are integrated and accepted by local government and communities. The components which target the most marginalised LGBT adolescents are less accepted. Their continued presence, and courage in maintaining a dialogue on often controversial issues, is acknowledged. Future support on programme implementation should consider the promising practice criteria as the foundation of good programme design. # and details of KII and FGD conducted FGD/KII # participants description M F KII

1

Executive Director PKBI, Aceh

1

0

References: PKBI (2015) Annual Report 2015. PKBI: Jakarta Located at http://pkbi.or.id/publikasi/annual-report/ [accessed 29.09.2017]


114


A.8

Community Youth Justice Light-touch Case Study Community Youth Justice

Name of the programme

5 x domains of the Health and Education Protection Transition Participation UNICEF wellbeing and learning to work and Adolescent engagement Country Tracker (ACT) Principle domain of √ ACT which applies to the programme Subsidiary domains of ACT which apply to the programme Short description of The programme is implemented by the NGO Pusat Kajian Pendidikan the programme dan Masyarakat (PKPM – Study, Education and Community Centre) in collaboration with the Restorative Justice Working Group, supported by UNICEF. It aims to use customary law, which is in line with existing Law on Criminal Justice System for Children, to use the non-formal system, (rather than through police and court system) in handling children in conflict with law. They are assisting villages to (a) to formulate a local regulation (called reusam); and (b) to record the cases and local judgements and sentencing. The information was provided by programme staff during a focus group discussion, however no programme documents were shared. * applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion;


Emerging Practice

Promising Practice

Promising Practices Criteria 1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: The programme is based on local Regulation No. 8/2008 concerning Development of Life and Customary Law, modeled the Justice System Law (No. 11/2012), especially for the article on diversion. There was insufficient further information additional to fully assess this criterion. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: There was insufficient information to make an assessment of this criteria.


*Score 0/1

0 Score 0/1 0

115


3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: The programme is defined within a robust regulatory framework of national policy and legislation and local secondary legislation. Detailed findings: PKPM programme staff reported that together with the community they compile a village regulation, ‘reusam’, on childrens development regarding community justice. This regulation details the community justice mechanism through customary village courts. To date, four village reusams have been signed. PKPM programme staff report the publication of a training module on the handling of children in conflict with the law through the mechanims of customary court.

Score 0/1

1

They have also developed the case record procedure which includes the mechanism for registration and progress of the case. Programme staff also report publication of a book detailing the communitybased mediation process to handle the cases of adolescents in conflict with the law through customary courts. These documents were not provided to the review team and are therefore unveriafiable. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: PKPM report that they have introduced the customary justice mechanism for mediation into four villages, with the cooperation of community leaders. There was no case information available to verify this; PKPM did not describe any successful interventions and the extent to which the community accept the programme is unconfirmed. Detailed findings: Before implementing the programme, PKPM conducted an assesment in 4 villages to obtain consent from the commmunity. Since the issue of child protection is not fully accepted within these communities, PKPM use the customary law system for welfare as the entry point. This involves the village leader and the customary community in the drafting of the resusam concerning the child life and development. Programme staff report that adolescents, especially those belonging to youth community groups in the villages involved in the programme, provided inputs to the development of PKPM capacity building workshops. PKPM delivered a series of capacity building workshop on parenting education, especially for vulnerable families. More than 100 parents are reported to have been involved in these trainings and have proposed that PKPM conduct others workshop for parents. 5. Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: PKPM programme staff report that they monitor the number of registered cases and that these cases have decreased. The reason for the decrease is not explained. PKPM did not provide sufficient information to verify. Detailed findings: As above. 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities?


Score 0/1

0

Score 0/1

0

Score 0/1

116


Headline: The programme is embedded in community leadership and customary law and is reported by programme staff as accepted in 4 villages. There is no information regarding the involvement of marginalised adolescents. Detailed findings: This programme aims for communities to manage cases of children in conflict with the law using mediation based in customary law. The village is involved in drafting the reusam - the village government, community members and the Customary Assembly of Aceh. There is insufficient information to make an assessment of the consideration of specific resuirements of specific groups. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: This programme targets children in conflict with the law. There is insufficient information regarding integrated mechanisms for working with other marginalised groups, or specific differences in prodecures for work with girls and boys. Detailed findings: As above. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: The programme has not be subject to an independent evaluation however the programme staff report that because of community mediation the number of cases referred to the customary court has reduced. Detailed findings: PKPM programme staff report that the number of cases reported to the customary court has decreased through the increased use of community mediation.

0

Score 0/1

0

Score 0/1

The staff suggest that the village apparatus is more competent in handling cases through mediation; before the case is brough to the customary court, the apparatus will mediate the case by meeting with the parent/family of child in conflict with the law, and the victim. If there is agreement, the case did not to bring to the customary court. There is also a reported increase in awareness of child protection issue resulting form the routine meetings conducted in the villages.

0

Adolescents are involved in child forum in the villages and have the capacity to facilitate discussion. Adolescents involved in these forums are reported as having requested that PKPM conduct workshops on a range of child and youth issues. Whilst cases were not previously recorded in the custormary courts, they are now registered. PKPM provided an example of an adolescent who had stolen and was sanctioned by the court to make the ‘adzan’, (Muslim call to prayer) every daya t the required times. This allowed the adolescent to remain in school. PKPM provided this as a sole example, perhaps due to the limitations of their monitoring mechanism. It is also reported that the village government has included child protection in the budget – for continuance of child forums and birth registration. It was not possible for the review team to verify the reported results. © Oxford Policy Management Ltd, November 2017

117


9. Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: PKPM has scaled the programme from four to six villages in 2017. In the initial four villages, the programme for child protection (community mediation and other child protection issues) is included in the village budget. Detailed findings: This program was scaled up from four to six villages in 2017. The initial four villages include the child protection program and budget in the the village budget. However, PKPM report that they continue to advocate for an increase in this budget. PKPM report the importance of involving adolescents in the village child forum and plan to strengthen this component of the programme. PKPM is currently drafting a book/guideline on child protection for Ustadz (a teacher of religion in Islam), the officer of KUA (Office of Religious Affairs), the preachers at the mosque, etc. It is not determined if this is the same publication referenced against criterion 3. 10. Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? Headline: PKPM programme staff report the involvement of adolescents in the village child forum and in the drafting of the reusam. Detailed findings: PKPM programme staff reported that although adolescents were invited to the initial assessment meetings, in some cases they did not attend because the meetings were held in the evenings.

Score 0/1

1

Score 0/1

1 Adolescents were involved in providing inputs to the draft of the reusam during a workshop conducted by PKPM. Adolescents in conflict with the law included in mediation or dealt with by the customary courts are invited to join the child forum. These forums have increased conficence amongst adolescents who are reported more willing to speak out on child protection issues. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: There is insufficient information to assess this criterion. Detailed findings: See above.

Score 0/1 0

Summany conclusion The Strengthening Child Protection (Restorative Justice) progrmme is an UNDETERMINED PRACTICE. It meets three of the 11 criteria. Whilst there is not enough evidence to identify the programme as a promising practice, the findings are of interest and may suggest the emergence of an innovative programme on community justice (and wider child protection). This programme may warrant a more in-depth evaluation. # and details of KII and FGD conducted # FGD/KII description M F participants PKPM programme staff and one village FGD 8 1 1 volunteer


118


A.9

Life Skills Papua Light-touch Case Study


Life Skills Papua



√

√

Protection

Transition to work


√ The current Life Skills Education Programme in Papua is a UNICEF Programme emerging from a life skills programme which ran from 20102014 in West Papua. An end of programme evaluation showed that 91 per cent of schools in the target area provided HIV education to students. Although the programme did not manage to produce significant and sustained changes in the lives of young people and noted that the HIV epidemic among the targeted population group remained unchanged (Burnett Institute and Survey Meter 2014). The significant result currently reported is that the local education department has accepted the life skills programme as a teaching programme with accredited teaching hours, which means that schools can implement within their approved curriculum. In association with a local university this programme is now being rolled out to schools. The light-touch case study is based on a focus group discussion with UNICEF staff and a review of documents related to the initial programme. * applies a score of 1 for YES meets criterion, and 0 for NO does not meet criterion


Emerging Practice

Promising Practice

Promising Practices Criteria 1. Evidence-based programme design: How does programme design comply with international and national obligations and commitments on human rights and gender equality; how is the programme design evidence based? Headline: The evidence base for the redesigned programme is difficult to determine because there is insufficient information regarding the current programme’s intended outcomes, outputs and activities. Detailed findings: Prior to the programme implementation in 2010 Papua and West Papua provinces were the only ones in Indonesia with a generalized HIV epidemic. The Averting New HIV Infection among Young People in Papua and West Papua, Indonesia: Education Sector Response to HIV & AIDS programme was initiated against this background. The 2014 evaluation considered the education sector as a platform was relevant for achieving broad coverage but insufficient to achieve coverage of all young people (i.e. disadvantaged populations - out of school/most © Oxford Policy Management Ltd, November 2017

*Score 0/1

0

119


marginalised). However the HIV epidemic data remained unchanged for the target population, “the epidemic of HIV among young people in both provinces continues unabated”, (Burnett Institute and Survey Meter 2014). The same evaluation found that linkages with health services were inadequate and thus a missed opportunity to increase the effectiveness of HIV education. The evaluation recommended that the programme, if it were to continue, was “best placed within broader Education Capacity Building initiatives, rather than as a stand-alone programme on HIV in the Education Sector”, (Ibid.) Recent data suggests that there has been a more than 60 per cent increase in HIV infection amongst adolescents in Indonesia between 2011 and 2015 (UNICEF analysis of SIHA May 2016). UNICEF has access to significant data on adolescent and young key populations in Indonesia which do not appear to have informed the current Papua Life Skills programme. However, this cannot be determined definitively because of insufficient documented information on the current programme. 2. A theory of change: How does the programme specify intended outcomes and describe the activities that are related to those outcomes? Headline: Staff directly involved in programme implementation were unable to articulate the intended programme outcomes or the intended results chain. The perceptions of intended outcome varied between, ‘reduction in HIV prevalence’, or, ‘helping young people to be more confident’, to, ‘helping teachers to teach life-skills better’. Detailed findings: The original programme from which this current intervention emerged was intended to “contribute to averting new HIV infections amongst young people aged 10-24 years in Papua and West Papua by end 2013”. Outcome and output results are described (Burnett Institute and survey Meter 2014 page 19). No programme results chain for the current programme has been made available. 3. Programme documentation: Is there a book, manual, or other available writings and training materials that specify the components of the programme and describe how to administer it? Headline: There is insufficient information to make a full assessment of this criterion. Programme report that the Life Skills Curriculum is available and currently being delivered through the teacher training institutes. Detailed findings: The documents provided to the review relate to the previous HIV and Life Skills programme which operated during the period 2010-2014. No documentation was shared in connection with the current phase of the programme, thus there is insufficient information to make a full assessment of this criterion. 4. Accepted practice: How do stakeholders demonstrate general acceptance of the programme as appropriate for use with adolescents? Headline: There was insufficient information to score this criteria in this programme phase. However, the 2014 evaluation noted the creation of supportive policy environment. Detailed findings: Programme staff report that pre-service teacher training incorporates HIV related content from the outset, and it is now possible to deliver this part of the curriculum in schools as ‘paid’ teaching hours. Given the content is delivered over a three-year teacher training period, it will take some considerable time to filter into generalised school learning. To address this programme staff, report a plan to provide an in-service capacity building exercise to qualified teachers. No information was available about broader community mobilisation on HIV and AIDS.


Score 0/1

0

Score 0/1

0

Score 0/1

0

120


5. Monitoring and evaluation (M & E): Is there a monitoring and evaluation plan and evidence of its execution? Headline: Whilst the 2010-2014 programme was subject to an evaluation and regular reports to donors, no documentary information on this current programme was made available. Detailed findings: As noted above programme staff were unable to articulate the indented outcomes of the programme, which suggests there is not a clear results chain/logic framework in place which would lend itself to rigorous M&E. 6. Cultural competency and partnerships: How does the programme consider the specific requirements of and involve adolescent boys and girls, adolescents of different religions or ethnic groups, adolescents with disabilities including intellectual disabilities? Headline: There is insufficient information to assess if the recommendations of the 2014 evaluation related to inclusion do not appear to have been addressed. Detailed findings: There was insufficient information to understand what measures were in place to enhance gender considerations in school-based delivery, addressing stigma and discrimination, and developing approaches for out-of-school youth. The inclusion of ethnic minorities and adolescents with disabilities is not evident. 7. Reaches most vulnerable and marginalised: How does the programme identify and reach the poorest, girls, adolescents with disabilities, adolescents affected by HIV; adolescents affected by violence; other marginalized populations? Headline: The 2014 evaluation recommends addressing equity imbalances by focussing on reaching adolescents in more remote and rural locations and by strengthening links between education and health sectors. There is insufficient information to understand if this approach has been adopted. Detailed findings: The 2014 evaluation provides detailed and specific recommendations for future programming in this sector. There is insufficient information available to assess if these have been incorporated into the programme design to focus on reaching the most vulnerable and marginalised. 8. Evidence of positive outcomes and/or impact: Has the programme been subject to an external independent study demonstrating positive outcomes? How do programme participants perceive the benefits of the programme? Headline: The current programme has not been subject to an independent evaluation. As this light touch review does not interview other programme participants, this data cannot be presented here. Detailed findings: The 2014 programme evaluation formulated clear and detailed findings and recommendations. There is insufficient information available to assess if these have been acted on. 9. *Sustainable and replicable: Did the programme put in place plans for the project to be sustainable after the initial phase was complete: this includes finance, policy and delivery structures? Headline: The 2014 evaluation notes that sustainability rests on the incorporation of the life-skills training into the teacher training curriculum. This has been a significant achievement of the programme. There is insufficient information to assess if the recommended supportive measures have been put in place. Detailed findings: The 2014 evaluation addresses sustainability (pages 67-69). However, there is insufficient information, including financial information to know how and if this has been addressed in the current programme phase. 10. *Involved and empowered adolescents: Did the programme create a safe space for adolescents to actively participate and contribute to the programme's plan and delivery? © Oxford Policy Management Ltd, November 2017

Score 0/1

0

Score 0/1

0

Score 0/1

0

Score 0/1

0

Score 0/1

0

Score 0/1

121


Headline: There is insufficient information to make an assessment of the participation of adolescents in programme design or delivery. Detailed findings: There is insufficient information to make an assessment of the participation of adolescents in programme design or delivery. 11. Innovative: Was the programme design flexible when required to change and adapt to new information and challenges? Headline: There is insufficient information to make an assessment of the participation of adolescents in programme design or delivery. Detailed findings: There is insufficient information to make an assessment of the participation of adolescents in programme design or delivery.

0 Score 0/1

0

Summary discussion: The Life-skills Education programme is an UNDETERMINED PRACTICE. It does not fully meet the 11 criteria. This may be in part because of the limited data available to the review team. The assessment of the current Life-skills programme as a promising practice is significantly constrained by the lack of documentation related to the programme. It is not possible to know if the detailed recommendations of the 2014 evaluation have been implemented, and if not why not. For example, due to significant changes in the operating environment. Further, the discussion with the programme staff elicited information which suggests there is confusion at grass-roots level about the intended outcomes of the programme. For example, some referred to it as a programme for HIV prevention, whilst others considered it as a programme for building adolescents confidence. This suggests that results will be limited, and where they are reported cannot necessarily be attributed to this intervention. There is some evidence to suggest that the success of having the life-skills curriculum integrated into the curriculum for teacher training institutes is considered the optimum result. The next phase should link this achievement to the reduction in HIV prevalence in intervention areas (or another agreed outcome). As a starting point the programme design could be reviewed to clarify the overall goal, and to create coherence across the education and health sector components. This will contribute to development of clear indicators and targets. # and details of KII and FGD conducted FGD/KII FGD

# participants 6

description UNICEF programme staff

M 1

F 5

References: Burnett Institute and Survey Meter (2014) End Programme Evaluation. Averting New HIV Infection among Young People in Papua and West Papua, Indonesia: Education Sector Response to HIV & AIDS. UNICEF: Jakarta Ministry of Health and UNICEF (2016) Brief overview: HIV among Adolescent and Young Key Populations (Aged 15-19 and 20-24) in Indonesia. UNICEF: Jakarta UNICEF (2017) Draft PowerPoint Presentation - Policy Dialogue. 14 September 2017. UNICEF: Jakarta


122


Annex B

“How do you know what’s good for me?; “A guide for planning and measuring promising practices in programming for adolescent wellbeing”


123