Monitoring 'monitoring' and evaluating 'evaluation': an ...

Downloaded from jme.bmj.com on October 30, 2012 - Published by group.bmj.com

JME Online First, published on October 30, 2012 as 10.1136/medethics-2012-100680 Public health ethics

PAPER

Monitoring ‘monitoring’ and evaluating ‘evaluation’: an ethical framework for monitoring and evaluation in public health Vijayaprasad Gopichandran, Anil Kumar Indira Krishna School of Public Health, SRM University, Kattankulathur, Tamil Nadu, India Correspondence to Dr Vijayaprasad Gopichandran, School of Public Health, SRM University, Medical College Third Floor, Kattankulathur, Kanchipuram District, Tamil Nadu 603 203, India; [email protected] Received 20 March 2012 Revised 26 September 2012 Accepted 2 October 2012

ABSTRACT Monitoring and evaluation (M&E) is an essential part of public health programmes. Since M&E is the backbone of public health programmes, ethical considerations are important in their conduct. Some of the key ethical considerations are avoiding conflicts of interest, maintaining independence of judgement, maintaining fairness, transparency, full disclosure, privacy and confidentiality, respect, responsibility, accountability, empowerment and sustainability. There are several ethical frameworks in public health, but none focusing on the monitoring and evaluation process. There is a need to institutionalise the ethical review of M&E proposals. A theoretical framework for ethical considerations is proposed in this paper. This proposed theoretical framework can act as the blueprint for building the capacity of ethics committees to review M&E proposals. A case study is discussed in this context. After thorough field testing, this practical and field-based ethical framework can be widely used by donor agencies, M&E teams, institutional review boards and ethics committees.

INTRODUCTION Health improvement is the overall goal of all health systems. Public health interventions aim to achieve preventive, promotional and curative health with universal coverage. Considerable public resources are allotted for these interventions. One of the essential components of management of these public health interventions is monitoring and evaluation (M&E). M&E provides vital information about performance of interventions, which helps in the making of management decisions. It helps programmes to optimise utilisation of resources and directs focus towards achievement of intended goals.1 Whereas monitoring is ongoing close supervision and observation of programme activities to compare ‘what is happening’ with ‘what ought to happen’, evaluation is a process of episodic assessment of achievement against standard criteria.1 M&E starts as a process of monitoring of performance in quantum terms in the initial stages of project implementation and gradually metamorphoses into evaluation of impact of the programme.1 The fact that public health interventions largely depend on good M&E for their smooth performance emphasises the importance of high-quality standards. As well as quality measures, there is also a need to consider ethical issues in M&E. In this article, we discuss J Med Ethics 2012;0:1–5. doi:10.1136/medethics-2012-100680 Copyright Article author (or their employer)

some critical issues in the ethics of M&E and present a comprehensive ethical framework for assessing M&E.

CRITICAL ANALYSIS OF THE ETHICS OF M&E Is M&E research? The two inter-related activities, M&E and surveillance, collect information from populations to help organise public health programmes and policies in an effective manner.1 Can M&E and public health surveillance be described as research in the pure sense of the word? The main difference between M&E and research is the intention of the activity. While research focuses on increasing the body of knowledge in any field, the main aim of M&E is to provide information specific to the functioning of the public health programme. Whereas findings of research may be applied widely, the findings of M&E are locally relevant.2 In M&E, no experimental or non-standard interventions are involved. The information is utilised only for the population from which it is obtained for a specific purpose, and the study is conducted as part of standard operating procedures.2 The International Network of Field Sites with Continuous Demographic Evaluation of Populations and their Health Guidelines give recommendations on ethical conduct of public health surveillance activities.3 The Oklahoma State University Institutional Review Board has suggested guidelines for determining when M&E becomes a research project.4 The following criteria have been laid down to classify M&E as research4: 1. If the main objective of the study is to test a hypothesis 2. If the study will benefit communities other than those on which it is carried out—that is, if it is intended that the findings of the study are generalised 3. If the study is not a routine activity in the project 4. If the data gatherers do not have regular contact with the community 5. If the activity of the study alters the timing or frequency of the standard procedures 6. If the party sponsoring the study is not the party conducting the public health intervention. There is increased emphasis on the need for evidence-based public health policy making. M&E is an important source of evidence-based public health policy making. For any organisation, the results of M&E will be essential for influencing its internal policy.5 The problem of whether M&E is

2012. Produced by BMJ Publishing Group Ltd under licence.

1


Public health ethics research arises when the evidence generated from the M&E is applied widely beyond the scope of the intended project population.5 Irrespective of whether the M&E qualifies as research, there is a definite need for ethical standards in its practice.

Need for an ethics framework for M&E Several codes of conduct, best practices and ethical guidelines for M&E have been proposed by evaluation societies and development agencies.6–12 These are based on sound ethical principles and can be generalised to any public health programme. An academic paper by Smith highlights the important ethical issues to be considered in the practice of programme evaluation.13 In an internet-based survey of evaluation practitioners, the respondents felt that ethical considerations are of utmost importance.14 A survey by the Australasian Evaluation Society revealed that evaluation practitioners came across ethical issues and dealt with them by informal discussion with colleagues and friends, following local evaluation standards, or ignoring the issues.15 Newman and Brown explained an ethical decisionmaking framework for programme evaluation in 1996, which had intuitions, rules, principles, theories and personal values as its components.16 The framework is very simple and gives general directions, but does not assess the various ethical issues in detail. There is a need for continued awareness building and training of evaluation professionals on ethical decision making.17 Berends in her review paper concludes that evaluation societies should continue discussions about ethical issues in M&E, and capacity of research ethics committees should also be built to assess M&E proposals.17 Therefore there is a definite need for an updated, consolidated and comprehensive ethical framework for M&E.

3.

4.

5.

6.

Institutionalisation of ethical review of M&E Evaluation professionals the world over have perceived the importance of institutionalising the process of ethical review of M&E.17 It is our opinion that institutionalisation of the ethical review of M&E will make the M&E process more accountable and responsible. The capacity of institutional review boards and ethics committees should be built to assess ethical performance of M&E. In the following paragraphs, we discuss some key ethical issues in M&E and propose a theoretical ethical framework for practical application. This theoretical framework can serve as the blueprint for the institutionalisation of the process of ethical review of M&E.

7.

Ethical considerations in M&E The ethical issues in M&E can be considered step by step based on the activities in M&E.6–12 1. Avoiding/disclosing conflicts of interest. There is the chance of conflict of interest at various levels in the M&E process. If the M&E is performed internally by programme staff, the financial or intellectual interests of the staff in the programme may be in conflict with the M&E. In the case of external evaluations also, there is a chance of conflicts of interest either in the form of financial stakes with the funding agency or other intellectual interests of the external evaluators. These conflicts of interest can impair the judgement of the person in charge of the M&E. Therefore the results may become unreliable.3 2. Maintaining independence of judgement. Recommendations based on the findings of M&E have to be free of any pressures or external influences. The judgements made by the M&E team must be independent. There may be 2

8.

circumstances where members of the team, funding agency or other stakeholders can influence the judgements of the M&E team. This may be the result of financial, intellectual or other interests. Such influences should be removed or resisted.18 Avoiding bias and being fair. Fairness, impartiality and avoiding bias are very important ethical aspects of M&E. The process of M&E as well as the results should be impartial to any group of the community or other stakeholders. Marginalised communities, women, persons with mental illnesses, persons with disabilities, children, migrants and other vulnerable persons must be given due importance and representation during the M&E process.18 19 Transparency. From the process of engaging stakeholders to planning and setting priorities, all steps must be performed in a transparent manner. Such transparency fosters trust and active participation by the community in the M&E process. Transparency also ensures credibility of the results. As well as transparency of the process, transparency of funding and finances are also important.18 20 Full disclosure. The importance of full disclosure or complete truth telling cannot be overemphasised in the M&E process. Often positive findings that are favourable to the programme are reported, whereas negative results, wrong doings and mishaps are concealed. This can be detrimental to the process and philosophy of M&E. It hampers learning from, and improvement of, the programme.18 19 Privacy and confidentiality. As in any research, M&E involves entering into individual lives and exploring aspects of their participation (or non-participation) in the public health programme, their experiences and their opinions. It often involves gathering private information from participants. Measures should be taken to maintain the privacy and confidentiality of the information collected from the participants. Measures to ensure privacy and confidentiality not only increase the credibility of the process of M&E, but also allow the participants to share their opinions and details freely.18 19 The collected data should be stored with passwords and access available only to those directly involved in data analysis. Respect for individuals. Another important ethical principle in M&E is respect for the agency of the individual participants in the process. Their opinions must be treated with respect. The M&E team should have the attitude of learning from the individual who is participating in the M&E.2 Obtaining informed consent from the respondents for participating in the M&E is essential. The participant should be given enough information about the study that they are fully informed and can opt to participate or not. Another important issue is compensation for the time of the participants. The issue of whether compensation should be provided for participants’ time should be adequately discussed and decided on. Responsibility to community. The M&E team has a responsibility to the community in which they are conducting the process. They have the responsibility to do what is good for the community. During the process of M&E, they may come across problems in the community and/or pitfalls in the operation of the programme. It is the responsibility of the team to report, or take steps to rectify, these problems.18 21 22 The M&E team may often not be directly in contact with the community other than in the process of data collection. In such instances, the M&E team has a responsibility to engage the organisation in doing what is due. J Med Ethics 2012;0:1–5. doi:10.1136/medethics-2012-100680


Public health ethics 9. Empowerment. The M&E process should aim to empower all the stakeholders at all stages. By engaging the stakeholders to freely share the concerns and problems with the programme managers, it can empower them by giving voice to their views. Not only does this make the process of M&E relevant, it also empowers the community members. Ensuring prompt and appropriate action on the results of the M&E can also empower the community.18 23 This can best be achieved through an in-built management-response mechanism within the M&E system. 10. Accountability. The M&E team is answerable to the community, stakeholders and the programme for the results of their evaluation. The concept of accountability is closely related to that of responsibility. The team should take responsibility for the credibility of its findings.18 19 11. Sustainability. In the M&E process, although evaluation is a periodic event, monitoring is an ongoing one. Therefore there is a need to introduce measures to sustain the monitoring process. Engaging the community, empowerment of stakeholders, putting emphasis and importance on the process can help sustain the efforts and participation of the community in the M&E.18 24 12. Hawthorne effect. The Hawthorne effect refers to a change in behaviour of the participants in a study or intervention, not purely by virtue of the intervention but due to the fact that they are being observed. It is a psychological behavioural effect. In many public health programmes, as long as the intervention lasts and close monitoring and supervision is being carried out, good outputs and outcomes are observed. However, once the funding for the programme runs out and the programme is withdrawn, the outcomes do not translate into long-term effects, as behaviours revert to the baseline state. This effect should be kept in mind while interpreting the results of the M&E. Measures should be taken to sustain the efforts over a long period of time.18 25–27 This is where processes gain importance. As mentioned above, if processes are given due importance, this may go a long way to strengthening the sustainability of the project. These ethical considerations have an intrinsic value in the conduct of the M&E. They are different from ethical considerations of public health research in several ways. The immediate goal of a pure public health research project is generation of evidence and knowledge base. Therefore the responsibility of the researcher stops at ensuring beneficence, non-maleficence, justice and respect for individuals. On the other hand, in M&E there is a different ethical demand, which extends to ensuring that the findings translate into actions and interventions in an empowering and sustainable manner. Thus the ethics of M&E extend beyond research ethics. A theoretical framework for ethical assessment of M&E can be derived from these considerations.

Ethical framework for M&E As discussed above, there is a need for an easy-to-use and practical ethical framework for application in M&E of public health programmes. Such an ethical framework is given in table 1. A brief case studyi from an M&E programme in India is

i Owing to the politically sensitive nature of the case study, exact details of the place, time and administrative body responsible are not revealed.

J Med Ethics 2012;0:1–5. doi:10.1136/medethics-2012-100680

presented, and the framework is applied to this programme to assess the ethical nature of this M&E. AI was involved in programme management and M&E of the ‘no scalpel vasectomy’ (NSV) intervention in a highly populated state of north India. As an initiative to promote the acceptance of NSV, which is a relatively easy and permanent method of male sterilisation, the district administrators and health officials were carrying out advocacy, communication, NSV camps and active promotion of NSV at the hospitals. One of the methods used by the district administration to increase the demand for the NSV intervention was to offer incentives such as free land registration for those who accepted NSV if they met other general eligibility criteria. Immediately after this announcement, there was a sudden increase in the number of acceptors of NSV from less than 50 in a year to more than 300. The author was part of the M&E team for this programme. As part of the ongoing monitoring, a monthly report of the number of acceptors of NSV was provided to the author. The monitoring unit was not allowed to interact with the clients during the programme implementation phase. Subsequently, the author went to evaluate this intervention as the head of an internal team. Appropriate stakeholder participation was ensured in the process of evaluation. Data were collected from the beneficiaries and non-beneficiaries in a respectful manner ensuring adequate privacy and confidentiality. The author observed that many of the acceptors of the NSV were men whose wives had already undergone permanent sterilisation by tubectomy. These men were not actually the targets for the NSV promotion campaign. Although the author reported this, following the principle of transparency, accountability and responsibility, the evaluation report was shelved and no action was taken on it. The information was also withheld from the stakeholders and the community. With time, this funding agency’s priorities changed and the programme itself was withdrawn. The first section in the ethical framework presented in table 1 explains the engagement of stakeholders in M&E. In this case, stakeholder engagement was ensured. However, it is noteworthy that the evaluation was performed by an internal team. Therefore potential conflicts of interest cannot be ruled out. An external impartial evaluation team would have been ideal, but it is often not feasible. Moreover, in the monitoring phase, the team was not allowed to interact with the clients. The process of data collection during the evaluation was carried out in a respectful manner ensuring adequate privacy and confidentiality to the participants. However, it is clearly evident from the case study that the process did not demonstrate responsibility to the community. A responsible action would have been to report the unnecessary NSV for men whose wives had already been sterilised and prevent further such unnecessary NSVs. When such important findings are not shared with the community, it prevents community empowerment. If the monitoring team had been allowed to interact with the clients during the early stages, such unnecessary surgical procedures could have been averted. At the stage of data analysis, interpretation and reporting, the author made a significant finding. The NSVs that seem to have increased were not in the target population, but in a group where there was actually no need for sterilisation. Although honest and complete data analysis was conducted, it was suppressed. Moreover, no action was taken on the basis of the findings. This shows lack of accountability. Most importantly, the findings of the M&E were not used effectively for the benefit of the community. Over time since the programme was withdrawn, some serious issues have been raised about sustainability of the intervention and the M&E process. 3


Public health ethics Table 1

Ethical framework for monitoring and evaluation (M&E) of public health programmes

Phase of M&E

Ethical question

Level of intervention

Engagement of stakeholders

1. Are there conflicts of interest?

Institutional level

2. Is the process of selection of stakeholders Institutional level transparent? 3. Is the methodology for sample selection realistic and Institutional level unbiased? 4. Is there equity in the selection of stakeholders? Community level

Setting objectives, selecting indicators and planning methodology of M&E

5. Are stakeholders being empowered as a result of being selected?

Institutional level

6. Are the selected objectives, indicators and methodology feasible and relevant? 7. Does the objective, indicator and methodology planning process empower the stakeholders and the community? 8. Is the process of objective, indicators and methodology selection transparent?

Institutional level Community level Community level

Institutional level

Community level Data collection

9. Does the process of data collection respect the individuals?

Institutional level Community level

10. Does the process of data collection maintain privacy Institutional level and confidentiality of the respondents?

11. Does the process of data collection demonstrate responsibility to the community?

Community level

12. Does data collection empower the field workers and Institutional level the community? Community level

Data analysis, interpretation, reporting and sharing of results

13. Does the data collection process follow least intrusion and non-judgemental attitudes towards respondents? 14. Is the data analysis, interpretation, reporting and sharing process impartial?

Community level

Community level

15. Are complete and honest data analysis, Institutional level interpretation, reporting and sharing performed at the end of the M&E process? 16. Does the process of data analysis, interpretation, Institutional level reporting and sharing of results demonstrate community accountability? Utilisation of results

17. Is adequate feedback provided to the programme Institutional and managers, stakeholders, community representatives and community levels the community at all stages? 18. Are the results of the M&E used appropriately and Institutional level in a timely manner?

This briefly demonstrates how the M&E process can be retrospectively evaluated on the basis of the proposed ethical framework. Similarly, the framework can be used prospectively at the stage of planning the M&E, so that all important ethical 4

Recommended intervention Consider external evaluation Disclosure of conflicts of interest Observe organisational, legal and other codes Disclose criteria for selection of stakeholders Disclose criteria for the choice of methodology and explain its logic Disclose criteria for including stakeholders and how equity has been achieved Selection of women, members of marginalised communities, persons with disabilities, etc Encourage stakeholders to participate actively in meetings, encourage contributions to discussions, and engage stakeholders in decision making Treat stakeholders with candour and honesty Keep cost effectiveness in mind Encourage community to participate in decision making Involve stakeholders in the process Engage in negotiations with stakeholders about appropriate objectives, indicators and methodology Display the objectives, indicators and methodology of the evaluation to all members of the stakeholders Justify to the stakeholders reasons for choosing the specific objective, indicators and methodology Inform community about the process of objective, indicator and methodology selection Training field workers on respecting individuals during data collection Obtain informed consent by providing the respondents with enough information along with the option to participate or not in the exercise Remove personal identifications from data files Protect data with passwords for data bases, hard copy protection and limiting access to eligible persons only Anonymity and confidentiality of the respondents must be maintained in data processing, analysing and report writing processes Emphasise reporting of mishaps, malpractices and wrongdoing Provide for suitable and well-tiered supervision and checks Work with service motive Training and capacity building of field workers Respond to participant’s questions and doubts. Clarify the agenda of the M&E. Encourage participation of the community Least intrusion into the lives of the respondents. Development of tolerance and non-judgemental attitude Ensure that data from all sections of the community have been considered for analysis. Ensure specifically that opinions of women, children, persons with disabilities and marginalised communities are given voice Full disclosure of all results Special emphasis on data collected using open-ended questions or through other qualitative methods Formation of accountability mechanisms such as community monitoring groups Accountability meetings with community to share the results and information There should be a cross cutting mechanism for providing feedback to the relevant stakeholders at all levels of the process of M&E The results should be used for constructive action on the programme Reciprocity should be enabled Dissemination to all stakeholders including a wider general audience who might be interested

issues can be identified. The M&E ethical framework can also be used during the process, for periodically assessing the ethical functioning of the process. While efforts are made to institutionalise the ethical review of M&E, this framework can be J Med Ethics 2012;0:1–5. doi:10.1136/medethics-2012-100680


Public health ethics used as an effective aid and checklist. Further field testing and operational evaluation of this framework is necessary for wider use and dissemination. Once adequately field tested, this framework can be adopted by non-government organisations and governments for their respective M&Es. In summary, there are several ethical codes of conduct and guidelines for M&E of public health programmes, but there is a definite need for a comprehensive ethical framework for M&E. This paper discusses some of the ethical considerations in the M&E process, presents an ethical framework, and discusses a case study applying the framework to it. Good codes of conduct for a proper M&E include avoiding conflicts of interest, transparency, full disclosure, accountability, empowerment of the community, independent judgement, impartiality, sustainability, respect, privacy, confidentiality, reporting wrong doing, due process and justice. Ultimately, ethical review of M&E should be institutionalised within organisations.

9.

10.

11.

12.

13. 14.

Acknowledgements We acknowledge the significant intellectual input of Dr Amar Jesani (ethicist).

15.

Contributors VG carried out the review of the literature and prepared the draft of the paper. AI added case studies, reviewed the draft, and revised subsequent versions of the manuscript.

16.

Funding VG is supported by the INSPIRE Fellowship of the Department of Science and Technology, Government of India. Competing interests None.

17. 18.

19.

Provenance and peer review Not commissioned; externally peer reviewed.

REFERENCES 1. 2. 3.

4.

5.

6. 7. 8.

Carlos M-A, Wolfgang B, Regina G, et al. The use of epidemiology in local health planning. London: Zed Books, 1997. Levin-Rozalis M. Evaluation and research: differences and similarities. Can J Program Eval 2003;18:1–31. Koplan JP, Milstein RL, Wetterhall S. Framework for program evaluation in public health. Atlanta, Georgia: US Department of Health & Human Services, 1999. http:// www.cdc.gov/pub (accessed 12 Aug 2012). Oklahoma State Univesity Institutional Review Board. Program evaluation: when is it research? http://compliance.vpr.okstate.edu/irb/documents/IRB_toolbox/ Program_Evaluation.pdf (accessed 28 May 2012). National Collaborating Centre for Methods and Tools. Evidence-informed policy making: the role of monitoring and evaluation. Hamilton, ON: McMaster University. (Updated 29 April, 2011). http://www.nccmt.ca/registry/view/eng/82.html (accessed 12 Aug 2012). American Evaluation Association. Guiding principles for evaluators. 2004. http:// www.eval.org/publications/guidingprinciples.asp (accessed 12 Aug 2012). Australasian Evaluation Society. Guidelines for the ethical conduct of evaluation. 2006. http://www.aes.asn.au (accessed 12 Aug 2012). International Federation of the Red Cross and Red Crescent Societies. IFRC framework for evaluation. Planning and Evaluation Department, IFRC Secretariat,

J Med Ethics 2012;0:1–5. doi:10.1136/medethics-2012-100680

20.

21.

22.

23.

24.

25. 26. 27.

2011. http://www.ifrc.org/Global/Publications/monitoring/IFRC-Framework-forEvaluation.pdf (accessed 12 Aug 2012). New Zealand State Services Secretariat, sdoing the right things and doing them right: improving evaluative activity in the New Zealand State Sector. September 2003, p. 47. http://www.ssc.govt.nz/display/document.asp?navid=177& docid=3506&pageno=1#P5_0 (accessed 12 Aug 2012). United Nations Evaluation Group. Norms for evaluation in the UN system, April 29, 2005. http://www.cepal.org/dppo/noticias/paginas/4/37534/ normsforevaluationintheunsystem.pdf (accessed 12 Aug 2012). Chaplowe SG. Monitoring and evaluation planning. Washington, DC and Baltimore, MD: American Red Cross/CRS M&E Module Series. American Red Cross and Catholic Relief Services (CRS), 2008. http://www.crsprogramquality.org/ storage/pubs/me/MEmodules_complete.pdf (accessed 18 Sept 2012). Thomas W, Charles L, Nicole B. Evaluation standards of SEVAL. The Swiss Evaluation Society. December 2005. http://www.seval.ch/en/documents/ SEVAL_Standards_2000_en.pdf (accessed 18 Sept 2012). Smith NL. An analysis of ethical challenges in evaluation. Am J Eval 2002;23: 199–206. Balzer L. What does a successful evaluation project need? Paper presented at the Sixth European Evaluation Society Conference. Berlin, Germany. September, 2004. http://www.lars-balzer.info/projects/erfolgreiche-evaluation/evaluation-success_ EES2004-balzer-paper_280904.pdf (accessed 18 Sept 2012). Turner D, (AES Ethics Committee). Evaluation ethics and quality: results of a survey of Australasian Evaluation Society members. 2003. http://www.aes.asn.au/about/ ethics.html (accessed 12 Aug 2012). Newman DL, Brown RD. Applied ethics for program evaluation. California, Thousand Oaks: Sage, 1996. Berends L. Ethical decision making in evaluation. Eval J Australas 2007;7:40–5. Fontanarosa PB, Flanagin A, DeAngelis CD. Reporting conflicts of interest, financial aspects of research, and role of sponsors in funded studies. JAMA 2005;294:110–11. The World Bank Group. Code of Professional Ethics. 1999 December(cited 2011 Nov 23). http://siteresources.worldbank.org/INTETHICS/Resources/CodeinEnglish.pdf (accessed 12 Aug 2012). Global Evironment Facility Evaluation Office Ethical Guidelines. 2007. http:// www.thegef.org/gef/sites/thegef.org/files/documents/EthicalGuidelines.pdf (accessed 12 Aug 2012). Kusek JZ, Chief GHIV. Ten step to a results based monitoring and evaluation system. 2008. http://www.oecd.org/derec/worldbank/35281194.pdf (accessed 12 Aug 2012). UNICEF Team. Monitoring and evaluation quick reference. 2005. http://www. uniceforg/evaluation/files/ME_PPP_Manual_2005_013006.pdf (accessed 12 Aug 2012). Fawcett SB, Paine-Andrews A, Francisco VT, et al. Empowering community health initiatives through evaluation. Empowerment evaluation: knowledge and tools for self-assessment and accountability. California, Thousand Oaks: Sage, 1996. Development Bank of Southern Africa, African Development Bank and World Bank. Monitoring & evaluation capacity development in Africa. Washington, D.C, 2001. Adair JG. The Hawthorne effect: a reconsideration of the methodological artifact. J Appl Psychol 1984;69:334. Brian R. Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Prev Med 1986;15:451–74. United Nations Development Program. Handbook on planning, monitoring and evaluating for development results. New York: UNDP, 2009.

5


Monitoring 'monitoring' and evaluating 'evaluation': an ethical framework for monitoring and evaluation in public health Vijayaprasad Gopichandran and Anil Kumar Indira Krishna J Med Ethics published online October 30, 2012

doi: 10.1136/medethics-2012-100680

Updated information and services can be found at: http://jme.bmj.com/content/early/2012/10/29/medethics-2012-100680.full.html

These include:

References

This article cites 6 articles http://jme.bmj.com/content/early/2012/10/29/medethics-2012-100680.full.html#ref-list-1

P