Journal of Global Health: Home Journal of Global Health
Watch: Survival: The Story of Global Health - FREE
JoGH Recommends:

Peer Review Conference

Creative Commons License
This work is licensed under a
Creative Commons Attribution
3.0 Unported License

Narendra K Arora, Archisman Mohapatra, Hema S Gopalan, Kerri Wazny, Vasantha Thavaraj, Reeta Rasaily, Manoj K Das, Meenu Maheshwari, Rajiv Bahl, Shamim A Qazi, Robert E Black, and Igor Rudan



Health research in low– and middle– income countries (LMICs) is often driven by donor priorities rather than by the needs of the countries where the research takes place. This lack of alignment of donor's priorities with local research need may be one of the reasons why countries fail to achieve set goals for population health and nutrition. India has a high burden of morbidity and mortality in women, children and infants. In order to look forward toward the Sustainable Development Goals, the Indian Council of Medical Research (ICMR) and the INCLEN Trust International (INCLEN) employed the Child Health and Nutrition Research Initiative's (CHNRI) research priority setting method for maternal, neonatal, child health and nutrition with the timeline of 2016–2025. The exercise was the largest to–date use of the CHNRI methodology, both in terms of participants and ideas generated and also expanded on the methodology.


CHNRI is a crowdsourcing–based exercise that involves using the collective intelligence of a group of stakeholders, usually researchers, to generate and score research options against a set of criteria. This paper reports on a large umbrella CHNRI that was divided into four theme–specific CHNRIs (maternal, newborn, child health and nutrition). A National Steering Group oversaw the exercise and four theme–specific Research Sub–Committees technically supported finalizing the scoring criteria and refinement of research ideas for the respective thematic areas. The exercise engaged participants from 256 institutions across India – 4003 research ideas were generated from 498 experts which were consolidated into 373 research options (maternal health: 122; newborn health: 56; child health: 101; nutrition: 94); 893 experts scored these against five criteria (answerability, relevance, equity, innovation and out–of–box thinking, investment on research). Relative weights to the criteria were assigned by 79 members from the Larger Reference Group. Given India's diversity, priorities were identified at national and three regional levels: (i) the Empowered Action Group (EAG) and North–Eastern States; (ii) States and Union territories in Northern India (including West Bengal); and (iii) States and Union territories in Southern and Western parts of India.


The exercise leveraged the inherent flexibility of the CHNRI method in multiple ways. It expanded on the CHNRI methodology enabling analyses for identification of research priorities at national and regional levels. However, prioritization of research options are only valuable if they are put to use, and we hope that donors will take advantage of this prioritized list of research options.

“Today's health research is tomorrow's health service” [1]. If the research agenda is not aligned to local needs and context, it can perpetuate disharmony, inequity and inefficiency in health services and contribute to lack of attainment of policy goals [2,3]. Given that the scope of research in health and nutrition is ever–expanding and far exceeds the available resources, relative prioritization among competing research options is imperative. This is difficult, liable to subjectivity and vulnerable to being funder–driven [4]. Prioritization using a systematic, transparent, objective and inclusive process could help policy makers and research funding agencies in making their investment decisions more co–aligned, efficient and impactful [2].

The 10–90 report of the Commission on Health Research for Development (1990) emphasized on the prevailing mismatch between local health research needs and the quantum and patterns of fund allocation, particularly in low– and middle– income countries (LMICs) [5]. Between 1990 and 2005, following the 10–90 report, several attempts were made at developing structured and objective methods to identify priorities. Prominent among these, were: (i) the Ad Hoc Committee on Health Research Relating to Future Intervention Options, 1996; (ii) The Council on Health Research and Development (COHRED); (iii) the Essential National Health Research and Priority Setting (ENHR), 1996–2000; (iv) The Grand Challenges in Global Health in 2003; and, (v) the Combined Approach Matrix (CAM) tool by the Global Forum for Health Research, 1999–2004 [3,6]. In 2006–07, the Child Health and Nutrition Research Initiative (CHNRI), informed by weaknesses in existing processes, developed a flexible yet systematic method for setting research priorities, called the CHNRI method. The CHNRI method has become increasingly popular and to date, over 50 CHNRI research priority setting exercises have been reported [7]. This method recognizes research priority setting as a multi–dimensional and multi–stakeholder decision–making process. It balances immediate contextual translational needs (the 'delivery' and 'development' instruments of research) with need for generation of new knowledge through long–term investment (“description” and “discovery”). The CHNRI method systematically delegates, ie, “crowdsources,” [8] the task of prioritization to the various constituencies of stakeholders (end–users of health research funding) [9]. Crowdsourcing is the use of collective wisdom or collective tasks for the benefit of an individual and or an organization, such as to solve a problem or complete a task [10]. The CHNRI method has been shown to be effective at the national level wherein input from local stakeholders can influence research investment policies [11].

India is the second most populous country in the world with many pressing health problems that, in fact, hugely determine the global health statistics. Maternal, neonatal, child health and nutrition (MNCHN) together contribute to the largest burden of disease in India. Public health research decisions in India have traditionally been guided by a small group of experts who are located mostly in the metropolis and are constrained by individual and organizational preferences. In 2011, in response to the seemingly unachievable Millennium Development Goals 4 and 5 (MDG4, MDG5), National Health Mission goals, and the upcoming Sustainable Development Goals 2030, the Indian Council for Medical Research (ICMR; the apex institution for medical research in India) and the INCLEN Trust International (INCLEN; which was the CHNRI Secretariat since 2010) came together to undertake this nationwide research priority setting exercise for MNCHN using the CHNRI methodology. Newborns, children (0–18 years), and reproductive age women (15–49 years, including pregnant women and lactating mothers) were identified to be the target population for prioritization along the life–course continuum. India has large population diversity along with regional– and state–level heterogeneity in governance, program performance, socio–cultural milieu and economics. Hence, it was decided that research priorities would be identified at national and sub–national (regional) levels with a 10–year reference time period (2016–2025) and through inclusion of a large number of stakeholders for representativeness.


The ICMR–INCLEN National Research Priority Setting (RPS) exercise was completed between 2012 and 2016. The exercise was coordinated by the RPS project management team at the Executive Office of INCLEN, New Delhi. The team had experts in the four core MNCHN disciplines (pediatrics, obstetrics and gynecology, community medicine, and public health nutrition) and was multilingual and hence, able to communicate and engage participants from across the country.

States and union territories were grouped into three regions in order to enable sub–national priorities. The three regions were: (i) Empowered Action Group (EAG) States (Rajasthan, Madhya Pradesh, Chattisgarh, Odisha, Jharkhand, Bihar, Uttar Pradesh and Uttarakhand) and North–Eastern (NE) States (Sikkim, Assam, Meghalaya, Tripura, Mizoram, Manipur, Nagaland, Arunachal Pradesh); (The Government of India has identified eight states with poor health and development indicators as EAG states for focused action. EAG and NE states share similarities in MNCHN contexts and program performance.); (ii) Northern states and Union territories (Jammu & Kashmir, Punjab, Himachal Pradesh, Haryana, Chandigarh, Delhi, and West Bengal); and (iii) States and Union Territories in Southern and Western part of the country (Kerala, Tamil Nadu, Karnataka, Andhra Pradesh and Telangana, Maharashtra, Gujarat, Goa, Puducherry).

Four key structures were created to accomplish the task, outlined as follows.

1. The National Steering Group (NSG)

The NSG was the highest body for policy making and oversight for the exercise. Its responsibilities included (i) setting the rationale and contour of the MNCHN research themes; (ii) establishment of research sub–committees (RSCs); (iii) critical review, interpretation and endorsement of the results of the exercise; and, (iv) dissemination of the final national and regional research priorities. The NSG was co–chaired by the Secretary, Department of Health Research (DHR) & Director General (DG–ICMR) and Executive Director of INCLEN. It included key officials from the Ministry of Health & Family Welfare (National Health Mission, Child Health, Maternal Health and Nutrition divisions, Directorate General of Health Services and DHR–ICMR), Ministry of Women and Child Development (Integrated Child Development Services, Food and Nutrition Board), and Ministry of Science and Technology (Department of Biotechnology, Department of Science & Technology). Its membership also included invited subject experts and representatives of national and international donors and multilateral agencies. The chairs of all four RSCs were also members of the NSG ( Table 1 ). Two NSG meetings were organized – the first (on 18th April 2013), at the initiation of the exercise to ratify the context ( Box 1 ) and protocol, and the second (on 4th February 2016), at the conclusion to review, refine and finalize the results.

Table 1  Profile of the National Steering Group
Expertise 18 Apr 2013 4 Feb 2016
Policy–Decision Makers and Program Managers (MNCHN), Government of India 22 24
Multilateral/ Bilateral Donor Agencies/Foundation – Funders 15 19
Technical Experts (MNCHN) 29 21
State Program Managers (ICDS, NRHM, Directorate of Health Services) 9 11
Biomedical Journal Editors 3 3
Total 78 78

Box 1  Context of the INCLEN ICMR national research priority setting exercise in maternal, newborn, children health and nutrition

Purpose: Priority setting in maternal, newborn, and child health and nutrition for efficient and rewarding investment in research using a systematic, transparent, inclusive, objective and quantitative method.

Target population: Women of reproductive age (15–49 years) including pregnant and lactating women, newborns (0–28 days), under–five children (0–59 months) and children up to the age of 18 years.

Geography: Priorities at National and three Regional levels: Empowered Action Group States and North–Eastern States, States and Union Territories in Northern India, and those in Southern and Western India.

Major areas of concern for research: Conditions that together contributed to 75% of the mortality and morbidity burden in Maternal, Newborn, Child Health and Nutrition in India during 2012–2013 as per the available evidence and expert opinion.

Time frame: For the next ten years ie, 2016–2025 (with due consideration to unachieved Millennium Development Goals 1, 4 and 5, and National Health Mission targets and the challenge of preparing the national agenda for achieving forthcoming Sustainable Development Goals 2030).

Stakeholder constituencies (operating in civil, public and private sectors, health and non–health sectors): Researchers, professionals, public health functionaries, policy makers, communities and their leadership, civil society, donor agencies and industries.

Translation and implementation context: Public and private health systems of India and their existing as well as future programs, national and international institutions & organizations funding research, research environment in academic & research institutions.

2. The thematic Research Sub–Committees (RSCs)

A RSC was constituted for each of the four themes. The RSCs' membership included technical experts (subject experts, basic scientists and public health specialists), social scientists, program specialists (health, and woman and child development), and donor agency representatives. Technical experts were identified through a literature search for active research contribution to respective MNCHN domains ( Table 2 ). The RSCs participated in the crowdsourcing processes along with the nationwide network. They also helped in the iterative refinement and consolidation of the research options (ROs) and in finalizing the scoring criteria and their definitions. Respective RSCs presented the study findings to the second meeting of the NSG for review.

Table 2  Profile of research sub–committees and nation–wide network (1st round of crowd–sourcing)
Group Expertise Theme (with components)
Maternal health Newborn health Child health Nutrition Total
Mortality Morbidity Stillbirths Maternal Childhood
Research Sub–Committee (RSC) Basic scientists* 1 1 1 3
Dietitians and nutritionists 8 9 17
Experts from ICMR institutes 1 1
Nursing & midwifery experts 1 1
Obstetricians and gynecologists 7 4 4 15
Pediatricians and neonatologists 13 13 26
Policy makers (Government of India)* 1 1 1 2 5
Scientists from research institutes (public health and allied sciences)* 2 2 2 2 2 1 11
State program managers* 3 1 1 2 1 2 10
Technical Experts from donor agencies* 3 2 1 7 7 1 2 23
Sub–total 16 9 9 25 26 12 15 112
Nation–wide network (beyond RSCs) Agriculturists 3 5 8
Basic scientists* 2 2 1 1 1 1 3 11
Community medicine experts 44 45 42 39 38 61 35 304
Dietitians & nutritionists 33 33 66
Experts from ICMR Institutes* 8 9 9 10 12 6 4 58
Miscellaneous* 1 1
Nursing & midwifery experts 3 3 3 9
Obstetricians and gynecologists 68 74 82 1 17 242
Pediatricians and neonatologists 111 122 47 280
Policy Makers (Government of India)* 2 2
Scientists from research institutes (public health and allied sciences)* 7 4 4 7 8 3 5 38
State program managers* 4 4 4 4 3 3 22
Technical Experts from donor agencies* 2 1 5 5 13
Sub–total 136 141 141 175 186 133 142 1054
Grand total 152 150 150 200 212 145 157 1166

*The experts in these categories were requested to identify their theme/ component of expertise.

3. The Nationwide Network for crowd sourcing

A network was established with experts identified from institutions and departments across the country. Faculty/researchers from departments that were directly or indirectly engaged in work pertaining to MNCHN (eg,, obstetrics & gynecology, pediatrics, neonatology, community medicine, biochemistry, physiology, pathology, microbiology, midwifery, public health nutrition and home sciences, social sciences, statistics and demography, and agriculture) were contacted through their respective institutional heads. The effort was to secure similar proportion of faculty members/researchers with more than 10 years of research or teaching experience (ie, 'senior' faculty) and those who are junior/middle level with 5–10 years of experience. National and zonal office–bearers of major professional associations in MNCHN (the Indian Academy of Pediatrics, the National Neonatology Forum, the Federation of Obstetrics and Gynecological Societies of India, the Indian Association of Preventive and Social Medicine, the Indian Public Health Association, the Nutrition Society of India, and the Indian Dietetic Association) were also contacted for participation. Central and state–level policy–makers and program managers were also invited to participate in the exercise. These were from departments of health and of women and child development. Experts were also identified through snow–balling and invitations in personal capacity.

The members in the nationwide network consented to be allocated into one of the four themes according to their expertise and publication history to achieve equitable regional and disciplinary representation in each theme. In this manner, for the first round of crowd sourcing, 1423 experts (including the 112 in the RSCs) were identified, of whom 1178 could be contacted. Of these, 12 declined to participate. Of the remaining 1166 experts ( Table 2 ), 668 did not respond. Overall, 498 (42.3%) experts contributed research ideas. For the second round (scoring activity), 1536 experts were contacted (including those contacted during the first round) of which 15 declined, 628 did not respond/ logged in but did not score, and 893 (58.1%) participated. Overall, 256 institutions including medical colleges, ICMR institutions, research organizations, NGOs, state health departments and donor agencies participated in the two rounds for crowdsourcing ( Table 3 ).

Table 3  Profile of participating institutions in the Nationwide Network
State/Union Territory Medical colleges ICMR institutions Other public health research institutes Non–governmental organizations State departments (health and nutrition) Donor agencies TOTAL
Assam 3 1 1 2 1 8
Manipur 1 1
Meghalya 1 1 2
Nagaland 1 1
Odisha 7 1 3 11
Sikkim 1 1
Tripura 2 1 3
West Bengal 11 1 2 14
Chandigarh 2 2
Delhi 8 2 8 5 2 25
Haryana 1 1 1 3
Himachal Pradesh 2 1 3
Jammu & Kashmir 1 2 3
Punjab 5 3 8
Uttar Pradesh 14 1 15
Uttarakhand 1 1
Goa 1 1 2
Gujarat 10 2 1 13
Maharashtra 19 3 1 1 2 26
Rajasthan 14 1 2 17
Andhra Pradesh 15 1 6 1 2 25
Karnataka 15 1 1 1 18
Kerala 9 1 2 12
Puducherry 1 1 2
Tamil Nadu 8 3 3 14
Bihar 4 2 1 7
Chattisgarh 4 4
Jharkhand 1 1
Madhya Pradesh 10 1 3 14
Grand total 169 15 38 12 19 3 256

4. The Larger Reference Group (LRG)

Beyond 75% of CHNRI exercises published have not employed a LRG (majorly due to trouble composing the group). Of those that could, most have been conducted at a national level [7]. To incorporate broader societal perspectives and values within the exercise, we employed a LRG which was composed of policy decision makers (n = 24; Central and State politicians and bureaucrats from key Ministries, eg, Health and Family Welfare, Woman and Child Development, Human Resource Development), senior researchers (n = 17), MNCHN program managers from central and state governments (n = 24) and representatives from research funding organizations (n = 19). The LRG attributed relative weights to the scoring criteria which helped to generate criteria–weighed priority ranks for the ROs.


Figure 1 shows the schematic flow of activities with timelines.

Figure 1Sequence of activities undertaken in the ICMR–INCLEN National Research Priority Setting Exercise.

Review of literature and identification of areas of concern

Extensive review of literature on burden of MNCHN related conditions was done in 2012–13 with focus on Indian data. We searched published literature (indexed and non–indexed), Government of India's policy documents and reports, program reviews and grey literature for the period of 1990 to 2012/2013. PubMed, CINAHL and Embase databases were searched. Based on the compendium of literature (available at: ), a draft list of “areas of concern” (AOCs) was prepared for each of the RPS themes and presented to the first meeting of the NSG for review. The AOCs accounted for conditions that collectively contributed to at least 75% of the mortality and morbidity burden in the respective theme.

The NSG suggested that maternal health encompassed three components viz., morbidity, mortality and stillbirths (most stillbirths occur in–utero and are thus are a maternal health concern). Similarly, it divided the nutrition theme into maternal and childhood nutrition components. The NSG advised to include three additional AOCs in each of the themes: “social determinants,” “impact and improvement of existing composite public health packages,” and “novel & innovative public health interventions.” The final approved list of AOCs under the four themes is presented in Table 4 .

Table 4  Areas of concern finalized by the National Steering Group
Maternal Health Theme: Maternal Mortality Component
1 Hemorrhage
2 Hypertensive disorders of pregnancy
3 Sepsis
4 Obstructed labor
5 Unsafe abortion
6 Anemia and other nutritional problems
7 Medical disorders in pregnancy [eg, chronic hypertension, epilepsy, liver disease, diabetes mellitus, renal disease, thyroid disease, lupus]
8 Malaria in pregnancy
9 Others (Please specify ________)
10 Social determinants of maternal mortality [eg, social isolation, stigmatization, marital disharmony, divorce, household dissolution, domestic violence, loss of community status; caste, religion, teenage pregnancy, cultural practices]
11 Economic [eg, impoverishment and poverty]
12 “Existing” composite public health packages with potential impact on MMR [eg, Janani Shishu Suraksha Karyakram (JSSK)]
13 “Novel & Innovative” composite public health packages with potential impact on MMR [eg, Innovative solutions to promote access to care]
Maternal Health Theme: Maternal Morbidity Component
1 Severe acute maternal morbidities (SAMMs) and Near miss events
2 Post partum morbidities and long term disabilities [eg, obstetric fistula, utero–vaginal prolapse, urinary incontinence, dyspaerunia, infertility]
3 Post partum depression and psychosis
4 Strong fear of pregnancy and child birth
5 Social [eg, social isolation, stigmatization, marital disharmony, divorce, household dissolution, domestic violence, loss of community status; caste, religion, teenage pregnancy, cultural practices]
6 Economic [eg, impoverishment and poverty]
7 “Existing” composite public health packages with potential impact on maternal morbidity [eg, syndromic management of RTI & STI; Reproductive, maternal, newborn, child and adolescent health (RMNCH+A)]
8 “Novel & Innovative” composite public health packages with potential impact on maternal morbidity [eg, innovative solutions to promote access to care]
Maternal Health Theme: Stillbirth Component
1 Maternal cause: Hypertensive disorders of pregnancy
2 Maternal cause: Maternal infections in pregnancy [eg, TORCH group of infections]
3 Maternal cause: Underlying chronic maternal illness [eg, chronic hypertension, epilepsy, liver disease, diabetes mellitus, renal disease, thyroid disease, lupus]
4 Maternal cause: Maternal malnutrition [eg, low maternal BMI, gestational diabetes]
5 Fetal cause: Intra uterine growth restriction
6 Fetal cause: Pre–term birth
7 Fetal cause: Congenital malformations
8 Intra partum cause: Acute hypoxic insult
9 Intra partum cause: Obstetric complications
10 Complications of placenta, cord and membranes
11 Unexplained [By known maternal, placental and fetal conditions]
12 Non-health factors [eg, Indoor air pollution, tobacco smoke]
13 Social determinants of stillbirths [eg, prevailing harmful traditional birth practices, lack of womens' empowerment, poverty, illiteracy]
14 “Existing” composite public health packages with potential to influence stillbirths[eg, Janani Shishu Suraksha Karyakram (JSSK)]
15 “Novel & Innovative” composite public health packages with potential to influence stillbirths[eg, Innovative solutions to promote access to care]
Child Health Theme:
1 Pneumonia (and ARI)
2 Diarrheal diseases
3 Measles and vaccine preventable diseases
4 Congenital anomalies
5 Malaria
6 Unintentional injuries
7 Acute bacterial sepsis
8 Meningitis/encephalitis
9 Other infections & parasitic diseases
10 Neuro–developmental disorders (NDD) [eg, early developmental delays, autism, speech & language disorders, intellectual disability, epilepsy, CP, neuro–motor impairment, audio–visual impairment]
11 Others (Please specify ____)
12 Social determinants of under 5 mortality rate [eg, immunization refusal, inappropriate feeding practices, poor health seeking behavior.]
13 “Existing” composite public health packages with potential impact on Under 5 Mortality Rate [eg, IMNCI, F-IMNCI, Reproductive, Maternal, Newborn, Child and Adolescent Health(RMNCH+A)]
14 “Novel & Innovative” composite public health packages with potential impact on Under–5 Mortality Rate [eg, Rashtriya Bal Swasthya Karyakram (RBSK) – Child Health Screening and Early Intervention Services]
Newborn Health Theme:
1 Preterm birth
2 Neonatal sepsis including pneumonia
3 Birth asphyxia & Birth trauma
4 Congenital malformations
5 Others (Please specify ____)
6 Social determinants of NMR [eg, newborn care practices, poverty, poor health seeking behavior]
7 “Existing” composite public health packages with potential to influence neonatal morbidity and mortality [eg, IMNCI, Home based newborn care, Reproductive, maternal, newborn, child and adolescent health (RMNCH+A)]
8 “Novel & Innovative” composite public health packages with potential to influence neonatal morbidity and mortality [eg, Innovative solutions to promote access to care]
Nutrition Theme: childhood nutrition component:
1 Protein energy malnutrition (PEM)
2 Low birth weight
3 Micro-nutrient deficiencies (iron/folic acid/zinc/iodine/Vitamin A)
4 Childhood overweight and obesity
5 Nutrition deficiency associated congenital malformations
6 Fetal and child nutrition and origin of adult chronic non–communicable diseases [eg, cardiovascular diseases, metabolic syndrome, obesity etc.]
7 Socio–cultural and economic determinants: time constraint with mothers entering into the work force
8 Socio–cultural and economic determinants: care and feeding practices
9 Socio–cultural and economic determinants: competing use of resources for goods and services other than nutrition/food
10 Socio–cultural and economic determinants: globalization & market forces influencing food habits
11 Socio–cultural and economic determinants: status of girl child and women in the community
12 Socio–cultural and economic determinants: use of pesticides & fertilizers
13 Socio–cultural and economic determinants: potable water, hygiene and sanitation
14 Socio–cultural and economic determinants: others (Please specify ____)
15 “Existing” composite public health packages with potential impact on Child nutrition [eg, ICDS, Mid-day Meal Program]
16 “Novel & Innovative” composite public health packages with potential impact on child nutrition [eg, Food fortification program, promotion of kitchen gardens/organic farming, deworming, convergent-innovation coalition to address issues of anemia, under-nutrition, obesity]
Nutrition Theme: Maternal Nutrition Component
1 Anemia among women of reproductive age group
2 Iodine deficiency disorders among women
3 Vitamin D deficiency among women
4 Maternal overweight & obesity and other non-communicable diseases
5 Socio–cultural and economic determinants: time constraint with mothers entering into the work force
6 Socio–cultural and economic determinants: care and feeding practices
7 Socio–cultural and economic determinants: competing use of resources for goods and services other than nutrition/food
8 Socio–cultural and economic determinants: globalization & market forces influencing food habits
9 Socio–cultural and economic determinants: women's status in the community, family structures and norms
10 Socio–cultural and economic determinants: others (Please specify ______)
11 “Existing” composite public health packages with potential impact on maternal nutrition
12 “Novel & Innovative” composite public health packages with potential impact on maternal nutrition

The NSG also suggested that all research ideas (RIs) be segregated into the four domains of research: (i) description (burden of disease, epidemiology, etiology and risk factors, biomarkers, pathophysiological descriptions); (ii) discovery (identification of novel pathways, discovery of novel clinical and public health interventions/package, technology inventions, discoveries and innovations); (iii) delivery (health policy and systems research, including program evaluation and implementation research); and (iv) development (improving the existing intervention, ie, design, deliverability, affordability and sustainability).


First round of crowdsourcing

Solicitation of research ideas (RIs) from the Nationwide Network: An online software was designed by INCLEN for submission of RIs by the network. The software had seven separate electronic forms: maternal health (n = 3: mortality, morbidity & stillbirth); newborn health (n = 1); child health (n = 1); and nutrition (n = 2: maternal & child nutrition) themes]. The experts in the nationwide network and RSCs were sent an initial email and then contacted over the phone: (i) to sensitize them about the method of the research priority setting exercise; (ii) to provide them the context and scope of the exercise; and, (iii) the provide them with the purpose of the first round of crowdsourcing. Each participant was provided with an individualized log–in username and password for the dedicated software. The participant could log in to only one of the seven electronic forms as pre–assigned to him/her. After logging–in, s/he was asked to enter personal details (name, area(s) of work, employment status (working/retired), institution, state/union territory, alternative email ID). S/he was then taken through a self–orientation power–point tutorial. The list of AOCs was then displayed on his/her computer screen and the participant was instructed to select any two AOCs to contribute RIs in the four domains of research (description, discovery, delivery and development). The expert was not limited in the number of RIs s/he could submit under each domain. The electronic forms allowed for completion over multiple sessions. An offline version of the form was prepared and shared with participants who had difficulty in accessing the internet. A total of 3497 RIs were obtained across the MNCHN themes from 498 experts (42.3% participation).

Refinement of the research ideas: The RPS project management team at INCLEN along with the RSCs closely examined each RI and rephrased, split, and combined the RIs (as required) keeping the core idea intact and without discarding any RI. The original RI list was maintained as a separate file for ready reference at any time. The process was intuitive, consultative and iterative (completed through brainstorming by teams over several sittings). As far as possible, the RIs were refined in a way that described the population, intervention, control, and outcome (PICO). This process led to a compendium of 4003 RIs from the original 3497 RIs. ( Table 5 ).

Table 5  Research ideas obtained through the first round of crowd–sourcing and subsequent refinement
Theme Component Total number of areas of concern Number of research ideas (received) Number of research ideas (after refinement)
Maternal health Mortality 13 436 523
Stillbirths 15 418 542
Morbidity 8 353 243
Lateral submissions* 11
Subtotal 1218 1308
Newborn health 8 641 626
Child health 12 596 648
Nutrition Maternal nutrition 12 450 590
Childhood nutrition 16 590 831
Lateral submissions* 2
Subtotal 1042 1421
Total 3497 4003

*Research ideas received from the National Steering Group as and when through hand–written submissions.

Development of research options (ROs): The 4003 RIs were consolidated onto 373 ROs. These were crystallized through iterative refinement to avoid duplication and redundancy. Each RO represented a portfolio of inter–related RIs that addressed a central research concept. Thus, the ROs addressed multiple AOCs and several of these pertained to cross–cutting issues across domains, components and themes. The ROs were finally categorized into four themes (maternal health: 122, newborn health: 56, child health: 101, nutrition: 94) ( Table 6 ).

Table 6  Distribution of the research options in the domains of research
Domain of research Frequency (%) of research options in themes
Maternal health Newborn health Child health Nutrition
Description 42 (34.4) 15 (26.8) 39 (38.6) 35 (37.2)
Delivery 57 (46.7) 24 (42.9) 37 (36.6) 42 (44.7)
Development 44 (36.1) 21 (37.5) 37 (36.6) 27 (28.7)
Discovery 8 (6.6) 4 (7.1) 4 (4.0) 2 (2.1)
Single domain 29 (23.8) 8 (14.3) 16 (15.8) 12 (12.8)
>1 domain 93 (76.2) 48 (85.7) 85 (84.2) 82 (87.2)
Total (N = 373) 122 (100.0) 56 (100.0) 101 (100.0) 94 (100.0)

Second round of crowdsourcing

Finalization of criteria for scoring: Previously published CHNRI exercises were reviewed extensively to retrieve scoring criteria used in past exercises. Two rounds of consultation were held with RSC members, international CHNRI experts, and experts from the World Health Organization who had been closely associated with previous CHNRI exercises. Five succinctly worded criteria (answerability, relevance, equity, innovation and out–of–the–box thinking, and investment on research) were finalized. These criteria were believed to be consistently applicable across domains, themes and ROs ( Box 2 ). The context and scope of the present exercise, nature of the ROs and the large number of scorers from various disciplines across India that were to score the research options were the key considerations while deciding on the scoring criteria to be used. The scorers were expected to evaluate the ROs against the criteria by choosing one of the following responses: 'Yes' if the research option favorably met the criterion query, 'No' if it did not, and 'Not my expertise' if the scorer felt that s/he was not sufficiently informed to adjudge the research option against the particular criterion. While other CHNRI exercises employed sub–questions under each criterion, we chose to forego sub–questions as we were advised that sub–questions usually had high agreement [12] and also because our exercise had a large number of ROs to be scored and we were aiming to maximize retention of participants by minimizing scorer fatigue, especially in order to preserve the validity of our planned regional analyses.

Box 2  Scoring criteria and their definitions
  1. Answerability. Can the research be done through ethical, transparent, well–designed, “do–able” studies with the existing local and national capacities and or by strengthening the existing capacities through regional or global collaboration?

  2. Relevance. Is it likely that the research would address a high burden condition and critical gap in knowledge?

  3. Innovation and out–of–box thinking to resolve complex, and refractory challenges. Does the new research have the potential for transformative change in the health system/ health care?

  4. Equity. Is it likely that the research product will address the differences in health and nutrition that are systematically associated with social, cultural and economic hierarchies, ethnicity, gender, environment and geographic disadvantages, thereby reducing inequities?

  5. Investment on research. Is it likely that the potential impact and benefits of the new knowledge on health/ nutrition will outweigh the consideration of investments on research?

Scoring of the research options by the Nationwide Network: The scoring exercise was done using a user–friendly online interface ( that allowed for having individualized scorer accounts that could be accessed through an invitation email from the INCLEN RPS project management team. The software could readily archive access details (email and IP addresses) and responses selected by the scorer. Once the scorer logged in, s/he underwent a comprehensive orientation of the context and method of the exercise, and the scoring criteria and process. Thereafter, ROs appeared in a random sequence, one at a time, on the scorer's computer/smart phone screen. The scorer was requested to score all the ROs for the assigned theme. As the number of ROs to be scored was high and could have led to high scorer burden and attrition, each scorer was randomly allocated a combination of two of the five criteria for scoring. Five such criteria combinations (survey questionnaires) had been prepared for scoring: (i) Answerability and Innovation; (ii) Answerability and Equity; (iii) Relevance and Innovation; (iv) Relevance and Investment on Research; and (v) Equity and Investment on Research. The nationwide network was stratified at two levels: first, according to their participation status in the first round of crowd sourcing ('participated', 'could not participate', or 'newly invited' experts); and, second, according to their region. Subsequently, the experts within each region were equally distributed across the five survey questionnaires within the theme through consecutive allocation (the expert with serial number 1 got Survey Questionnaire 1; the next in line got Survey Questionnaire 2 and so on; the questionnaire allocation cycle was restarted with every 6th expert).

It was mandatory for the scorer to evaluate the RO on the screen against both of the assigned criteria before moving on to the next RO (ie, skip logic was disabled). However, the scorer could review and edit his previous responses once s/he had moved forward. Completion over multiple sessions was allowed to avoid effects of scorer fatigue and overcome time constraints. The RPS project management team at INCLEN remained vigorously engaged with the nationwide network through email and telephone for immediate troubleshooting and timely reminders, and used continuous real–time data monitoring to check progress. Scorers who requested hard copies of the questionnaires instead of the online process were provided with the same for recording the responses. In the second round of crowdsourcing, 893 scorers participated (58.1% participation rate) ( Table 7 ).

Table 7  Distribution of experts who participated in the 2nd round of crowd–sourcing (the Scoring Exercise)
Region Maternal health Newborn health Child health Nutrition Overall
Male Female Total Male Female Total Male Female Total Male Female Total Male Female Total
EAG States and North Eastern States 39 48 87 39 23 62 55 14 69 42 28 70 175 113 288
Northern States and UTs (including West Bengal) 25 44 69 57 15 72 52 16 68 31 29 60 165 104 269
Southern and Western States and UTs 39 55 94 37 27 64 69 27 96 32 50 82 177 159 336
Total 103 147 250 133 65 198 176 57 233 105 107 212 517 376 893

EAG – Empowered Action Group, UT – Union Territories

Assignment of relative criteria weights by the LRG

The LRG members were given an in–depth explanation of the CHNRI exercise. They were then requested to assign relative weights to the scoring criteria by distributing a hypothetical amount of Indian Rupees (INR) 100 across the five criteria, giving the maximum amount to the criteria they felt to be the most important and the minimum to the least important. The relative weight for each criterion was computed by calculating the arithmetic mean of the average amount received by the respective criterion in each LRG constituency ( Table 8 ). Of 84 members approached for the LRG, 79 participated (94.0% participation). The LRG ascribed maximum relative weight to Relevance (0.254), followed by Innovation and Out–of–Box Thinking (0.199), Equity (0.193), Answerability (0.192), and Investment on Research (0.161).

Table 8  Relative weights assigned to the scoring criteria by the Larger Reference Group
LRG categories Answerability Relevance Equity Innovation Investment on research
  Policy decision makers, politicians (N = 18) 0.197 0.229 0.209 0.203 0.162
  Eminent researchers (N = 17) 0.212 0.245 0.169 0.197 0.177
  MNCHN program managers from central and state governments (N = 24) 0.186 0.254 0.201 0.198 0.162
  Funding agencies (N = 20) 0.173 0.288 0.195 0.200 0.145
  Overall (N = 79) 0.192 0.254 0.193 0.199 0.161

LRG – Larger Reference Group, MNCHN – Maternal, Newborn, Child Health and Nutrition

Data management and analysis

The “Yes” and “No” responses were scored as “1” and “0” respectively. The “Not my expertise” responses were excluded from the calculations. Relative ranking and Research Priority Scores (RPS) were calculated as follows [13]:

Average scores received against each of the five criteria were calculated for each RO.

  1. The criteria weights (as assigned by the LRG) were applied to the mean score received by each criterion.

  2. Research Priority Scores (RPS) were calculated by adding together each criterion's weighted scores for each RO.

The ROs were arranged in descending order of their RPS to get national and regional rankings. Work location of the scorer as entered by him/ her at the time of the scoring determined the regional ranking.

Average Expert Agreement (AEA) [14] was also calculated for each RO. The AEA is a proportion of scorers who scored the most common score for a particular RO divided by the total number of scorers who scored that RO.

The second meeting of the NSG reviewed the ranked list of national and regional research priorities. The group further suggested to identify ROs relevant to three more themes: (i) adolescence; (ii) issues cutting across four MNCHN themes for greater impact on health and health systems; and, (iii) areas requiring biotechnology methods from the compendium of 373 ROs, and generate ranked lists according to their RPS for each of these.

The results from all exercises are reported in–depth separately in manuscripts prepared for submission to the Journal of Global Health. The overall discussions by the National Steering Group on the results and way forward for the exercise have been has been accepted for publication in the Indian Journal of Medical Research.


The COHRED Working Group on Priority Setting highlighted that engagement of a wide spectrum of stakeholders is essential to identify priorities that reflect research needs, available technical and financial capacity, and societal values and ethics [15]. Stakeholder engagement, and data and capacity constraints frequently impeded the process for setting priorities, more so in the LMICs [16]. The current exercise, through systematic inclusion of diverse range of national stakeholders in a LMIC setting, identified priorities for maternal, neonatal and child health and nutrition at national and sub–national (regional) levels. The exercise leveraged the inherent flexibility of the systematic CHNRI method and built further methodological robustness. CHNRI exercises hitherto had taken a conservative approach in considering active contribution to research/policy as a selection pre–requisite for scorers. In contrast, we expanded the stakeholder base to include diverse range of doers and users (techno–managerial) of research in the field of MNCHN. This helped in including a variety of viewpoints in the scoring process and possibly, led to prioritization of ROs that was important to both.

Having Indian nationals as the exclusive contributors and scorers to this exercise makes it unique from previous exercises. In this way, this CHNRI exercise is truly a representation of, and driven by, India's health and nutrition community. Moreover, the exercise is the first to conduct su–national–level analysis which, in a country as large and diverse as India, is imperative to truly explore research priorities and enable the country to tailor interventions regionally. With effective use of technology and building on INCLEN's network for multi–centric studies, 498 experts from across India contributed research ideas and 893 experts were involved in the scoring process. About 75 (60–96) experts were involved per region per theme to score the ROs. The large number of scorers (“sample size”) should have led to saturation and stable estimate of priority ranks at national and sub–national (regional) levels [8]. The improved response rates between first and second rounds of crowdsourcing should have reduced bias [17]. Gender distribution of scorers is a reflection of skewed gender participation in program management, research and academia for the themes considered in this exercise. The scorer profiles have been discussed in details in the respective thematic papers prepared for submission to JoGH.

To minimize scorer fatigue, we asked the participants to score against predefined pairs of criteria allocated randomly to them instead of all five criteria. The AEA for each evaluated research option represents the proportion of scorers that gave the most frequent (modal) response [14]. For the top 10 ROs at national level across the themes, the AEA for both individual and aggregate of the five criteria was fairly high (maternal health: 0.887—0.929; newborn health: 0.871—0.902; child health: 0.899—0.923; nutrition: 0.869—0.923) indicating consistency among the scorers. This also indicates minimal bias due to partial criteria scoring adopted in the current exercise and appears to be a pragmatic approach for better participant compliance without affecting the validity of the priority setting scoring. There were four distinct constituencies among the LRG; the LRG is to be viewed as a strength since different constituencies are likely to have differences in their collective perspective about research priorities [18]. It was interesting to observe that “Relevance” was accorded the highest weight by all the LRG sub–groups highlighting that priorities should be suited to the context.

In view of the disease burden and significance of the health systems in the implementation and delivery of services, the NSG suggested developing ranked priority lists for adolescent health, cross cutting themes and biotechnology related ROs from the 373 ROs spread across different themes. These lists will, at best, be an indicative priority list because the ROs were picked up from different thematic groups, scored by dissimilar set of experts with differences in their professional expertise. Although the overall AEA was high across themes, the validity of RO scores to determine their relative ranking shall remain unknown for these additional lists.

The exercise was the largest to–date use of the CHNRI methodology in terms of research ideas collected, processed and scored, and the number of participants and spectrum of stakeholder constituencies engaged. It expanded on the CHNRI methodology and thus, contributes to further evolution of the CHNRI method as a robust, inclusive, participatory, transparent and objective technique for identification of research priorities. It has been opined that prioritization processes will have an impact only if funders have a buy–in. It is also anticipated that there is an imminent challenge to develop tools to detect and evaluate the impact of CHNRI exercises on funder decision making and priorities [19]. A recent article in Lancet affixes with the research funders and research regulators, the primary responsibility of addressing the sources of avoidable waste once research priorities are set [20]. We hope that ICMR–INCLEN collaborative effort helps in rational distribution of health and nutrition research budget by the Government of India and donor agencies funding research in India and in similar LMIC contexts, and also inform any mid–course correction of currently funded research portfolio as needed. Sub–national (regional) prioritization should further help in matching the exercise's findings to other LMIC contexts. This exercise can serve as a guidance for other LMICs, especially those with diversity among their populations, in setting research priorities nationally.


The team is grateful to the experts from different parts of India for their enthusiastic participation in this exercise and kind support. We are also thankful to Dr Geeta Chhibber, Dr Manish Singh Chundawat, Dr Kriti Agarwal, Mr Amit K Sagar, Mr Gaurav Banyal and Mr Chandan Singh for their assistance during various phases of the exercise.

Disclaimer: Rajiv Bahl and Shamim Qazi are staff members of the World Health Organization. The expressed views and opinions do not necessarily express the policies of the World Health Organization.

Ethics: The study protocol was reviewed and approved by the Independent Institutional Ethics Committee of INCLEN.


[1] Financial disclosure Funding: This exercise was co–funded by ICMR, the CHNRI Secretariat and INCLEN, New Delhi.

[2] Authorship contributions: NKA was the PI and provided overall leadership to the entire exercise by chairing the NSG meetings. He was responsible for the conception and design of the exercise, acquisition and interpretation of data, revising the manuscript critically and for the final approval of the version submitted. All communications with the network and request for participation in the exercise were made on his behalf. AM, HSG, MM and MKD were part of the INCLEN RPS project management team. AM was responsible for review of literature, execution of the project, identification of participants, communication with the network, management and analysis of the data and for drafting the manuscript. HSG and MM were responsible for management of the network, data archiving and retrieval, supporting AM in data analysis and in preparing the draft manuscript. KW and IR provided critical technical and strategic inputs for the design and execution of the exercise. All authors were responsible for providing technical guidance to the exercise at different points of the project lifecycle, and in critically reviewing and editing the manuscript for intellectual content and in approving the final version for submission.

[3] Conflicts of interest Competing interests: All authors have completed the ICMJE uniform disclosure form at (available upon request from the corresponding author) and declare no conflict of interest. IR is the editor–in–chief of the Journal of Global Health. To ensure that any possible conflict of interest relevant to the journal has been addressed, this article was reviewed according to best practice guidelines of international editorial organizations.


[1] A New Approach in Systematic Priority Setting in Child Health Research Investment. 2006. Bangladesh: Child Health and Nutrition Research Initiative (CHNRI) 2006. Available: Accessed: 20 August 2016.

[2] I Rudan J Gibson L Kapiriri MA Lansang AA Hyder J Lawn Setting priorities in global child health research investments: assessment of principles and practice. Croat Med J 2007 48 595 604 17948946

[3] Fontaine O, Martines J, Bahl R, Briend A, Muhe L, El Arifeen S, et al. Setting health research priorities to address Millennium Development Goal 4 and Child Mortality at the Global Level. 2006. Available: Accessed: 20 August 2016.

[4] S Yoshida Approaches, tools and methods used for setting priorities in health research in the 21st century. J Glob Health 2016 6 010507 10.7189/jogh.06.010302 26401271

[5] The 10/90 Report on Health Research 1999. Global Forum for Health Research. Geneva Switzerland. 1999. Available: Accessed: 15 January 2017.

[6] A Ghaffar Setting research priorities by applying the combined approach matrix. Indian J Med Res 2009 129 368 75 19535830

[7] I Rudan S Yoshida KY Chan D Sridhar K Wazny H Nair Setting health research priorities using the CHNRI method: VII. A review of the first 50 applications of the CHNRI method. J Glob Health 2017 7 011004 10.7189/jogh.06.010303 27303649

[8] S Yoshida I Rudan S Cousens Setting health research priorities using the CHNRI method: VI. Quantitative properties of human collective opinion. J Glob Health 2016 6 010503 10.7189/jogh.06.010503 27350874

[9] I Rudan M Chopra L Kapiriri J Gibson MA Lansang I Carneiro Setting priorities in global child health research investments: universal challenges and conceptual framework. Croat Med J 2008 49 307 17 10.3325/cmj.2008.3.307 18581609

[10] K Wazny 'Crowdsourcing' ten years in: a review.J Glob Health20177forthcoming

[11] Tomlinson M, Chopra M, Sanders D, Bradshaw D, Hendricks M, Greenfield D, et al. Setting priorities in child health research investments for South Africa. Available: Accessed: 20 August 2016.

[12] Wazny, Kerri (University of Edinburgh, Scotland, UK). In conversation with: Rudan, Igor (University of Edinburgh, Scotland, UK). 12 May 2015.

[13] I Rudan JL Gibson S Ameratunga S El Arifeen ZA Bhutta M Black Setting priorities in global child health research investments: guidelines for implementation of the CHNRI Method. Croat Med J 2008 49 720 33 10.3325/cmj.2008.49.720 19090596

[14] R Bahl J Martines N Ali MK Bhan W Carlo KY Chan Research priorities to reduce global mortality from newborn infections by 2015. Pediatr Infect Dis J 2009 28 1 Suppl S43 8 10.1097/INF.0b013e31819588d7 19106763

[15] The Working Group on Priority Setting Priority setting for health research: lessons from developing countries. Health Policy Plan 2000 15 130 6 10.1093/heapol/15.2.130 10837035

[16] S McGregor KJ Henderson JM Kaldor How are health research priorities set in low and middle income countries? A systematic review of published reports. PLoS One 2014 9 e108787 10.1371/journal.pone.0108787 25275315

[17] S Yoshida S Cousens K Wazny KY Chan Setting health research priorities using the CHNRI method: II. Involving researchers. J Glob Health 2016 6 010302 10.7189/jogh.06.010302 27350870

[18] I Rudan The complex challenge of setting priorities in health research investments. Indian J Med Res 2009 129 351 3 19535827

[19] I Rudan S Yoshida KY Chan S Cousens D Sridhar R Bahl Setting health research priorities using the CHNRI method: I. Involving funders. J Glob Health 2016 6 010301 26401269

[20] I Chalmers MB Bracken B Djulbegovic S Garattini J Grant AM Gülmezoglu How to increase value and reduce waste when research priorities are set. Lancet 2014 383 156 65 10.1016/S0140-6736(13)62229-1 24411644

Journal of Global Health (ISSN 2047-2986), Edinburgh University Global Health Society
Designed by