Research Quality Plus: A holistic approach to evaluating research

A new approach for evaluating research intended for the “real world."
July 06, 2018

In recent years there’s been increasing interest in extending the scope of research evaluation beyond the traditional measures of validity, reproducibility, and fiscal accountability to include the broader impact of research on society — its practicality and utility in the “real world”. Such a view is particularly relevant to IDRC. In the realm of international development, the quality and value of research — in fact, its very purpose — is directly linked to its relevance and translation to practical applications. The hallmarks of successful research for development include the ability to drive and shape policy, to affect communities and social systems, and ultimately to improve the quality of people’s everyday lives.

As a funder and facilitator of research for development throughout the Global South, IDRC set out to develop a more holistic approach to evaluate its research activities. The result is Research Quality Plus (RQ+), an approach that can be applied and adapted to a variety of research contexts and organizational needs.

RQ+ accepts that research doesn’t occur in a vacuum. Instead, RQ+ challenges evaluators to systematically examine the context in which research happens, and consider this context in an assessment of quality. It rejects the convention that research quality rests on methodological excellence alone. Instead, RQ+ requires an evaluator to consider methods and additional dimensions of quality that underpin the greater value of science for society.

Three men on a staircase.
IDRC / Shiho Fukada

The components of RQ+

The real world application of research is a key aspect of the work that IDRC funds. An effective evaluation framework must accommodate research that ranges across many fields (and that is often multidisciplinary) while generating consistent, comparable results.

The RQ+ approach consists of three main components.

The first is the contextual factors that are most likely to affect the quality of research. Identifying these factors allows evaluators to meaningfully and systematically consider how research is being enabled or constrained.  

Depending on an organization’s mandate and priorities, the key influences they identify could vary greatly. For IDRC, the five main contextual factors are:

  • maturity of the research field;
  • research capacity strengthening;
  • risk in the data environment;
  • risk in the research environment; and
  • risk in the political environment.

The second component of RQ+ is a multi-dimensional view of quality in research. Scientific rigour is fundamental, but concepts of quality should include other values and objectives that matter. For IDRC, these included:

  • Research integrity

Technical quality and the appropriateness and rigour of design and execution.

  • Research legitimacy

The extent to which the research accounts for the concerns and insights of relevant stakeholders.

  • Research importance

The importance and value of the knowledge generated by the research for key intended users.

  • Positioning for use

The extent to which the research was managed and products/outputs were prepared to enhance the likelihood that research is used. In other words, for positioning the research findings for influence and eventual impact.

RQ+ infographic
IDRC

The third component of RQ+ is evidence and evaluative rubrics. As with the research we conduct, our judgement of quality must be underpinned by empirical evidence and not just opinion. Using systematic rubrics, RQ+ asks evaluators to draw evidence-based conclusions based on clear judgment criteria. Rubrics specify how performance on the dimensions of quality should be measured.

Putting the approach into practice

In IDRC’s application, conducting an assessment of research quality using RQ+ involves four steps:

  1. Selecting a representative sample of research projects and associated outputs for assessment;
  2. Developing an understanding of how each key contextual factor may impact the quality of the research under evaluation (by examining project documentation, interviewing relevant parties, and considering the expertise of the external review team);
  3. Collecting data and evidence to inform ratings for the dimensions (and sub-dimensions) of quality for each research project, using an evaluative rubric;
  4. Synthesizing the ratings to produce assessments that can be aggregated to program or organizational levels.

Flexibility is an important attribute of RQ+. The approach can be adapted to various research contexts and management needs. RQ+ was initially designed for use in a set of external IDRC program reviews undertaken in 2015. Since then, RQ+ has been used, and often adapted, in a number of ways.

Broader potential

IDRC’s use of the approach has highlighted the broader potential for its use. Evaluations using RQ+ are based on priorities, values, and principles that can be used to facilitate matchmaking and coordinated planning among funding or research partners.

The framework’s evaluative criteria can contribute to the development and assessment of research proposals and the ongoing assessment of projects and programs throughout their lifecycles. It can also facilitate meta-analysis, a technique that collates the results of multiple scientific studies into a single record to which statistical methods are applied — this can increase the precision and generalizability of results.

Furthermore, RQ+ could contribute to a deeper understanding of how research impact is achieved. By combining information gathered through the evaluation process with an examination of longer-term research impacts, the research and contextual factors that produce real-world results may become clearer. These insights could help shape the practice of research for development and improve the potential for long-term positive change.

Learn more about RQ+

Read our position paper

Learn the ins and outs of the RQ+ evaluation framework

IDRC / Vijay Kutty