By Andres Navarrete

We know why mixing qualitative and quantitative research methods is important but we need to discuss cost-effective ways to mix methods during evaluation.

In an article from the World Bank, The Mystery of Mixing Methods, Jos Vaessen outlines five functions of mixed methods research (or research that combines qualitative and quantitative methods).

After presenting five possible reasons we would conduct mixed-methods research, the article challenges us to find practical ways to take advantage of combining these methods. In this post, I want to discuss what these practical ways may be.

The five objectives that were identified by Vaessen were triangulation, complementarity, development, initiation, and expansion:

• Triangulation refers to using different forms of data and checking if they yield similar conclusions.
• Complementarity seeks to highlight a different aspect of phenomena using different data types.
• Development refers to using results from a method to develop a new method or research tools.
• Initiation is used to get started on new topics by challenging existing ones.
• Continuation seeks to find out new information on a given phenomena after previous information was found.

Let’s discuss ways we can practically implement these in monitoring and evaluation (especially when time and money are tight):

1. Conduct Two Phases for Field Data Collection (not just one).
2. Investigate what social norms exist and evaluate their prevalence through a later survey.
3. Identify possible factors affecting experimental comparisons between treatment and control groups.
4. Investigate mechanisms of change and adapt activities accordingly.
5. Use open-ended questions strategically in reports or surveys.
6. Evaluate the quality of your qualitative sampling through non-representative sampling.

1. Conduct Two Phases for Field Data Collection (not just one)

Qualitative tools and quantitative surveys are usually designed at the same time, so that we may start collecting data as soon as possible. 

This could be a missed opportunity to use mix-methods to initiate new topics, challenge existing ones, or develop new tools based on findings.

Having (at least) two phases of data collection (or visiting sampling sites twice within the same evaluative period) solves this issue because we can:

• Pilot research tools: check their form, length, difficulty, and appropriateness of concepts and translations.
• Identify relevant points of view: learn who exactly you should be speaking with during data collection and adapt research tools accordingly. This makes it easier to develop accurate sampling plans later on.
• Organize snowball sampling: once points of view are identified, use the time between phases to organize interviews with participants referenced by other participants.
• Develop research team’s capacity: identify strengths and weaknesses in your research team early on.
• Explain puzzling or seemingly counterintuitive findings: Often we find different quantitative results to what we expect and need to explore why this is the case through qualitative research.

A phase 1 can be small or substantial, depending on your project needs and resources. Often one or two focus groups work just fine. When resources are limited, you can plan to recruit researchers living near sampling sites, so budgets can easily be split. What’s important is to allocate sufficient time between phase 1 and 2 to be able to process and reflect on research findings.

2. Investigate what social norms exist and evaluate their prevalence through a later survey.

Focus group discussions are a great way to obtain a wide range of opinions in a short space of time. They are ideally suited to discuss how people perceive certain topics or behaviours and what social norms exist.

You can use focus groups to find out what norms exist and then measure their prevalence through a probability survey.

For example, if you are leading an HIV-prevention initiative, you can organize a focus group discussion to discuss what norms exist about key behaviors like condom-use, and then measure how prevalent these opinions are in program sites. Based on this information, this initiative can tailor health promotion strategies and messages accordingly.

3. Identify possible factors affecting experimental comparisons between treatment and control groups.

In her article ‘The Promising Integration of Qualitative Methods and Field Experiments‘ by Elizabeth Levy Paluck, she identifies ways qualitative research can be integrated into field experimentation. One of such ways (and the one I consider most practical), is to explore the contextual nature and heterogeneity of causal effects within each experimental group.

In other words, through qualitative methods, we can explore how certain contextual differences can mediate the effects of a given project and include these differences a control variables in a given impact regression model. An obvious example would be the effects of natural disasters or social policies affecting one group more than the other, which can explain why differences between treatment and control groups do not occur as expected.

Qualitative research helps us appraise how truly comparable experimental groups may be, and help us control for unobserved factors after these groups are selected.

4. Investigate mechanisms of change and adapt activities accordingly.

Theories of change exist to justify that the impact of a project was not the result of random occurrence but the result of a particular development project. Even when field experimentation can isolate the impacts of a project (through randomized control trials or difference-in-difference techniques), such experiments are only justified when there is a plausible causal explanation in the form of a theory of change.

These causal explanations may be found first through qualitative research and later tested through field experiments. This helps understand how or why specific project activities have certain outcomes, who benefits the most from them, what unforeseen outcomes could exist, and how can activities be modified for greater impact.

5. Use open-ended questions strategically in reports or surveys.

While most of a survey questionnaire is devoted to closed-ended questions (that is, questions that produce a single response -for example, ‘yes’ or ‘no’), a few open-ended questions can be integrated into surveys to explore new topics that can be explored further through follow-up focus group discussions and in-depth interviews.

For example, an education survey can ask a student if they “missed school for more than two weeks in the past month” (yes/no closed-ended) and then follow-up by asking “why?” in a follow-up question.

This opens up new possibilities for research. We can use answers to that question in a next survey as multiple-choice responses, or initiate new topics in focus group discussions.

6. Evaluate the quality of your qualitative sampling through non-probability sampling.

Qualitative methods excel when there is a diversity of participants and opinions captured by the research. After all, if we want to get a range of opinions about a given topic, it is important to know just how large that range of opinions is.

A non-probability sample is a non-representative type of sample that does not aim to make claims about a given population but, instead, help us describe how diverse a group truly is.

To make use of this technique, we can apply a brief survey at the end of focus group discussions to learn differences in age, sex, or backgrounds of participants is present and appraise just how much data saturation we are able to reach with those that participated. This helps to understand if there were any particular group whose views are not present in the study and organize further in-depth interviews or focus group discussions accordingly. Results can also point at the quality of the sampling method, adding credibility to it when we can show that all relevant groups participated in the study.

Conclusion

There are several ways we can mix research methods and we can think creatively about strengthening our claims by doing so. Not all the ways we discussed here might be feasible for all monitoring and evaluation efforts, but a few might be implementable depending on what level of resources or time you are able to spend.

There are probably other ways not envisaged here and I would love to hear them in your comments. My intention was to get the conversation started, so please let me know what else we should consider!

References

Levy Paluck, E. (2010). The promising integration of qualitative methods and field experiments. The ANNALS of the American Academy of Political and Social Science, 628(1), 59-71.

Vaessen, J. (2017) The Mystery of Mixing Methods. World Bank group.