Why do we all use the same methods in Work and Organizational Psychology?
“Oh god no… not another moderated mediation model”
During several conferences the past years, I heard people silently uttering these words as they listened to a presentation. Even though I presented papers at conferences in which I used moderated mediation models myself, I could understand the sentiment. Many presentations at conferences resemble each other, because they use similar methods, designs, and analytical approaches. Not surprisingly, some conference attendants—including myself— started to grow bored of research that seemed to have reached a stalemate.
Do we all use the same methods in Work and Organizational Psychology? Of course, there is variety—for one, we do not all use moderated mediation models. But that being said, there is some evidence suggesting that there is considerable methodological isomorphism in our field. For example, Casper and colleagues (2007) reviewed the work-family literature and documented the variety in research methods used. They showed that 89% of all studies used cross-sectional designs, 97% were field studies, and 89% were correlational studies. Longitudinal designs, experiments, and qualitative approaches were substantially underrepresented in the work-family literature. This situation is not limited to the work- family literature of course; it is likely that these problems emerge in other domains in Work and Organizational Psychology as well.
There are several reasons that could explain the existence of methodological isomorphism in our field. First, we tend to train PhD students in a limited set of research methods and analytical techniques. Most of the time, these methods and techniques are chosen because they are commonly used in the field, thus perpetuating the problem. Second, researchers may believe that using certain method may improve their chances of getting published. Third, reviewers acting as gatekeepers can be more critical of methods or techniques that they are not familiar with. As a result, manuscripts using non-traditional methods may have more difficulties getting published.
One might wonder if methodological isomorphism is problematic. After all, this may just reflect a field converging to set of methods that have proven their worth, with inferior methods no longer being used. Call it methodological evolution if you will. However, the methods and techniques that are commonly used are not necessarily optimal (e.g., cross- sectional designs are in many cases suboptimal). Moreover, methodological isomorphism may carry certain dangers. For one, it may lead to researchers choosing a research question that fits a certain method or technique, rather than picking the method which they believe is most appropriate for their research question. Moreover, by relying on a limited set of methods and analytical techniques, we may never fully come to understand workplace phenomena.
We discussed this issue of methodological isomorphism during the Future of Work and Organizational Psychology day at the 2019 EAWOP conference in Turin, Italy (http://eawop2019.org). The Future of Work and Organizational Psychology movement strives to build a better future for our field (https://www.futureofwop.com; click here for a manifesto on the future of WOP). We believe that our field can improve in several areas, including our research, our teaching, and the way that we organize academia. One of the workshops that we organized during the Future of Work and Organizational Psychology day focused on methods and aimed to tackle the issue of methodological isomorphism. We departed from the idea that we should use diverse methods and analyses to explore research questions if we want Work and Organizational Psychology to be a robust science (Grand et al., 2018).
Participants of the workshop took part in two exercises to come up with possible solutions to tackle the issue of methodological isomorphism. The result of this brainstorming exercise was a list of 9 actions (click here to see slides of workshop):
- Use existing platforms to stimulate collaboration and trust
- Tackle your next research question with a group of people and multiple methods
- Think more thoroughly about research designs and consider applying multiple approaches
- Encourage mixed-methods research by creating a mixed-methods journal
- Involve statisticians/methodologists as full partners in research projects
- Train students in qualitative research methods and in assessing good qualitative research
- Create an online platform to get feedback on study designs from the broad scientific community and to find potential collaborators
- Create a community of methods experts, similar to Researchgate
- Start ManyLab type collaborations with multiple registered report studies that tackle a single research question using various methods
What was striking to me during this workshop, was that participants acknowledged that we need more collaborations to tackle the issue of methodological isomorphism. In particular, substantive-methodological synergies should be established, meaning that substantive experts should ideally collaborate with methodological experts. However, we often lack the resources to start such collaborations. For example, we might not know who has expertise with a certain analytical approach or it may be necessary to develop trust before one can approach potential collaborators. In addition, it was clear that participants acknowledged that we should think more carefully about our research designs. Open science practices may help here: by submitting research as registered reports to journals, researchers get valuable feedback on their methods before data is collected. The list of journals that accept registered reports is growing, although the number of Work and Organizational Psychology journals on the list still remains fairly limited (click here for overview of journals that accept registered reports).
The Future of Work and Organizational Psychology initiative will try to use the actions that came out of this workshop as input to take further steps. In particular, we aim to take actions to stimulate substantive-methodological synergy in our field. This could be done by organizing workshops or summer schools in which researchers can be introduced to a broad variety of methods and that would allow substantive and methodological experts to meet each other. In addition, specific workshops could be set up in which we introduce researchers to registered reports and in which we stimulate collaborative registered reports. These collaborative registered reports consist of three stages. In the first stage, a research question is submitted to a platform and researchers can opt in to tackle this research question jointly. Next, a registered report application is written and submitted to a journal. These applications could include multiple approaches (e.g., qualitative approach, experimental approach, experience sampling approach) to address the same research question. Finally, data is collected following in-principle acceptance.