Finding the best research method to answer your research question(s)

With my dissertation, I aim to provide a state-of-the-art set of answers to the overarching question of what would be best practices for API documentation. 

To find answers, I wanted to review the body of knowledge from technical communication as well as software documentation. I wanted to identify studies that could provide reliable answers. So, I talked to one of my professors about my idea, and she acknowledged that it would be a suitable topic for a dissertation.

The systematic literature review

During my first broad reading, I came across articles that were “systematic literature reviews”, and I figured that this described pretty well what I intended to do.

The doubt

Later, when I had to lay down a research proposal for my dissertation, I came back to that “systematic literature review” method, read more about it, and decided that this would be an appropriate method to deal with my research topic. However, I was not convinced.

Did I decide on that method because it was the right one to answer my research questions? Or did I decide on it because of lack of other options?

Suddenly, I felt very insecure. In class, we had read a lot about research methods during the last semester. We needed a profound knowledge of research methods etc. to conduct our usability tests and write the corresponding research reports. However, we were still beginners. How would I be able to know all available research methods for working with literature only? 

One working-with-literature-only method that I knew was called content analysis. But this was not what I was looking for. My topic did not require evaluating bodies of texts regarding, for example, their structure or terminology. 

What I needed was a method to extract practical advice from the research that had been conducted on API documentation. To do that, I wanted to look at more than one study and, where appropriate, compare and contrast results of similar studies.

Recently, my supervisor contacted me and advised me on issues in the research proposal that I had submitted. She also sent me an article from a journal relevant to my field. This article described The Systematic Literature Review as a Research Genre.

The article

Spoiler alert: This article convinced me to stop doubting my decision. 

It convinced me that I would not need to further investigate the vast amount of literature on research methods. I would not need to detect and evaluate all the available research methods for my topic/ my approach (i.e., working with literature rather than conducting a study). 

It convinced me that I had already found the appropriate research method. This is why:

• Interdisciplinary research? Yes, as I will search for resources in at least the fields of technical communication and software engineering, and I remain open to whatever other field comes up during further reading.
• Focus on the implications for practitioners? Yes, as I want to provide knowledge for practitioners based on research results.

The article further specified the weaknesses of the approach, and I could immediately relate to quite a few of them because I came across them through my first broad reading, too.

• Keywords are not used consistently. (I had found literature for my topic under “reference documentation” instead of “API documentation”.)
• Titles and abstracts are not crafted consistently. I had to expand my search fields to other parts of a reference than just title, author, and abstract. As a consequence, I ended up with way more search results which often were not even relevant for my research question. They only came up because of their reference lists that included some works dealing with “API documentation”. However, I rather wanted to sort more search results than missing an interesting article.
• Different databases have different search functions. I could tell. The article showed approaches to deal with that in a systematic literature review.

The “Aha!” moment

The main insight I got from the article was a simple explanation of how to accomplish the “data extraction” from the articles that made it to the “corpus” (i.e., the selection of articles to be examined):

“We accomplished this by specifying a column in our Excel spreadsheet for each feature of interest.”

This was the key sentence that made me understand the method and removed my insecurity. Yes, I had found the right research method for my research topic. I could already picture myself creating those columns in an Excel sheet and filling the cells with the answers from the articles.

The next step

Now, that I am convinced about which research method to employ, I have to elaborate the next iteration(s) of my research questions. I have to read more, refine the research questions more, search more, check the search results more, and so on.

I am still in the planning phase, where I try to find research questions that are as un-biased and precise as possible. This is important because my research design will determine the corpus that I then can investigate for results and advice. I want to come up with the best possible corpus in order to end up with the best possible answers. I am looking forward to the upcoming refinements.