Experiences using systematic review guidelines

Critics of Kitchenham's guidelines for software engineering researchers performing systematic reviews caution researchers to clearly and narrowly define the research questions they will investigate by systematic review, to reduce the overall effort and to improve the quality of the selection of papers and extraction of data.

Abstract

Systematic review is a method to identify, assess and analyse published primary studies to investigate research questions. We critique recently published guidelines for performing systematic reviews on software engineering, and comment on systematic review generally with respect to our experience conducting one. Overall we recommend the guidelines. We recommend researchers clearly and narrowly define research questions to reduce overall effort, and to improve selection and data extraction. We suggest that "complementary" research questions can help clarify the main questions and define selection criteria. We show our project timeline, and discuss possibilities for automating and increasing the acceptance of systematic review.