Systematic reviews aim to answer specific research questions, by collating all evidence that fits predefined criteria. To ensure high quality and minimize bias, systematic reviews are carried out by using systematic, transparent and precise methods (ref. 1 - 2).
This includes a clarification of the subject, and if similar reviews exist.
Various models exist to help formulating the research question (conceptualizing models). To prepare the review protocol, tools are available, please see Research questions and protocol.
A systematic literature search is an essential part of a systematic review, and forms the basis of the collection of data, please see Literature search.
The screening and selection of studies for further analysis is based on the predefined eligibility criteria.
The quality of the evidence is appraised according to the type of study, please see Appraisal of the literature.
The evidence synthesis includes a qualitative analysis (a methodological description of the included studies, strengths and weaknesses (please see (ref. 1)), and if possible, a quantitative meta-analysis.
The research question is pivot for a systematic review, and essential for the preparation of the systematic search strategy (ref. 5). Conceptualizing models can be used to help formulating a clear and structured research question. The choice of model to use, depends on the specific subject:
The PICO model is the most commonly used model to help formulating clinical research questions (ref. 1). PICO is useful for framing clinical research questions that have do with the effect of interventions, and The Cochrane Handbook refer to this model (ref. 6).
The CoCoPOP model can be used when formulating a research question related to the incidence or prevalence of a disease, symptom, health condition, etc. (ref. 6a). Co (Condition), Co (Context) POP (Population)
The model describes four elements of a focused clinical research question:
P (Patient / Problem / Population), I (Intervention), C (Comparator) og O (Outcome) (ref. 5, 7).
This PICO scheme can be used to help formulating your research question, and structure your literature search.
PICO can be extended to PICOT, if T (Timing / Type of studie) or PICOS if S (Setting) is crusial.
The PICo model can be used, when a particular phenomenon is interesting in relation to a patient group, for instance in a particular context: P (Patient/problem/population), I (Phenomenon of Interest) and Co (Context) (ref. 8).
This PICo scheme can be used to help formulating your research question, and structure your literature search.
The PIRD model can be used to formulate a research question that addresses diagnostic test accuracy. (ref . 8a) P (Population), I (Index test), R (Reference test), D (Diagnosis of interest).
This model can be used as a starting point when formulating qualitative research questions: P (Population), E (Exposure) and O (Outcome) (ref. 9 – 10). PEO can help structuring and limiting the search for qualitative studies.
The SPICE model is formulated as follows: S (Setting), P (Population), I (Intervention), C (Comparison) and E (Evaluation) (11).
The SPIDER model is a second model to help framing qualitative research questions, and is formulated as follows: S (Sample), PI (Phenomenon of Interest), D (Design), E (Evaluation) and R (Research type) (ref. 12).
According to Cochrane Handbook, a protocol is a structured plan that must be followed in a study. As part of the preparation of a systematic review (and to minimize bias in the process), the most appropriate is to devise a protocol that describes the rationale, hypothesis, where the literature has been searched and what additional methods that are going to be used when preparing the systematic review (ref. 13).
Several tools exist to help preparing a protocol (overview from PRISMA (ref. 14)):
- PRISMA for systematic review protocols (PRISMA-P)
- Institute of Medicine: STANDARD 2.6 Develop a systematic review protocol
- Cochrane Handbook Chap. 4 (ref. 15)
Protocols for systematic reviews with a health-related outcome can be registered in PROSPERO. If the systematic review and thus the protocol does not fall within the applicable requirements of PROSPERO, a protocol can be prepared based on PRISMA-P. The protocol can then be uploaded to a freely accessible place, e.g. a web page or Open Science Framework (OSF) Registries.
In principle, a systematic literature search does not differ from a comprehensively conducted literature search. It is recommended that the literature search has a high recall (recall, a theoretical measure that indicates how much of the relevant literature in a database that has been identified in the literature search (ref. 19)), this may result in a low precision (precision, how much of the found literature of which is relevant) (ref. 20).
In practice this means that when you do a comprehensive, systematic literature search, which aims to collect as much evidence / literature as possible, this will result in some irrelevant literature as well (low precision).
Preparation of the systematic literature search
In order to conduct a systematic literature search, it is important to group / structure your search in correct search blocks (see the Literature search guide), so you can start preparing lists of search terms. If you have used a conceptualizing model (e.g. the PICO model), this may help to identify the most important elements to include in your search.
The preparation of the actual systematic literature search requires the knowledge of complex literature search, including subject headings, indexing, free text searching and search technical methods (see the Literature searching manual and Advanced search techniques manual) – as well as how to search the individual databases. There is a lot of work involved in preparing the search, before you can execute the final search and move on with the search result. Bramer et al.describes how to plan and perform a systematic literature search step by step (REFERENCE: Bramer, W.M., et al., A systematic approach to searching: an efficient and complete method to develop literature searches. J Med Libr Assoc, 2018. 106(4): p. 531-541.).
It is important to critically appraise the specific search strategy, and PRESS 2015 Guideline Evidence-Based Checklist (published in McGowan et al. (ref. 21)) can be used to this.
As previously mentioned, the validity and reproducibility of the systematic search is essential. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) has developed a PRISMA for Searching - a checklist with 16 reporting items that should be reported when reporting a systematic literature search (REFERENCE:Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, Koffel JB; PRISMA-S Group. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021 Jan 26;10(1):39. doi: 10.1186/s13643-020-01542-z.).
The checklist ensures that what should be included in a systematic search is reported and clear, so that the search is valid, reproducible, and transparent. The checklist can thus also be used prospectively, i.e., in the planning of the systematic search.
Databases and additional information sources
In addition to conduct the literature search technically correct, it is important to search multiple databases and information sources, to identify as many relevant studies as possible (ref. 20). Choice of databases and information sources to search depends on the specific subject.
The most widely used bibliographic databases within health science include: PubMed / Medline, Embase and Cochrane Library.
Subject specific databases
Subject specific (bibliographic) databases with more narrow focus exists, e.g. CINAHL (Cumulative Index to Nursing and Allied Health), PsycInfo and AMED (The Allied and Complementary Medicine Database) (see health scientific databases for an overview of databases).
Cross-disciplinary search engines
The cross-disciplinary search engines include: Science of Index (Expanded), Scopus and Google Scholar. Web of Science and Scopus (to a lesser degree PubMed) register citations and can be used to make a forward-looking citation search, identifying which publications have cited a given relevant publication (ref. 21).
In addition to the traditional databases and search engines, there are other sources of information and other types of literature that may be relevant to search, when preparing a systematic review:
Clinical trial records
Clinical trial records register clinical trial information, and may be an important sources of information if you want to identify, for example, randomized clinical trials (RCTs)(ref. 22) for example ClinicalTrials.gov .
Gray literature refers to literature that has not been completed for publication (ref. 23), and includes: research reports, dissertations, conference contributions, etc. Some of the most widely used databases and search engines contain some conference contributions, including Embase, Scopus and Web of Science. There are also spefific databases with gray literature, among other things. OpenGrey and Gray Matters.
A preprint refers to a version of a scientific article that precedes peer review and publication. You can search relevant preprint servers to identify manuscripts and data that have not yet been formally published or accepted for publication and thus not searchable in traditional bibliographic databases, such as gray literature or elsewhere. Depending on the subject area, there are e.g. medRXV (health sciences), bioRxiv (biology) and SocArXiv (social sciences).
Assessment of diagnostic primary studies: QUADAS-2
Assessment of prognostic primary studies: QUIPS
Assessment of qualitative studies: No specific recommendations, see Cochrane Collaboration´s guidelines
Several organizations have taken GRADE in use, including Cochrane Collaboration, BMJ Clinical Evidence, the National Institute for Health and Clinical Excellence (NICE), UpToDate®, etc. The GRADE Working Group.
CASP Checklists: Critically Appraised Skills Programme includes tools for critically evaluating a variety of study types, including: economic evaluations.
Reference Management Tools
Reference management tools, such as Endnote, help to keep track of references and are very relevant to use when preparing reviews. Endnote can (as well can Covidence) be used to delete the duplicates that occur when searching multiple databases. Read more about Endnote and find guides here.
The software platform Covidence, is a reference management tool developed for the preparation of systematic reviews. The program can be used for sorting literature, quality assessment and data extraction in connection with systematic reviews. Read more about Covidence and find instructions for creating user access here.
The university Library offers courses on Endnote and Covidence. Find a list of relevant courses here.
There is a large amount of literature on systematic reviews, among which the following can be recommended:
Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021. Available from www.training.cochrane.org/handbook.
Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press, link.
Aromataris E, Munn Z (Editors). Joanna Briggs Institute Reviewer's Manual. The Joanna Briggs Institute, 2017:, link.
(ref. 1) Green S, Higgins JPT, Alderson P, Clarke M, Mulrow CD, Oxman AD. Chapter 1: Introduction. In: Higgins JPT, Green S (editors), Cochrane Handbook of Systematic Reviews of Interventions. Chichester (UK): John Wiley & Sons (2008).
(ref. 2) Campbell Collaboration. Website. (last visited 9.8.2018).
ref. 3) Institute of Medicine of the National Academies. Standards for Systematic Reviews (2011). Link.
(ref. 4) O´Connor D, Green S, Higgins JPT (editors). Chapter 5: Defining the review question and developing criteria for including studies. In: Higgins JPT, Green S (editors), Cochrane Handbook of Systematic Reviews of Interventions. Chichester (UK): John Wiley & Sons (2008
(ref. 5) Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. JMLA (2018), 106(4): 420-431.
(ref. 6) O´Connor D, Green S, Higgins JPT (editors). Chapter 5: Defining the review question and developing criteria for including studies. In: Higgins JPT, Green S (editors), Cochrane Handbook of Systematic Reviews of Interventions. Chichester (UK): John Wiley & Sons (2008).
(ref. 6a) Munn Z., Moola S., Lisy K, Riitano D., Tufanaru C. Methodological guidance for systematic reviews of observational epidemiological studies reporting prevalence and cumulative incidence data. Int J Evid Based Healthc. 2015 Sep;13(3):147-53.)
(ref. 7) Frandsen TF, Dyrvig AK, Christensen BC, Fasterholdt I, Ølholm AM. En guide til valide og reproducerbare systematiske litteratursøgninger. Ugeskrift for Læger (2014), 176(7):647- 651.
(ref. 8) Joanna Briggs Institute 2011. Joanna Briggs Institute Reviewers' Manual 2011. The University of Adelaide, South Australia, 2011.
(ref. 8a) Campbell J, Klugar M, Ding S, Carmody DP, Hakonsen SJ, Jadotte YT, White S, Munn Z. Diagnostic test accuracy: methods for systematic review and meta-analysis. Int J Evid Based Healthc. 2015 Sep;13(3):154-62.).
(ref. 9) Khan KS, Kunz R, Kleijnen J, Antes G. Systematic Reviews to Support Evidence-Based Medicine. How to Review and Apply findings of Health Care Research. London: RSM Press, 2003.
(ref. 10) Bettany-Saltikov J. How to do a systematic literature review in nursing: a step-by-step guide. Second edition ed. London: McGraw Hill Open University Press; 2016.
(ref. 11) Booth A. Clear and present questions: formulating questions for evidence based practice. Libr Hi Tech. 2006;24(3):355-68.
(ref. 12) Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012; 22(10):1435-43.
(ref. 13) Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021, link.
(ref. 14) PRISMA (last visited 9.10.2018).
(ref. 15) McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefevbre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol 2016, 75: 40-46.
(ref. 16) Koffel JB, Rethlefsen ML. Reproducibility of Search Strategies Is Poor in Systematic Reviews Published in High-Impact Pediatrics, Cardiology and Surgery Journals: A Cross-Sectional Study. PLoS One. (2016); 11(9):e0163309.
(ref. 17) Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. J Med Libr Assoc. (2016); 104(4):267-277.
(ref. 18) Faggion, CM, Jr, Huivin R, Aranda L, Pandis N, Alarcon M. The search and selection for primary studies in systematic reviews published in dental journals indexed in MEDLINE was not fully reproducible. J Clin Epidemiol (2018); 98: 53-61.
(ref. 19) Frandsen TF, Dyrvig AK, Christensen JB, Fasterholdt I, Ølholm AM. En guide til reproducerbare og systematiske litteratursøgninger. Ugeskr Læger (2014); 176: V02130141.
(ref. 20) Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane, 2019. Link
(ref. 21) McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefevbre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol (2016); 75: 40 – 46.
(ref. 22) Baudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ (2017); 356: j448.
(ref. 23) Informationsordbogen, ordbogsopslag: Grå litteratur. Website (last visited 17.10.2018).
(ref. 24) The Method Handbook - Model for preparing national clinical guidelines. The National Board of Health (2017), version 3. Only available in Danish.
(ref. 25) BMJ Best Practice. What is GRADE. (last visited 25.10.2018).