Systematic reviews aim to answer specific research questions, by collating all evidence that fits predefined criteria. To ensure high quality and minimize bias, systematic reviews are carried out by using systematic, transparent and precise methods (ref. 1 - 2).
Overview – the different steps of a systematic review
The process of doing a systematic review consist in several (possible) steps. Depending on the review subject, a quantitative analysis (meta-analysis) is included. The overview below highlights the main steps involved in doing systematic reviews (more or less steps can be included in the process) (ref. 3):Defining the aim and scope
This includes a clarification of the subject, and if similar reviews exist.Preparation of the research question and review protocol
Various models exist to help formulating the research question (conceptualizing models). To prepare the review protocol, tools are available, please see Research questions and protocol.Description of inclusion and exclusion criteriaSystematic reviews are characterized by clearly predefined quality criteria that describes which studies to include or exclude (ref. 4).Literature search
A systematic literature search is an essential part of a systematic review, and forms the basis of the collection of data, please see Literature search.Sorting and selection of the literature
The screening and selection of studies for further analysis is based on the predefined eligibility criteria.Extraction of data and critical appraisal of included studies
The quality of the evidence is appraised according to the type of study, please see Appraisal of the literature.Evidence synthesis
The evidence synthesis includes a qualitative analysis (a methodological description of the included studies, strengths and weaknesses (please see (ref. 1)), and if possible, a quantitative meta-analysis.
Research question and protocol
The research question is pivot for a systematic review, and essential for the preparation of the systematic search strategy (ref. 5). Conceptualizing models can be used to help formulating a clear and structured research question. The choice of model to use, depends on the specific subject:
The PICO model is the most commonly used model to help formulating clinical research questions (ref. 1). PICO is useful for framing clinical research questions that have do with the effect of interventions, and The Cochrane Handbook refer to this model (ref. 6).
The CoCoPOP model can be used when formulating a research question related to the incidence or prevalence of a disease, symptom, health condition, etc. (ref. 6a). Co (Condition), Co (Context) POP (Population)
The model describes four elements of a focused clinical research question:
P (Patient / Problem / Population), I (Intervention), C (Comparator) og O (Outcome) (ref. 5, 7).
This PICO scheme can be used to help formulating your research question, and structure your literature search.
PICO can be extended to PICOT, if T (Timing / Type of studie) or PICOS if S (Setting) is crusial.
The PICo model can be used, when a particular phenomenon is interesting in relation to a patient group, for instance in a particular context: P (Patient/problem/population), I (Phenomenon of Interest) and Co (Context) (ref. 8).
This PICo scheme can be used to help formulating your research question, and structure your literature search.
The PIRD model can be used to formulate a research question that addresses diagnostic test accuracy. (ref . 8a) P (Population), I (Index test), R (Reference test), D (Diagnosis of interest).
This model can be used as a starting point when formulating qualitative research questions: P (Population), E (Exposure) and O (Outcome) (ref. 9 – 10). PEO can help structuring and limiting the search for qualitative studies.
The SPICE model is formulated as follows: S (Setting), P (Population), I (Intervention), C (Comparison) and E (Evaluation) (11).
The SPIDER model is a second model to help framing qualitative research questions, and is formulated as follows: S (Sample), PI (Phenomenon of Interest), D (Design), E (Evaluation) and R (Research type) (ref. 12).
ProtocolAccording to Cochrane Handbook, a protocol is a structured plan that must be followed in a study. As part of the preparation of a systematic review (and to minimize bias in the process), the most appropriate is to devise a protocol that describes the rationale, hypothesis, where the literature has been searched and what additional methods that are going to be used when preparing the systematic review (ref. 13).
Several tools exist to help preparing a protocol (overview from PRISMA (ref. 14)):
- PRISMA for systematic review protocols (PRISMA-P)
- Institute of Medicine: STANDARD 2.6 Develop a systematic review protocol
- Cochrane Handbook Chap. 4 (ref. 15)
Protocols for systematic reviews with a health-related outcome can be registered in PROSPERO. If the systematic review and thus the protocol does not fall within the applicable requirements of PROSPERO, a protocol can be prepared based on PRISMA-P. The protocol can then be uploaded to a freely accessible place, e.g. a web page or Open Science Framework (OSF) Registries.
Systematic literature search
The literature search aims to collect all evidence that forms the basis of the systematic review, and is therefore a very important and essential part of a systematic review. The literature search therefore potentially has a large impact of the quality of the systematic review (ref. 16). Several studies indicate that the quality and reproducibility of literature searches in systematic reviews often are of a low quality (ref. 17-19).
In principle, a systematic literature search does not differ from a comprehensively conducted literature search. It is recommended that the literature search has a high recall (recall, a theoretical measure that indicates how much of the relevant literature in a database that has been identified in the literature search (ref. 20)), this may result in a low precision (precision, how much of the found literature of which is relevant) (ref. 21).
In practice this means that when you do a comprehensive, systematic literature search, which aims to collect as much evidence / literature as possible, this will result in some irrelevant literature as well (low precision).
Principles for a systematic literature search
The validity and reproducibility of the systematic literature search is, as mentioned, crucial, and therefore, it is recommended to prepare a search protocol. According to Frandsen et al., a search protocol should describe what methods are used in the literature search and act as documentation for the process, thereby contributing to the reproducibility of the literature search. A search protocol may advantageously include the following (ref. 20):
1. Background and thesis
2. The focused search question
3. Inclusion and exclusion criteria
4. Databases and informations sources that have been searched
5. Seach history and date for searching
6. Strategy for screening of literature and strategy for quality assessment of the included literature
Preparation of the systematic literature search
In order to conduct a systematic literature search, it is important to group / structure your search in correct search blocks (see the Literature search manual), so you can start preparing lists of search terms. If you have used a conceptualizing model (e.g. the PICO model), this may help to identify the most important elements to include in your search.
The preparation of the actual systematic literature search require the knowledge of complex literature search, including subject headings, indexing, free text searching and search technical methods (see the Search technique and indexing manual)
It is important to critically appraise the specific search strategy, and PRESS 2015 Guideline Evidence-Based Checklist (published in McGowan et al. (ref. 22)) can be used to this.
Databases and additional information sources
In addition to conduct the literature search technically correct, it is important to search multiple databases and information sources, to identify as many relevant studies as possible (ref. 21). Choice of databases and information sources to search depends on the specific subject.
The most widely used bibliographic databases within health science include: PubMed / Medline, Embase and Cochrane Library.
Subject specific databases
Subject specific (bibliographic) databases with more narrow focus exists, e.g. CINAHL (Cumulative Index to Nursing and Allied Health), PsycInfo and AMED (The Allied and Complementary Medicine Database) (see health scientific databases for an overview of databases).
Cross-disciplinary search engines
The cross-disciplinary search engines include: Science of Index (Expanded), Scopus and Google Scholar. Web of Science and Scopus (to a lesser degree PubMed) register citations and can be used to make a forward-looking citation search, identifying which publications have cited a given relevant publication (ref. 21).
In addition to the traditional databases and search engines, there are other sources of information and other types of literature that may be relevant to search, when preparing a systematic review:
Clinical trial records
Clinical trial records register clinical trial information, and may be an important sources of information if you want to identify, for example, randomized clinical trials (RCTs)(ref. 23): ClinicalTrials.gov and WHO ICTRP.
Gray literature refers to literature that has not been completed for publication (ref. 24), and includes: research reports, dissertations, conference contributions, etc. Some of the most widely used databases and search engines contain some conference contributions, including Embase, Scopus and Web of Science. There are also spefific databases with gray literature, among other things. OpenGrey and Gray Matters.
A preprint refers to a version of a scientific article that precedes peer review and publication. You can search relevant preprint servers to identify manuscripts and data that have not yet been formally published or accepted for publication and thus not searchable in traditional bibliographic databases, such as gray literature or elsewhere. Depending on the subject area, there are e.g. medRXV (health sciences), bioRxiv (biology) and SocArXiv (social sciences).
The evidence of included studies must be assessed based on the study type. The Code of Practice for the Preparation of National Clinical Guidelines (The Danish Health Authorities) (ref. 25) contains recommendations for evaluating the methodical quality of studies when making evidence synthesis.Assessment of guidelines: AGREE IIAssessment of systematic reviews: AMSTARAssessment of RCTs: Cochrane Risk of Bias ToolAssessment of observational studies (cohort studies, case-control studies): ROBINS-I
Assessment of diagnostic primary studies: QUADAS-2
Assessment of prognostic primary studies: QUIPS
Assessment of qualitative studies: No specific recommendations, see Cochrane Collaboration´s guidelinesGRADE (The Grades of Recommendation, Assessment, Development and Evaluation) is a system for evaluating the quality of evidence for outcomes for systematic reviews and guidelines. The overall evidence is evaluated on 5 parameters:Risk of bias
Publication biasThis results in an overall quality assessment from 'high - moderate - low - very low' (ref. 26).
Several organizations have taken GRADE in use, including Cochrane Collaboration, BMJ Clinical Evidence, the National Institute for Health and Clinical Excellence (NICE), UpToDate®, etc. The GRADE Working Group.
CASP Checklists: Critically Appraised Skills Programme includes tools for critically evaluating a variety of study types, including: economic evaluations.
Literature search help
There are various options for literature search help:
- Guides and instructions for the databases and for doing literature searches are available from the health sciences libguide or on the library's general overview of guides and instructions.
- The University Library of Southern Denmark also offers several free courses in literature search and reference management.
- In addition, there may be an opportunity for individual guidance (book a librarian) within literature search regarding major assignments / projects. Read more about the book a librarian arrangement here.
- The PhD School of Health Sciences at SDU also offers PhD courses in literature search, see an overview of all PhD school courses here.
For questions and guidance related to evidence-based methodology in health-related research (including systematic reviews and meta-analyses), please contact: Centre for Evidence-Based Medicine Odense (CEBMO).
University Library of Southern Denmark can perform literature searches as consultancy services.
Please contact us for further information:
Mette Brandt Eriksen
Research Librarian, PhD
T 65 50 83 55
Reference management and sorting of literature
There are several tools that can contribute to reference management and sorting of literature in the preparation of reviews.
Reference Management Tools
Reference management tools, such as Endnote, help to keep track of references and are very relevant to use when preparing reviews. Endnote can also be used to delete the duplicates that occur when searching multiple databases. Read more about Endnote and find guides here.
The software platform Covidence, is a reference management tool developed for the preparation of systematic reviews. The program can be used for sorting literature, quality assessment and data extraction in connection with systematic reviews. Read more about Covidence and find guidance: https://libguides.sdu.dk/Sundhedsvidenskab/litteratursoegning.
Literature on the subject
There is a large amount of literature on systematic reviews, among which the following can be recommended:
Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011, link.
Institute of Medicine. 2011. Finding What Works in Health Care: Standards for Systematic Reviews. Washington, DC: The National Academies Press, link.
Aromataris E, Munn Z (Editors). Joanna Briggs Institute Reviewer's Manual. The Joanna Briggs Institute, 2017:, link.
(ref. 1) Green S, Higgins JPT, Alderson P, Clarke M, Mulrow CD, Oxman AD. Chapter 1: Introduction. In: Higgins JPT, Green S (editors), Cochrane Handbook of Systematic Reviews of Interventions. Chichester (UK): John Wiley & Sons (2008).
(ref. 2) Campbell Collaboration. Website. (last visited 9.8.2018).
ref. 3) Institute of Medicine of the National Academies. Standards for Systematic Reviews (2011). Link.
(ref. 4) O´Connor D, Green S, Higgins JPT (editors). Chapter 5: Defining the review question and developing criteria for including studies. In: Higgins JPT, Green S (editors), Cochrane Handbook of Systematic Reviews of Interventions. Chichester (UK): John Wiley & Sons (2008
(ref. 5) Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. JMLA (2018), 106(4): 420-431.
(ref. 6) O´Connor D, Green S, Higgins JPT (editors). Chapter 5: Defining the review question and developing criteria for including studies. In: Higgins JPT, Green S (editors), Cochrane Handbook of Systematic Reviews of Interventions. Chichester (UK): John Wiley & Sons (2008).
(ref. 6a) Munn Z., Moola S., Lisy K, Riitano D., Tufanaru C. Methodological guidance for systematic reviews of observational epidemiological studies reporting prevalence and cumulative incidence data. Int J Evid Based Healthc. 2015 Sep;13(3):147-53.)
(ref. 7) Frandsen TF, Dyrvig AK, Christensen BC, Fasterholdt I, Ølholm AM. En guide til valide og reproducerbare systematiske litteratursøgninger. Ugeskrift for Læger (2014), 176(7):647- 651.
(ref. 8) Joanna Briggs Institute 2011. Joanna Briggs Institute Reviewers' Manual 2011. The University of Adelaide, South Australia, 2011.
(ref. 8a) Campbell J, Klugar M, Ding S, Carmody DP, Hakonsen SJ, Jadotte YT, White S, Munn Z. Diagnostic test accuracy: methods for systematic review and meta-analysis. Int J Evid Based Healthc. 2015 Sep;13(3):154-62.).
(ref. 9) Khan KS, Kunz R, Kleijnen J, Antes G. Systematic Reviews to Support Evidence-Based Medicine. How to Review and Apply findings of Health Care Research. London: RSM Press, 2003.
(ref. 10) Bettany-Saltikov J. How to do a systematic literature review in nursing: a step-by-step guide. Second edition ed. London: McGraw Hill Open University Press; 2016.
(ref. 11) Booth A. Clear and present questions: formulating questions for evidence based practice. Libr Hi Tech. 2006;24(3):355-68.
(ref. 12) Cooke A, Smith D, Booth A. Beyond PICO: the SPIDER tool for qualitative evidence synthesis. Qual Health Res. 2012; 22(10):1435-43.
(ref. 13) Cumpston M, Chandler J. Chapter II: Planning a Cochrane Review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated August 2019). Cochrane, 2019. Link.
(ref. 14) PRISMA (last visited 9.10.2018).
(ref. 15) Cumpston M, Chandler J. Chapter II: Planning a Cochrane Review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated August 2019). Cochrane, 2019. Link.
(ref. 16) McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefevbre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol 2016, 75: 40-46.
(ref. 17) Koffel JB, Rethlefsen ML. Reproducibility of Search Strategies Is Poor in Systematic Reviews Published in High-Impact Pediatrics, Cardiology and Surgery Journals: A Cross-Sectional Study. PLoS One. (2016); 11(9):e0163309.
(ref. 18) Meert D, Torabi N, Costella J. Impact of librarians on reporting of the literature searching component of pediatric systematic reviews. J Med Libr Assoc. (2016); 104(4):267-277.
(ref. 19) Faggion, CM, Jr, Huivin R, Aranda L, Pandis N, Alarcon M. The search and selection for primary studies in systematic reviews published in dental journals indexed in MEDLINE was not fully reproducible. J Clin Epidemiol (2018); 98: 53-61.
(ref. 20) Frandsen TF, Dyrvig AK, Christensen JB, Fasterholdt I, Ølholm AM. En guide til reproducerbare og systematiske litteratursøgninger. Ugeskr Læger (2014); 176: V02130141.
(ref. 21) Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane, 2019. <ahref="http://www.training.cochrane.org>Link. </ahref="http://www.training.cochrane.org>
(ref. 22) McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefevbre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol (2016); 75: 40 – 46.
(ref. 23) Baudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ (2017); 356: j448.
(ref. 24) Informationsordbogen, ordbogsopslag: Grå litteratur. Website (last visited 17.10.2018).
(ref. 25) The Method Handbook - Model for preparing national clinical guidelines. The National Board of Health (2017), version 3. Only available in Danish.
(ref. 26) BMJ Best Practice. What is GRADE. (last visited 25.10.2018).