Utilisation of search filters in systematic reviews ... - Wiley Online Library

DOI: 10.1111/hir.12004

Utilisation of search filters in systematic reviews of prognosis questions Trish Chatterley* & Liz Dennett*† *JWS Health Sciences Library, University of Alberta, Edmonton, AB, Canada and †Institute of Health Economics, Edmonton, AB, Canada

Abstract Background: Search filters are designed to increase efficiency of information retrieval and can be particularly useful in limiting the large numbers of articles retrieved for systematic reviews (SRs). Existing published prognosis search filters (or hedges) have lower sensitivity and precision values than their therapy counterparts. Objectives: Taking into account the relatively poor performance of prognosis filters, this study seeks to identify which methods of limiting search results to prognostic studies are most often used by SR teams. Methods: One hundred and three SRs of prognostic studies published in 2009 and indexed in MEDLINE were retrieved. Each review’s search strategy was reviewed and prognosis-related search terms were extracted. Results: Forty-seven of 103 studies used prognosis-related terms to limit the search. Six SRs of 103 did not specify their search terms, and the remaining 50 SRs used content terms only (no terms related to methodology or prognosis). Of the 47 strategies using prognosis-related terms, only six used a published filter. Many SRs used few or poorly selected prognosis-related search terms which are unlikely to provide the sensitivity generally sought for SRs. Conclusions: Published prognosis search filters are used in only a small minority of prognosis SRs. Keywords: information retrieval, methodological filters, prognosis, search strategies.

Key Messages

• • •

Systematic review teams should ensure that the prognosis searches they develop are at least as sensitive as available validated prognostic search filters. Librarians should increase awareness of validated search filters and promote use of the sensitive strategies for systematic review searching. Further research into the development and testing of search filters for prognosis is needed.

Introduction Prognosis, the likely outcome of an illness, is identified as one of the four domains of clinical questions in evidence-based clinical practice along with 1 therapy, diagnosis and aetiology. Research on methodological search filters largely focuses on methods for retrieving therapeutic, and to a lesser

extent, diagnostic studies. There are a small number of published search filters for limiting search results to prognostic studies in MEDLINE, EMBASE and CINAHL but it is unknown how frequently these search filters are being incorporated in the search strategies of prognostic systematic reviews. Objectives

Correspondence: Trish Chatterley, JWS Health Sciences Library, University of Alberta, 2K3.28 Walter C. Mackenzie Health Sciences Centre, Edmonton, AB, Canada. Email: [email protected]

This study seeks to identify which methods of limiting search results to prognostic studies are

© 2012 The authors. Health Information and Libraries Journal © 2012 Health Libraries Group Health Information & Libraries Journal, 29, pp. 309–322

309

310

Use of search filters in prognosis reviews, Trish Chatterley & Liz Dennett

currently used in the systematic review process and to provide an indication of the relative frequency of use of these methods. Literature review Methodological search filters are ‘predetermined search strategies designed to retrieve particular types of study design when combined with subject 2 search terms of your choice’. The need for search filters is not new. More than 15 years ago, it was recognised that the large number of records in MEDLINE, in addition to the low number of relevant articles on any given topic, the limitations of indexing and the lack of search skills among users made it extremely difficult for clinicians and/ or researchers to locate the handful of relevant 3 results. These problems have not been resolved, and the exponential increase in literature published annually compounds the problem. Please note that OVID conventions have been used to express any search statements presented in this section. Details (i.e. Terminology, sensitivity, specificity, precision when available) about the search filters described below may be viewed in Table 1. The first prognosis hedges for MEDLINE were published in 1994 by a health informatics team at McMaster University, now known as the Health Information Research Unit or simply the Hedges 3 Team. The article presented separate filters designed for best sensitivity, specificity or accuracy for the four domains of clinical questions. ‘Sensitivity for a given topic is defined as the proportion of high-quality articles for that topic that are retrieved; specificity is the proportion of low-quality articles not retrieved; precision is the proportion of retrieved articles that are of high quality; and accuracy is the proportion of all arti4 cles that are correctly classified’. These searches were run in MEDLINE, and the results were compared with those garnered through a manual review of the literature from ten top journals published in the years 1986 and 1991. The search filters were proposed as a means of assisting clinicians to locate preferred study designs in the various domains. The authors suggested further research was needed to test how the filters performed when combined with topic searches,

and to develop filters that maximise sensitivity while preserving precision. Buckingham attempted to improve the precision 5 of the McMaster MEDLINE search filters. Instead of combining the various research protocol terminology (e.g. Cohort) with domain terminology (e.g. Prognosis) using the Boolean operator ‘OR’, the terms were combined with the Boolean operator ‘AND’. In this way, research articles would be retrieved but articles that spoke anecdotally about prognosis or the other domains of study would not. Buckingham’s prognosis filter also included a qualifier, stating that ‘where mortality was a possible outcome, increase the search’s sensitivity’ by adding (Mortality/or Survival Analysis/) to the strategy. The 1991 McMaster hedges did not include survival analysis as a heading in the sensitive search filter, only the specific one. In 1999, Ann McKibbon from the original McMaster Hedges Team compiled comprehensive lists of possible subject headings, subheadings, keywords and publication types for retrieving prognosis studies from MEDLINE, CINAHL, PsycINFO and EMBASE. In this work, the 1991 MEDLINE hedges for high sensitivity and high specificity were republished. McKibbon suggested that the best single term to use for retrieving prognostic studies was the exploded Medical Subject 6 Heading ‘Epidemiologic Studies’ (see Table 1). The Hedges Team revisited and expanded their 4 earlier work in 2004. They tested prognosis searches in MEDLINE against literature identified during a manual review of 161 journals published in 2000. They stated that ‘the best performing strategy for sensitivity was the same as that reported in 1991’; however, the originally published strategy included mortality as a subheading, while the revised strategy did not. Interestingly, the MeSH term prognosis is not included in even the most sensitive search strategy. This strategy has a sensitivity of 90.1% which is much lower than the corresponding therapy and diagnosis sensitive filters which both achieve 99% sensi7 tivity. The Hedges Team continued their work with 8 publication of prognosis hedges for EMBASE. Again, best sensitivity, specificity and optimised (i.e. Providing the best balance between sensitivity and specificity) searches for both single term and


5


Yale Prognosis and Natural History Best Terms, 10 PubMed 8 Hedges Team, Single Term EMBASE, Sensitivity 8 Hedges Team, Single Term EMBASE, Specificity 8 Hedges Team, Single Term EMBASE, Optimised Hedges Team, Combination of Terms EMBASE, 8 Best Sensitivity

Yale prognosis and natural history, 10 PubMed

Buckingham, MEDLINE

NS 56 86 59 50.6

87 60 51 98.7

NS

NS

NS

NS

84.1

82.9

NS

94.2

52.3

79.7

90.1

incidence.sh. OR exp mortality OR follow-up studies.sh. OR prognos:.tw. OR predict:.tw. OR course:.tw. prognos:.tw. OR first episode.tw. OR cohort.tw. prognosis.sh. OR diagnosed.tw. OR cohort:.mp. OR predictor:.tw. OR death.tw. OR exp models, statistical ((prognos* or outcome* or follow-up or predict*).ti,ab,sh. OR exp disease progression OR ((natural* or disease*) adj (progress* or course* or history)).ti,ab,sh.) AND (exp cohort studies OR (cohort* or compar* or longitudinal or prospective* or multi-variate or reproducib*).ti,ab,sh.) NOTE: Where mortality is a possible outcome, increase the search’s sensitivity by combining the following search statement to the above with an ‘OR’: mortality/or survival analysis/ cohort studies[mh] or prognosis[mh] or mortality[mh] or morbidity[mh] or ‘natural history’ or prognost*[tiab] or course[tiab] or predict*[tiab] or outcome assessment[mh] or outcome*[tiab] or inception cohort* or disease progression[mh] or survival analysis[mh] cohort studies[mh] or prognosis[mh] or disease progression[mh] exp general aspects of disease exp disease course exp physical disease by body function exp disease course OR risk:.mp. OR diagnos:.mp. OR follow-up.mp. OR ep.fs. OR outcome.tw

Hedges Tem, Combination of Terms 3 MEDLINE, Best Specificity Hedges Team, Combination of Terms 3 MEDLINE, Best Optimisation

79

65

exp epidemiologic studies/

Hedges Team, Single Term MEDLINE 3 (Best Sensitivity, Specificity and optimised) Hedges Team, Combination of Terms 3 MEDLINE, Best Sensitivity

Specificity (%)

Sensitivity (%)

Strategy

Filter

Table 1 Sensitivity, Specificity and Precision values of current (non-superseded) published prognostic filters†

1 12 1 1.1

NS

NS

NS

1.8

3.2

1.6

1

Precision (%)

Use of search filters in prognosis reviews, Trish Chatterley & Liz Dennett 311

86.2 73.4

56.9 72.5

60 85 60 50.0

79.9

93.4

Specificity (%)

1.6

2.4

1 2 1 1.1

2.1

3.9

Precision (%)

NS = not stated. Exp – Indicates that a subject heading has been ‘exploded’ in OVID. The heading and all of its narrower subject headings are combined with ‘OR’. .sh. – Field tag that retrieves only records containing the exact subject heading. .tw. – In OVID MEDLINE, terms are searched in the title and abstract fields only. In OVID EMBASE, terms are searched in the title, abstract and drug trade name. In OVID CINAHL, terms are searched in the title, abstract, instrumentation and identifiers fields. : OR * – Truncation used to search for variant endings of the same root word. .mp. – Short for multi-purpose. In general, the search looks in the Title, Original Title, Abstract, Subject Heading, Name of Substance and Registry Word fields. MH – EBSCO field tag for Major Heading. + – EBSCO notation that a subject heading has been ‘exploded’. †Please note that search strategies are presented here exactly as presented in the original articles. They have not been modified to follow standard database-specific conventions, such as the OVID conventions of capitalising MeSH headings or ending subject headings with a forward slash (/). ‡CINAHL strategies were developed using the OVID version of the database, which is no longer available. OVID syntax is presented first as expressed in the original article, but for your information and convenience is followed by EBSCO syntax in parentheses. Because of the differences in the interfaces, the EBSCO translations may not be entirely equivalent.

Hedges Team, Combination of Terms CINAHL‡, 9 Best Specificity Hedges Team, Combination of Terms CINAHL‡, 9 Best Optimisation

80.4

follow-up.mp. OR prognos:.tw. OR ep.fs. 78 51 51 92.2

50.7

prognos:.tw. OR survival.tw.

Hedges Team, Combination of Terms EMBASE, 8 Best Specificity Haynes, Combination of Terms EMBASE, Best 8 Optimisation 9 Hedges Team, Single Term CINAHL‡, Sensitivity 9 ‡ Hedges Team, Single Term CINAHL , Specificity 9 ‡ Hedges Team, Single Term CINAHL , Optimised ‡ Hedges Team, Combination of Terms CINAHL , 9 Best Sensitivity exp study design (MH ‘Study Design+’) prospective stud:.mp. (prospective stud*) exp economics (MH ‘Economics+’) exp study design OR diagnos:.mp. OR outcome.mp. (MH ‘Study Design+’ OR diagnos* OR outcome) prognos:.tw. OR prospective studies.sh. (prognos* OR MH ‘Prospective Studies’) diagnos:.tw. OR exp outcomes (healthcare) OR prospective studies.sh. (diagnos* OR MH ‘Outcomes (HealthCare)’ OR MH ‘Prospective Studies’)

Sensitivity (%)

Strategy

Filter

Table 1. (continued)

312 Use of search filters in prognosis reviews, Trish Chatterley & Liz Dennett



combinations of terms were recommended. The difficulties of developing a sensitive and specific search filter for prognosis are illustrated by their conclusions that the best single term strategies were ‘General aspects of disease’ for high sensitivity and ‘physical disease by body function’ for optimised results and that the keyword ‘diagnosis’ performed better than the seemingly obvious keyword ‘prognosis’ in the sensitive search. As Table 1 shows, the most sensitive combined term EMBASE search filter offers much greater sensitivity (98.7) than the most sensitive MEDLINE search filter (90.1), although the precision is still similarly low (1.1 compared to 1.6). The Hedges Team then published prognosis 9 search filters for the CINAHL database in 2008. Similar to the EMBASE single term filters, the best single term filters for CINAHL are not intuitive (e.g. exp Economics for the best optimised search). The most sensitive CINAHL search offers sensitivity of 92% and, like all other prognostic search filters, has low precision (1.1). It is also noteworthy that, yet again, diagnosis as a keyword was determined to be more sensitive than prognosis. Prognosis and natural history filters have also been posted on the website of the Harvey Cushing/John Hay Whitney Medical Library at Yale 10 University. This page was last updated in August 2007, but there is no indication of the date when these filters were first made publicly available. There is a longer search filter that combines 13 MeSH headings and keywords, as well as a ‘Best Terms’ strategy of three MeSH terms. The sensitivity, specificity and precision of these filters are not indicated; therefore, their utility in practice is unclear. The method for development of these filters is also not described. Table 1 provides a summary of the aforementioned current filters (those not superseded by other work by the same authors) and their sensitivity, specificity and precision. While some of the best therapy search filters have both high sensitivity (i.e. >95% is desirable for a systematic review search) and reasonable precision (i.e. ~10% to 11, 12 keep retrieval at a manageable number), no prognostic filter is able to provide that balance. In general, they have extremely low precision; those with higher precision levels have sensitivity values

that are undesirable for SRs. Low precision can mean prohibitively large numbers of abstracts to review in order to find relevant studies. There has been little research carried out to assess how prognosis filters have been used in systematic review searches. A systematic review uses explicit and reproducible methods to generate a comprehensive appraisal and synopsis of the best research evidence in order to answer a specific question and as such, requires extensive searches. This study seeks to determine the methods systematic review teams are using to limit their searches to prognosis-related studies. Are they using the validated Hedges Team filters, other published, but not validated filters, or are they creating their own filters? Methods In January 2010, we conducted a simple search in MEDLINE to retrieve English-language systematic reviews or meta-analyses published in 2009 that answer prognosis-related questions. The strategy ((Prognosis/ or prognos*.ti.) AND (systematic review.mp. or meta-analysis.mp,pt.)) was designed to maximise precision, because the goal was to retrieve a manageable sample of prognosis systematic reviews. The search was updated in September 2010 to retrieve any 2009 publications indexed during 2010. Titles and abstracts were reviewed, and studies that were either not systematic reviews or did not address prognostic questions were excluded. Because articles were not assessed for quality, the sole inclusion criterion used to determine if a study was a ‘systematic review’ was that a literature search was used to find included studies. The full-text articles of the remaining studies were retrieved. Data were extracted to identify if studies were systematic reviews; if reviews were prognostic in nature (i.e. they analysed studies that reported the impact of prognostic factors on the course of a disease or the ability of prognostic factors to predict the likely outcome of an illness); which (if any) prognosis-related search terms were used in the search strategy of the review; and identify the number of authors who specified having consulted a librarian for assistance in the development of their search strategies. The prognosis-related search terms were then reviewed


313

314


to determine how many times particular filters or search terms were used. Results The search yielded 268 unique studies. Upon review, 103 met the inclusion criteria of being systematic reviews of a prognosis question. (The complete list of studies is available from the authors upon request). The article topics were from the following clinical specialties as listed in order from greatest to least frequency: oncology, cardiology, neurology, pain, injuries, mental health, surgery, musculoskeletal diseases, gastroenterology, obstetrics and gynaecology, haematology, sexually transmitted diseases and a few other miscellaneous topics. The included reviews varied greatly in number of databases included in the search; some involved only a MEDLINE or PubMed search, while others extended the search to a large number of databases. EMBASE, CINAHL, PsycINFO and parts of the Cochrane Library were the other most commonly searched databases. For the purposes of this analysis, the word ‘term’ or ‘search term’ will be used interchangeably as a non-specific term to refer to both keywords and subject headings. A keyword search implies that the word or phrase would retrieve any instance of that word or phrase in the title, abstract or descriptors/subject headings, and potentially other fields included by specific databases. A subject heading is a preferred term from database-specific thesauri. Figure 1 shows a breakdown of the prognosis search strategies employed in the reviews. Six studies (6%) provided no details about the search strategy used, although two indicated it would be made available upon request. Those authors were not contacted for their strategies. Fifty reviews (49%) were based on search strategies comprised of terms related to the topic only and did not include any prognosis-related search terms beyond a specific prognostic factor (e.g. the effect of delay 13 of treatment on psychosis outcomes ). It should be noted that this is the most sensitive approach, as the search results are not limited to a particular methodology or domain. As in the example provided above, when the search terms for the disease and prognostic factor are combined, they do not

produce large result sets, and therefore, no search filter is necessary. However, a filter becomes desirable when larger result sets make abstract reviewing unmanageable for the researchers. This is likely to occur with systematic reviews of all prognostic factors for a condition or event (e.g. all possible prognostic factors for persistence of 14 non-specific musculoskeletal pain ). Only five reviews (5%) employed the validated sensitive search filter developed by the Hedges 15–19 Team at McMaster University. One review specified use of the Yale search strategy for prog20 nosis and natural history. In the remaining 41 reviews that incorporated prognosis-related terms, a wide range of strategies were used. Appendix 1 displays the prognostic search terms employed by the authors of the reviews. As can be seen from the wide variety of combinations of terms used, there is little consistency in the strategies employed. There are some strategies that appear to be fairly comprehensive, while many others likely lack sensitivity and others where, because of the lack of Boolean specification, it is impossible to tell how effective they would be. Table 2 shows the most commonly used search terms and their variants used by the 41 reviews that included unvalidated prognostic search strategies. Keywords were used with much greater frequency than were Medical Subject Headings, although the lack of reporting of these distinctions sometimes made differentiation difficult. Prognosis and its variations were used 33 times in 26 search strategies. (Sometimes several variants were included in the same search strategy). More specifically, prognosis itself was used 19 times, prognostic three times, prognostic factor twice and the truncated version (prognos*) six times. Prognostic factors, prognostic methods and prognost* were each used once as well. Prediction was the next most commonly used concept with 18 uses combined among predict, predict*, predictor, predictive, etc. These were followed in frequency by the concepts survival (used 16 times), outcome (11 times), cohort studies (eight times) and mortality (seven times). Follow-up, longitudinal and prospective studies were each used six times. Approximately, 50 other subject headings or keywords were used in only one or two strategies. Floating subheadings, an advanced search feature



SystemaƟc reviews of prognosƟc quesƟons in 2009 idenƟfied in this project (n = 103)

Did not include prognosis terms or filter (n = 50; 49%)

Included prognosis terms or filter (n = 47; 46%)

Used a previously published filter (n = 6; 6%)

Hedges team Prognosis filter (n = 5; 5%)

Unknown search strategy (n = 6; 6%)

Created own prognosis search strategy (n = 41; 40%)

Yale Natural History filter (n = 1; 1%)

Figure 1 Prognosis search strategies used in the included reviews

available in some search interfaces such as Ovid, were used in two strategies. In only seven of the 103 studies reviewed (7%) was it indicated that the authors had received assistance with the development of their search strategies. In six cases, recognition of assistance was given in the acknowledgement section. In two of those six instances, this help was also mentioned within the methods section of the review. In one instance, only the names of the assistants were provided, and we were unable to confirm if the individuals were qualified librarians. In the seventh review, two information specialists participated in the review and were listed as authors of the publication. This may not be an accurate representation of the number of times librarians were consulted, as assistance may simply not have been recognised. Inclusion of advanced searching techniques such as floating subheadings, adjacency searching (the searching for keywords in proximity to each other) and use of validated filters in some of the other reviews may indicate that assistance was received but not acknowledged. It was not one of our objectives to identify errors in search strategies or to document incompleteness of reporting; however, we did notice instances of

each. Some authors combined terms incorrectly, employing the Boolean operator ‘OR’ to combine different concepts. Thirteen of 41 authors did not provide clear indication of how search terms were combined (i.e. whether they used ‘AND’ or ‘OR’ to combine the prognostic-related search terms to the topic segment of the search). These strategies are listed in section II of Appendix 1. Sometimes one keyword (e.g. prognosis) was used when there were other possible variants of that word (e.g. prognostic) that could have been included; the authors could easily have employed the truncated term to locate the other variants. Discussion In just under half the systematic reviews studied (49%), no prognosis or methodology terms were included in the search at all. It is unknown if this is due to a lack of awareness of available search filters, or the result of explicit decisions made by the researchers. Researchers might choose not to use existing hedges or their own combinations of prognostic terms because they feel that approach is not sensitive enough, or simply because their searches were sufficiently narrowed by combining


315

316

Use of search filters in prognosis reviews, Trish Chatterley & Liz Dennett Table 2 Most commonly used terms

Concept

Variants

Prognosis

Prognosis Prognos* Prognostic Prognostic factor Prognostic factors Prognostic methods Prognost* Predict* Predict Predictor Predictive factors Predictive Predictive value of tests Prediction Predictors of outcomes Radiographic predictors Survival Survival analysis Surviv* Disease-free survival Survival* Survival rate Outcome Outcomes Outcome Assessment (HealthCare) Predictors of outcomes Patient-reported outcomes Cohort Cohort studies Mortality Follow-up Follow-up studies Longitudinal Longitudinal Studies Prospective Prospective studies

Prediction

Survival

Outcome

Cohort studies Mortality Follow-up studies Longitudinal studies Prospective studies

Variant frequency

Total frequency

19 6 3 2 1 1 1 6 3 2 2 1 1

33

18

1 1 1 9 3 1 1 1 1 6 2 1

16

11

1 1 6 2 7 4 2 5 1 4 2

8 7 6 6

6

Distinctions between keywords and MeSH terms have not been identified because frequent lack of reporting often made such differentiation impossible.

population search terms with a specific prognostic factor or the health outcome of interest.

Of the 47 strategies that included prognosisrelated terms, only five used published validated filters (i.e. Hedges Team filter) and one more used a published but not validated filter. When selecting an appropriate prognosis search filter for use in a systematic review, authors may wish to use one of the existing search filter appraisal tools to aid them 2, 21, 22 in making the decision. The remaining 41 strategies (see Appendix 1) appear to have been created by whomever was responsible for the search. Many use few or poorly selected prognostic search terms. This is a concern because the Hedges Team determined that the greatest sensitivity in MEDLINE can be achieved by using the following six terms: incidence.sh. OR exp mortality OR follow-up studies.sh. OR prog4 nos*.tw. OR predict*.tw. OR course*.tw. As stated in chapter six of the Cochrane Handbook of Systematic Reviews, ‘searches should seek high sensitivity, which may result in relatively low pre23 cision’. Any search strategy containing fewer prognostic-related search terms than the sensitive Hedges Team filter is likely to be less sensitive. While it is possible that a search strategy of only a few terms may be highly sensitive in some contexts, as a general rule, a search that combines (using the Boolean ‘OR’) more synonyms will have increased sensitivity. This is especially true in instances where the terminology is ambiguous or complex, and the indexing is therefore less consistent. The concept of prognosis is a good example of terminological complexity, because appropriate search terms could include keywords describing the clinical domain itself, study designs used in prognostic research or specific outcomes such as death. Searches with only a few terms may indicate either lack of experience in developing comprehensive search strategies, or the desire for a smaller or more precise result set. Table 2 shows the most commonly used terms from the 41 strategies listed in Appendix 1 (i.e. all the strategies that used prognosis-related terms to limit the search but did not use a published filter). The most commonly used search terms from the searches listed in Appendix 1 are those that are also included in published prognosis search strategies (i.e. prognosis, prediction, survival and outcome). However, researchers did not always use the most sensitive variant of the term (e.g. used



prognostic rather than prognos*). Terms related to study design such as cohort studies, follow-up studies or prospective studies were also used but less frequently. There are also a number of terms shown to be quite sensitive in validated filters (remember that the Hedges Team filters are the only validated prognosis filters) but were not commonly included in searches. For example, the term course was used three times and the term incidence only twice. Other issues, such as phrase searching, were also identified that would limit search sensitivity. For instance, using the phrase prognostic factors as opposed to the single term prognostic significantly restricts the result set; therefore, the search will likely fail to identify some relevant studies. Some phrases, such as predictors of outcomes, are much too specific and therefore lead to low yield. Sampson and McGowan have reported on the frequency and type of errors found in search strat24 egies. Examples of several of these errors were evident in the search strategies reviewed. The errors in the search strategies and the frequency of limited or poorly selected prognosis-related search terms suggest that experienced searchers are not being consulted or involved in many of these reviews. The small number of articles acknowledging the assistance from librarians seems to confirm this suspicion. Incompleteness of reporting is also a concern. Authors should take care to outline their search strategies accurately and transparently because systematic reviews are research and as such should provide enough methodological detail to make it possible for another researcher to replicate the review and obtain the same results. Search strategies should include notation to indicate whether terms were employed as keywords or as subject headings. Boolean operators should be included to provide clear indication of how search terms about different concepts are combined. Without the specification of Boolean operators, it prevents others from reproducing the searches as well as affecting readers’ abilities to appraise the quality or sensitivity of the search. With other searches that do state how concepts are combined, the reader is better able to evaluate how sensitive a search is likely to be. When using validated strategies, it should be stated whether the sensitive version was used or

the specific one. The correct database names should be listed, and the platform for the databases given. Authors should follow published search 23, 25 strategy documentation guidelines. Creating sensitive and precise prognosis searches would be much easier if the indexing of the original studies were improved. For this to happen, it is essential that authors take care to craft their titles and abstracts thoughtfully, so that important terms are included and will lead to appropriate retrieval of studies. Stating the clinical domain of the question (e.g. prognosis) will also help with the indexing process, as indexers will be less likely to apply inappropriate subject headings or miss assigning relevant ones. Buckingham highlighted this in her 1998 study, stating that ‘[al] most invariably, if a methodologic term had not been used in the title or abstract, it certainly would 5 not appear as a MeSH index term’. Reporting standards such as the Consolidated Standards of Reporting Trials (CONSORT) Statement and the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement 26, 27 also address this issue. The first items on their respective checklists state that the authors should use terms in the title or abstract that clearly identify the study’s design type. There are several limitations to this study. It is based on a small sample size from a narrow search in one database for a single publication year. The results may therefore not be representative of all prognosis systematic reviews. Our inclusion criteria for determining whether an article was a systematic review was very inclusive; had we been more selective or applied a more rigorous definition of a systematic review (e.g. Cochrane), fewer studies may have been included. The included reviews were also not assessed for methodological quality, so it is unknown whether well-designed reviews were more likely to have used validated search filters than reviews conducted with less methodological rigour. Conclusion There is currently no published prognosis filter that has both the high sensitivity (>95%) and reasonable precision (>10%) of the best treatment study 11, 12 filters. Hopefully, further research into the


317

318


development and testing of search filters for prognosis will yield better performing filters. Almost half of all prognosis systematic reviews do not use any prognosis-related terms to narrow the search. While this is the most sensitive approach to searching, in some cases, it may have resulted in unnecessarily large numbers of results, the review of which is very resource-intensive. Only 5% of prognosis systematic reviews use a validated filter. The rest limited their searches using self-selected prognosis-related terms. Despite the limitations of published prognosis search filters, the sensitivity of many of the searches may have been increased if the reviewers had employed ‘sensitive’ published filters. Librarians should increase awareness of these validated filters and promote their use when appropriate. References 1 Guyatt, G. Users’ Guides to The Medical Literature: A Manual for Evidence-Based Clinical Practice, 2nd edn. New York: McGraw Hill Medical, 2008. 2 Jenkins, M. Evaluation of methodological search filters–a review. Health Information and Libraries Journal 2004, 21, 148–63. 3 Haynes, R. B., Wilczynski, N., Mckibbon, K. A., Walker, C. J. & Sinclair, J. C. Developing optimal search strategies for detecting clinically sound studies in medline. Journal of the American Medical Informatics Association 1994, 1, 447–58. 4 Wilczynski, N. L., Haynes, R. B. & Team, H. Developing optimal search strategies for detecting clinically sound prognostic studies in MEDLINE: an analytic survey. BMC Medicine 2004, 2, 23. 5 Buckingham, J. Evidence-based medicine quality filters in MEDLINE records: theory, practice and practical reality. Bibliotheca Medica Canadiana 1998, 20, 7–11. 6 McKibbon, A. K. Natural history and prognosis. In: PDQ Evidence-Based Principles and Practice. Hamilton, ON: B. C.Decker Inc., 1999: 105–20. 7 Search Strategies for MEDLINE in ovid Syntax and the PubMed Translation [Internet]. Hamilton, ON: McMaster University. Accessible at: http://hiru.mcmaster.ca/hiru/ HIRU_Hedges_MEDLINE_Strategies.aspx (accessed 29 June 2010). 8 Wilczynski, N. L. & Haynes, R. B. Optimal search strategies for detecting clinically sound prognostic studies in EMBASE: an analytic survey. Journal of the American Medical Informatics Association 2005, 12, 481–85. 9 Walker-Dilks, C., Wilczynski, N. L., Haynes, R. B. & Team, H. Cumulative index to nursing and allied health literature search strategies for identifying methodologically sound causation and prognosis studies. Applied Nursing Research 2008, 21, 98–103.

10 Finding Evidence in PubMed [Internet]. New Haven, CT: Harvey Cushing/John Hay Whitney Medical Library, Yale University, 2007. Accessible at: http://library.medicine.yale. edu/guides/page/71 (accessed 9 September 2012). 11 Haynes, R. B., McKibbon, K. A., Wilczynski, N. L., Walter, S. D., Werre, S. R. & Team, H. Optimal search strategies for retrieving scientifically strong studies of treatment from medline: analytical survey. British Medical Journal 2005, 330, 1179. 12 Glanville, J. M., Lefebvre, C., Miles, J. N. & CamossoStefinovic, J. How to identify randomized controlled trials in MEDLINE: ten years on. Journal of the Medical Library Association 2006, 94, 130–36. 13 Farooq, S., Large, M., Nielssen, O. & Waheed, W. The relationship between the duration of untreated psychosis and outcome in low-and-middle income countries: a systematic review and meta analysis. Schizophrenia Research 2009, 109, 15–23. 14 Lakke, S. E., Soer, R., Takken, T. & Reneman, M. F. Risk and prognostic factors for non-specific musculoskeletal pain: a synthesis of evidence from systematic reviews classified into ICF dimensions. Pain 2009, 147, 153–164. 15 Smith, M. D. & McCall, J. L. Systematic review of tumour number and outcome after radical treatment of colorectal liver metastases. British Journal of Surgery 2009, 96, 1101– 3. 16 Wright, A. A., Cook, C. & Abbott, J. H. Variables associated with the progression of hip osteoarthritis: a systematic review. Arthritis and Rheumatism 2009, 61, 925–936. 17 olde Hartman, T. C., Borghuis, M. S., Lucassen, P. L., van de Laar, F. A., Speckens, A. E. & van Weel, C. Medically unexplained symptoms, somatisation disorder and hypochondriasis: course and prognosis. A systematic review. Journal of Psychosomatic Research 2009, 66, 363–377. 18 Whiteley, W., Chong, W. L., Sengupta, A. & Sandercock, P. Blood markers for the prognosis of ischemic stroke: a systematic review. Stroke 2009, 40, e380–e389. 19 Whiteley, W., Jackson, C., Lewis, S., Lowe, G., Rumley, A., Sandercock, P., Wardlaw, J., Dennis, M. & Sudlow, C. Inflammatory markers and poor outcome after stroke: a prospective cohort study and systematic review of interleukin-6. PLoS Medicine 2009, 6, e1000145. 20 van Velzen, J. M., van Bennekom, C. A., Edelaar, M. J., Sluiter, J. K. & Frings-Dresen, M. H. Prognostic factors of return to work after acquired brain injury: a systematic review. Brain Injury 2009, 23, 385–395. 21 Bak, G., Mierzwinski-Urban, M., Fitzsimmons, H., Morrison, A. & Maden-Jenkins, M. A pragmatic critical appraisal instrument for search filters: introducing the CADTH CAI. Health Information and Libraries Journal 2009, 26, 211–219. 22 Glanville, J., Bayliss, S., Booth, A., Dundar, Y., Fernandes, H., Fleeman, N. D., Foster, L., Fraser, C., Fry-Smith, A., Golder, S., Lefebvre, C., Miller, C., Paisley, S., Payne, L., Price, A. & Welch, K. So many filters, so little time: the development of a search filter appraisal checklist. Journal of the Medical Library Association 2008, 96, 356–361.


Use of search filters in prognosis reviews, Trish Chatterley & Liz Dennett 23 Higgins, J. P. T. & Green, S. (eds). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Accessible at: www.cochrane-handbook.org. 24 Sampson, M. & McGowan, J. Errors in search strategies were identified by type and frequency. Journal of Clinical Epidemiology 2006, 59, 1057–63. 25 Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. Centre for Reviews and Dissemination. University of York, 2009. Accessible at: http://www. york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf (accessed 25 November 2011). 26 CONSORT Group. CONSORT Statement. 2010. Accessible at: http://www.consort-statement.org/consort-statement/ (accessed 31 January 2011). 27 STROBE Statement [Internet]. Bern: University of Bern, 2011. Accessible at: http://www.strobe-statement.org/ (accessed 1 January 2011). 28 Schoenleber, S. J., Kurtz, D. M., Talwalkar, J. A., Roberts, L. R. & Gores, G. J. Prognostic role of vascular endothelial growth factor in hepatocellular carcinoma: systematic review and meta-analysis. British Journal of Cancer 2009, 100, 1385–92. 29 Papakostidis, C., Kanakaris, N. K., Kontakis, G. & Giannoudis, P. V. Pelvic ring disruptions: treatment modalities and analysis of outcomes. International Orthopaedics 2009, 33, 329–38. 30 Bramer, J. A., van Linge, J. H., Grimer, R. J. & Scholten, R. J. Prognostic factors in localized extremity osteosarcoma: a systematic review. European Journal of Surgical Oncology 2009, 35, 1030–6. 31 Chida, Y. & Vedhara, K. Adverse psychosocial factors predict poorer prognosis in HIV disease: a meta-analytic review of prospective investigations. Brain, Behavior, & Immunity 2009, 23, 434–45. 32 Cecelja, M. & Chowienczyk, P. Dissociation of aortic pulse wave velocity with risk factors for cardiovascular disease other than hypertension: a systematic review. Hypertension 2009, 54, 1328–36. 33 Detaille, S. I., Heerkens, Y. F., Engels, J. A., van der Gulden, J. W. & van Dijk, F. J. Common prognostic factors of work disability among employees with a chronic somatic disease: a systematic review of cohort studies. Scandinavian Journal of Work, Environment & Health 2009, 35, 261– 81. 34 Gould Rothberg, B. E., Bracken, M. B. & Rimm, D. L. Tissue biomarkers for prognosis in cutaneous melanoma: a systematic review and meta-analysis. Journal of the National Cancer Institute 2009, 101, 452–74. 35 Fekadu, A., Wooderson, S. C., Markopoulo, K., Donaldson, C., Papadopoulos, A. & Cleare, A. J. What happens to patients with treatment-resistant depression? A systematic review of medium to long term outcome studies Journal of Affective Disorders 2009, 116, 4–11. 36 de Graeff, P., Crijns, A. P., de Jong, S., Boezen, M., Post, W. J., de Vries, E. G., van der Zee, A. G. & de Bock, G. H. Modest effect of p53, EGFR and HER-2/neu on prognosis in epi-

37

38

39

40

41

42

43

44

45

46

47

thelial ovarian cancer: a meta-analysis. British Journal of Cancer 2009, 101, 149–159. Alvaro, T., Lejeune, M., Escriva, P., Pons, L. E., Bosch, R., Jaen, J., Lopez, C., Salvado, M. T. & de Sanjose, S. Appraisal of immune response in lymphoproliferative syndromes: a systematic review. Critical Reviews in Oncology Hematology 2009, 70, 103–13. Walton, D. M., Pretty, J., MacDermid, J. C. & Teasell, R. W. Risk factors for persistent problems following whiplash injury: results of a systematic review and meta-analysis. Journal of Orthopaedic and Sports Physical Therapy 2009, 39, 334–50. de Geus-Oei, L. F., Vriens, D., van Laarhoven, H. W., van der Graaf, W. T. & Oyen, W. J. Monitoring and predicting response to therapy with 18F-FDG PET in colorectal cancer: a systematic review. Journal of Nuclear Medicine 2009, 50 (Suppl 1), 43S–54S. Mummaneni, P. V., Kaiser, M. G., Matz, P. G., Anderson, P. A., Groff, M., Heary, R., Holly, L., Ryken, T., Choudhri, T., Vresilovic, E., Resnick, D. & Joint Section on Disorders of the Spine and Peripheral Nerves of the American Association of Neurological Surgeons and Congress of Neurological Surgeons. Preoperative patient selection with magnetic resonance imaging, computed tomography, and electroencephalography: does the test predict outcome after cervical surgery? Journal of Neurosurgery. Spine 2009, 11, 119–29. Bruinstroop, E., Klok, F. A., Van De Ree, M. A., Oosterwijk, F. L. & Huisman, M. V. Elevated D-dimer levels predict recurrence in patients with idiopathic venous thromboembolism: a meta-analysis. Journal of Thrombosis and Haemostasis 2009, 7, 611–18. Araujo, J. P., Lourenco, P., Azevedo, A., Frioes, F., Rocha-Goncalves, F., Ferreira, A. & Bettencourt, P. Prognostic value of high-sensitivity C-reactive protein in heart failure: a systematic review. Journal of Cardiac Failure 2009, 15, 256–266. Tandon, P. & Garcia-Tsao, G. Prognostic indicators in hepatocellular carcinoma: a systematic review of 72 studies.[see comment]. Liver International 2009, 29, 502–510. Terasawa, T., Lau, J., Bardet, S., Couturier, O., Hotta, T., Hutchings, M., Nihashi, T. & Nagai, H. Fluorine-18-fluorodeoxyglucose positron emission tomography for interim response assessment of advanced-stage hodgkin’s lymphoma and diffuse large B-cell lymphoma: a systematic review.[see comment]. Journal of Clinical Oncology 2009, 27, 1906– 1914. Celestin, J., Edwards, R. R. & Jamison, R. N. Pretreatment psychosocial variables as predictors of outcomes following lumbar surgery and spinal cord stimulation: a systematic review and literature synthesis. Pain Medicine 2009, 10, 639–653. Korenromp, E. L., Williams, B. G., Schmid, G. P. & Dye, C. Clinical prognostic value of RNA viral load and CD4 cell counts during untreated HIV-1 infection–a quantitative review. PLoS One 2009, 4, e5950 [Electronic Resource]. Pan, L., Gu, P., Huang, G., Xue, H. & Wu, S. Prognostic significance of SUV on PET/CT in patients with esophageal


319

320


48

49

50

51

52

53

54

55

56

cancer: a systematic review and meta-analysis. European Journal of Gastroenterology and Hepatology 2009, 21, 1008–1015. Gravante, G., Garcea, G., Ong, S. L., Metcalfe, M. S., Berry, D. P., Lloyd, D. M. & Dennison, A. R. Prediction of mortality in acute pancreatitis: a systematic review of the published evidence. Pancreatology 2009, 9, 601–614. Kuncova, K., Janda, A., Kasal, P. & Zamecnik, J. Immunohistochemical prognostic markers in intracranial ependymomas: systematic review and meta-analysis. Pathology Oncology Research 2009, 15, 605–614. Shah, T., Casas, J. P., Cooper, J. A., Tzoulaki, I., Sofat, R., McCormack, V., Smeeth, L., Deanfield, J. E., Lowe, G. D., Rumley, A., Fowkes, F. G., Humphries, S. E. & Hingorani, A. D. Critical appraisal of CRP measurement for the prediction of coronary heart disease events: new data and systematic review of 31 prospective cohorts.[see comment]. International Journal of Epidemiology 2009, 38, 217–231. Montazeri, A. Quality of life data as prognostic indicators of survival in cancer patients: an overview of the literature from 1982 to 2008. Health and Quality of Life Outcomes 2009, 7, 102. Nair, V. S., Krupitskaya, Y. & Gould, M. K. Positron emission tomography 18F-fluorodeoxyglucose uptake and prognosis in patients with surgically treated, stage I non-small cell lung cancer: a systematic review. Journal of Thoracic Oncology 2009, 4, 1473–1479. Ip, S., Tatsioni, A., Conant, A., Karagozian, R., Fu, L., Chew, P., Raman, G., Lau, J. & Bonis, P. Predictors of clinical outcomes following fundoplication for gastroesophageal reflux disease remain insufficiently defined: a systematic review. American Journal of Gastroenterology 2009, 104, 752–758. Sutcliffe, P., Hummel, S., Simpson, E., Young, T., Rees, A. & Wilkinson, A., Hamdy, F., Clarke, N. & Staffurth, J. Use of classical and novel biomarkers as prognostic risk factors for localised prostate cancer: a systematic review. Health Technology Assessment 2009;13:iii, xi,xiii 1–219. Zhan, P., Wang, J., Lv, X. J., Wang, Q., Qiu, L. X., Lin, X. Q., Yu, L. K. & Song, Y. Prognostic value of vascular endothelial growth factor expression in patients with lung cancer: a systematic review with meta-analysis. Journal of Thoracic Oncology 2009, 4, 1094–1103. Briggs, C. D., Neal, C. P., Mann, C. D., Steward, W. P., Manson, M. M. & Berry, D. P. Prognostic molecular markers in cholangiocarcinoma: a systematic review. European Journal of Cancer 2009, 45, 33–47.

57 Scott, P. A., Barry, J., Roberts, P. R. & Morgan, J. M. Brain natriuretic peptide for the prediction of sudden cardiac death and ventricular arrhythmias: a meta-analysis. European Journal of Heart Failure 2009, 11, 958–966. 58 Argo, C. K., Northup, P. G., Al-Osaimi, A. M. & Caldwell, S. H. Systematic review of risk factors for fibrosis progression in non-alcoholic steatohepatitis. Journal of Hepatology 2009, 51, 371–379. 59 Haapaniemi, E. & Tatlisumak, T. Is D-dimer helpful in evaluating stroke patients? A systematic review Acta neurologica Scandinavica 2009, 119, 141–150. 60 Mommersteeg, P. M., Denollet, J., Spertus, J. A. & Pedersen, S. S. Health status as a risk factor in cardiovascular disease: a systematic review of current evidence. American Heart Journal 2009, 157, 208–218. 61 Nausheen, B., Gidron, Y., Peveler, R. & Moss-Morris, R. Social support and cancer progression: a systematic review. Journal of Psychosomatic Research 2009, 67, 403 –415. 62 Morris, B. H., Bylsma, L. M. & Rottenberg, J. Does emotion predict the course of major depressive disorder? A review of prospective studies The British Journal of Clinical Psychology 2009, 48(Pt 3), 255–273. 63 Jeannon, J. P., Calman, F., Gleeson, M., McGurk, M., Morgan, P., O’Connell, M., Odell, E. & Simo, R. Management of advanced parotid cancer. A systematic review. European Journal of Surgical Oncology 2009, 35, 908–915. 64 Cole, M. G., Ciampi, A., Belzile, E. & Zhong, L. Persistent delirium in older hospital patients: a systematic review of frequency and prognosis. Age and Ageing 2009, 38, 19– 26. 65 Sansam, K., Neumann, V., O’Connor, R. & Bhakta, B. Predicting walking ability following lower limb amputation: a systematic review of the literature. Journal of Rehabilitation Medicine 2009, 41, 593–603. 66 Chio, A., Logroscino, G., Hardiman, O., Swingler, R., Mitchell, D., Beghi, E., Traynor, B. G. & Eurals Consortium. Prognostic factors in ALS: a critical review. Amyotrophic Lateral Sclerosis 2009, 10, 310–323. 67 Ip, H. Y., Abrishami, A., Peng, P. W., Wong, J. & Chung, F. Predictors of postoperative pain and analgesic consumption: a qualitative systematic review. Anesthesiology 2009, 111, 657–677. Received 12 April 2011; Accepted 17 August 2012



Appendix 1: Prognosis strategies employed by review authors I Strategies where Boolean combinations were obvious 28 • Prognosis 29 • outcome 30 • prognosis, survival 31 • prospective or longitudinal 32 • ‘determin*’ or ‘predict*’ 33 • prognostic factors or predictive factors 34 • [prognos*] or [surviv*] or [progress*] (this strategy was condensed from original version but would have the same result) 35 • prognosis, course, follow-up 36 • title search for marker* or prognost* or survival 37 • (‘Survival Analysis’ OR ‘Prognosis’ OR ‘Outcome Assessment (HealthCare)’) 38 • ‘prognosis,’ ‘prospective’ or ‘cohort’ 39 • Prediction or therapy monitoring or response or response monitoring • predictors of outcomes or radiographic predictors or computed tomography or magnetic resonance imaging or 40 electromyography 41 • Recurrence OR risk OR follow-up OR prognosis 42 • Prognosis OR Death OR sever* OR Risk OR Rate OR Progress* 43 • survival [ALL] OR mortality [ALL] OR predictor [ALL] OR prognosis [ALL] OR prognostic [ALL] 44 • exp sensitivity-and-specificity or predict$ or diagnos$ or di.fs. or du.fs. or accura$ 45 • Screen or outcomes or predict or risk factor or psychological (this strategy was condensed from original version but would have the same result) • (longitudinal OR cohort OR seroconver* OR prognos*) and (AIDS OR mortality OR survival OR progress* OR (natural 46 history)) 47 • ‘prognosis’ or ‘prognostic factor’ or ‘predict’ or ‘outcome’ or ‘survival’ 48 • prognosis, mortality, factors, score, criteria 49 • Patient(s), prognosis, diagnosis, prognostic, diagnostic, monitoring, follow-up 50 • limited to prospective studies (including cohort, nested case–control or case–cohort studies) • ‘quality of life’, ‘patient-reported outcomes’, ‘prognostic’, ‘predictor’, ‘predictive’ and ‘survival’ in the titles of 51 publications • Medline: (‘epidemiology ‘[Subheading] OR ‘aetiology ‘[Subheading] OR ‘Probability’[Mesh] OR ‘Time Factors’[Mesh] OR ‘Prognosis’[Mesh] OR prognos*[tw] OR ‘Causality’[Mesh] OR ‘Cohort Studies’[Mesh] OR ‘Incidence’[Mesh] OR risk [tw] OR predict*[tw] or causal*[tw] or Cohort[tw] or incidence[tw] or prospective[tw] or retrospective[tw] or 14 longitudinal[tw]) • prognos* [tw] OR predict*[tw] OR course [tw] OR ‘natural history’ [tw] OR incidence[sh] OR death* OR ‘models, statistical’ [mesh] OR cohort* [tw] OR occur* [tw] OR recur* [tw] OR ‘long term’ [tw] OR diagnosed [tw] OR ‘first 52 episode’ [tw] OR prospective [tw] or mortality [mesh] or mortality[sh] or follow-up studies [mesh] • follow-up studies/or (follow-up or followup).tw. or exp cohort studies/or cohort.tw. or exp Case–Control Studies/or (case adj20 control).tw. or exp Longitudinal Studies or longitudinal.tw. or (random$ or rct).tw. or exp Randomised Controlled Trials/or exp random allocation/or exp Double-Blind Method/or exp Single-Blind Method/or randomised controlled trial.pt. or clinical trial.pt. or controlled clinical trials/or (clin$ adj trial$).tw. or ((singl$ or doubl$ or trebl$ or tripl$) adj (blind$ or mask$)).tw. or exp Research Design/or exp Evaluation Studies/or exp Prospective Studies/or exp 53 Comparative Study/ • prognostic methods.mp. or predictive factors.mp. or ((prognos$ or predict$ or neural network$ or algorithm$) adj10 (relapse$ or recurrence$ or survival$ or death$ or mortality or progress$ or disease free or psa failure$ or biochemical failure$)).ti,ab or survival rate/or ((exp prognosis/or neural networks computer or exp models statistical/or algorithms/) and (relapse$ or recurrence$ or survival$ or death$ or mortality or progress$ or disease free or psa failure$ or biochemical failure$)).ti,ab. or disease free survival/or mortality/or recurrence/or exp survival analysis/or nomogram$. mp. or ((marker$or biomarker$) adj10 (prognos$ or predict$)).mp… (this strategy was condensed from original version 54 but would have the same result)


321

322

Use of search filters in prognosis reviews, Trish Chatterley & Liz Dennett (continued) II Strategies where Boolean combinations between prognosis and topic segments of the search are unclear 55, 56 • Prognosis 57 • predictive value of tests, sudden death 58 • risk factors 59 • outcome, prognosis 60 • mortality, survival analysis, prognosis 61 • progression, survival/mortality 62 • ‘outcome’, ‘longitudinal’ and ‘course’ 63 • survival 64 • outcome, prognosis 65 • predict*, prognos* and probability 66 • ‘survival’, ‘outcome’, ‘prognosis’ and ‘prognostic factor’ • ‘risk factors,’ ‘risk assessment,’ ‘predict,’ ‘univariate analysis,’ ‘multi-variate analysis,’ ‘regression analysis,’ ‘regression 67 model,’ ‘logistic regression,’ ‘diagnostic model,’ ‘analysis of variance’

Note Note: when search strategies for multiple databases were made available, only the prognostic terms searched in Medline are reported here.
