Quantitative approaches to summarize all relevant information pertaining to the research question depend on the type of data that is being integrated. Currently there are two main approaches to integrating data; fixed-effects model and random-effects model.
Fixed-effects Models
These models assume that the studies included in the meta-analysis have no differences in underlying study populations, patient selection criteria, patient's response to treatment, and the methods of treatment. The apparent differences in study results are assumed to be purely due to chance during sampling. Indeed, this method assumes that individual study effects sizes are random draws from a single frequency distribution of an effect size and that the only source of variation between individual study effect sizes is with-in study heterogeneity. In other words, patients who were enrolled in different studies but were assigned the same treatment are taken to be exchangeable.
If we denote individual study effects with Yi then this model can be expressed as below
A test for homogeneity can be performed as given above to confirm the presence of homogeneity between the study effect sizes and results are compared with chi-square distribution.
Various methods for performing a fixed-effect model meta-analysis for a binary outcome have been proposed and include Mantel-Haenszel's method, Woolf's method, Peto's method, and logistic regression. Although fixed effects model is easier to develop, it has several limitations. The most important limitation is the assumption of homogeneity which is quite unrealistic and counter-intuitive. There are hardly any two studies that have similar study designs, enroll same kind of patients, and give treatment in the same manner. Even if there are such trials, it is unlikely that they have been performed at the same time, and thus may suffer from the bias due to the improvements in health care. Furthermore, above and other statistical test for homogeneity have low power and may not detect heterogeneity between the study results.
Probably the most important function of meta-analysis in Medicine is to accumulate and integrate evidence for a particular intervention. Meta-analysis is very helpful in practicing evidence-based medicine and in developing clinical guidelines. Meta-analysis can not only help us in better estimation of the overall benefit of an intervention, it can also help us in pointing to possible factors for inconsistent results in different studies. These factors can be investigated in future studies.
This leads us to the second most important role of meta-analysis in Medicine. Meta-analyses can play a key role in planning new studies. Through a systematic research of the literature meta-analysis can help us to identify which questions have already been answered and which remain to be answered. Meta-analysis can further guide us in selecting proper outcome measures or study-populations and aspects of the planned intervention that are likely to be most helpful in answering a research question.
If need for a clinical study is shown and supported with the best possible assimilation of the available clinical evidence, a funding agency is likely to support such a study. Meta-analyses can be used not only to justify the need for a new study, it can put a study protocol on a stronger footing. The meta-analysis can show the potential utility of the planned study by putting the available evidence in context. The graphical elements of the meta-analysis, such as the forest plot, provide a mechanism for presenting the data clearly, and for capturing the attention of the reviewers. Some funding agencies now require a meta-analysis of existing research as part of the grant application to fund new research.
Meta-analysis is also helpful in a researcher's career. Generally, meta-analysis and systematic reviews are considered of a higher standard than narrative reviews. Indeed, a recent study has found that meta-analysis are the most-frequently cited type of research articles. Thus, many journals encourage researchers to submit systematic reviews and meta-analyses that summarize the body of evidence on a specific question. Meta-analyses also play a supporting role in other papers. For example, a paper that reports results for a new primary study might include a meta-analysis in the discussion to synthesize prior data and help to place the new study in context.
Meta-analysis has multiple other uses in medical and non-medical fields. For example, applied researchers in education, psychology, criminal justice, and a host of other fields use meta-analysis to determine which interventions work, and which ones work best. Meta analysis is also widely used in basic research to evaluate the evidence in areas as diverse as sociology, social psychology, sex differences, finance and economics, political science, marketing, ecology and genetics, among others. Pharmaceutical companies use meta-analysis to gain approval for new drugs, with regulatory agencies sometimes requiring a meta-analysis as part of the approval process.
Explosion in the number of papers published in medical journals makes it difficult to keep pace with the primary research. Experts synthesize this accumulating knowledge in a summary for the benefit of a busy clinician. We find these reviews in all medical journals.
This traditional method of reviewing literature, commonly known as narrative review, has several disadvantages. One obvious problem is that, reviewers rarely begin with an open mind and review can be biased by their professional opinions. Further, reviewers may include only those studies that agree with their own opinions and may completely ignore studies that have reached to a different opinion.
A better way of reviewing medical literature is to develop a search strategy in order to identify all the relevant clinical trials and systematically review all these trials. Such a review is generally known as a ‘Systematic Review’. This approach should eliminate the bias resulting from selective inclusion of studies. Although a systematic review is better than a narrative review, it also has one major limitation. A systematic review is, generally, unable to reach to a conclusion without ignoring sample size, effect size, and research design of the clinical trial.
The limitation of systematic review can be dealt by statistically combining results of all relevant clinical trials. This method of reviewing literature is known as ‘Meta-analysis’. Such a review not only evaluates all the available literature on a particular topic, but also provides a summary estimate of the effect size after taking into consideration sample size, effect size and study design.
What is a Meta-Analysis?
Meta-analysis is a statistical procedure for combining data from multiple studies. When the treatment effect (or effect size) is consistent from one study to the next, meta-analysis can be used to identify this common effect with more precision than individual studies. When the effect varies from one study to the next, methods of meta-analysis can be extended to identify the reason for the variation.
Why do a Meta-Analysis?
Even a casual reader of medical literature will notice that results of clinical studies vary from one study to another. This variation in study results may not due to some problem with the study design or conduct, but rather can be due to pure chance alone. Thus, decisions about the utility of a treatment or the validity of a hypothesis cannot be based on the results of a single study. Now if we need multiple studies to identify the real effect (benefit or harm) of the treatment, we do need a mechanism to synthesize data across such studies. Narrative reviews are largely subjective, in contrast, meta-analysis applies objective formulas (much as one would apply statistics to data within a single study), to combine the results of any number of studies.
Before one starts planning to combine study-results, one needs to consider whether it is appropriate to combine these studies. This is important as the studies may be so different in methodology that combining them may provide misleading or unreliable results. If all studies can’t be combined, one can further evaluate whether some of the studies, with similar methodology, can be combined. For example, it may not be appropriate to combine randomized controlled trials with trials that have no control group and compare results before and after a treatment. However, it may be appropriate to combine randomized trials only or to analyze these two different types of trials separately. If studies can’t be combined meaningfully, then one should not perform a meta-analysis and instead, should stop at systematic review of the literature.
Meta-analysis is performed in two steps or levels. First step involves calculation of an effect size for each individual study. Second step is to pool the results from individual studies to calculate an overall effect size. It is important to note from this two-step or two-level approach that in meta-analysis, data is not combined from all the trials as if they are from a single trial. In other words, one can consider meta-analysis as an example of multilevel modeling.
Selection of a summary statistic to express effect size is probably one of the most important steps in performing a meta-analysis. This selection depends on the study question and the type of data at hand. There are different summary statistics for trials with events data (binary outcome) as compared to trials that report results on other scales.
In case of binary outcomes, where there are only two possibilities (for example dead or alive, sick or healthy, etc.), multiple summary statistics are available. Most commonly used summary statistics are odds ratio, relative risk ratio, relative risk reduction, absolute risk reduction, and number needed to treat. Sometimes risk ratios are expressed as percentage; however, statistical analyses are performed on original values and not on percentage values. A summary statistic should be easy to interpret and should have a reliable variance estimate which is important in performing a meta-analysis. As number needed to treat does not have such an estimate for its variance, it is not a good choice for a summary effect. Another important point is that odds ratio and relative risk ratios are combined on a natural log scale. For a typical 2x2 table following are formulas for calculating these statistics
If outcomes are on a continuous scale, choice of a summary statistic is either mean difference (if all studies used same scale for outcome measurement) or standardized mean difference (if studies used different scales for outcome measurement). For example, change in BP in response to a certain treatment is measured on the same scale and thus the summary statistic will be mean difference. On the other hand, there are multiple scales for evaluation of depression and different studies may use different scales. In such a scenario, a standardized mean-difference will be used to summarize trial results. However, pooled summary statistics obtained from meta-analysis of trials summarized with standardized mean-difference may be difficult to interpret.
Once we have a research question and we have established inclusion and exclusion criteria for relevant studies, we need to determine a search strategy to identify relevant clinical trials. In developing our search strategy, we should keep in mind that our research should find as many studies as possible, while at the same time it should be efficient. One can try to find all relevant trials ever done on a particular topic, but this is practically impossible and quite inefficient. Generally, the harder one tries to find studies, more relevant studies one will find, but after a certain number of studies are identified, every incremental effort result in a decrease in the number of identified studies. When should one stop searching for additional studies is controversial.In general, relevant studies are identified by searching medical databases. PubMed is a database containing more than 10 million references, and more than 400,000 references are added annually. It covers more than 3900 medical journals in 40 languages (88% in English). One can put “Limits” to one’s search, which helps to decrease the number of returned references. Its major deficiencies are that it covers only about 33% of medical journals and that it only goes back to 1966. A second database is EMBASE, it is somewhat larger than PubMed, but is commercial and no free version is available. Another important source of randomized controlled trials is “The Cochrane Controlled Trials Register”. This Register includes all randomized controlled trials published in 1700 medical journals. It does not contain non-randomized clinical studies. OVID Online is another database that can be searched. Although OVID is a commercial database, one can access it through Merck Medicus website. One should remember that there is overlap between these databases and most of the articles retrieved will be the same. Other databases that can be searched are AMED, BIOSIS, CINHAL, PsycINFO, and Science Citation Index. An important aspect of search for relevant clinical trials is to perform a hand-search of references of the retrieved articles as well as relevant review articles. This search usually retrieves a significant number of relevant studies that have not been properly indexed by databases.Whether one decides to include clinical trials that are not (yet) published in medical journals determines the next step in search. If one decides to search unpublished clinical trials, there are multiple resources that can be searched. For example, abstracts from relevant conference proceedings, ClinicalTrials.gov, CRISP database of NIH, or FDA database of clinical trials. Investigators can be contacted individually to learn about ongoing or recently completely but unpublished trials.If properly done, a comprehensive search of the relevant clinical trials can tremendously improve the quality of the meta-analysis. On the other hand, an incomplete search will result in publication bias (to be discussed later) which can severely compromise the results and conclusion of the meta-analysis.
Once a research question for meta-analysis has been formulated, the next step is to establish inclusion and exclusion criteria.Research question itself guides the inclusion and exclusion criteria. It excludes studies that don't fit its four components, i.e. study population (e.g. patients with CHD), exposure (e.g. an intervention vs no intervention), control population (e.g. patients without CHD), and clinical outcome (e.g. death). Only those studies that address these aspects can be considered for inclusion. There are other criteria that are often used:STUDY DESIGN: e.g. randomized placebo-controlled trials, pre-post design or repeated-measure design studies, observational studies etc.LANGUAGE: published in English only, or published in English and other languages.PUBLICATION TYPE: e.g. articles published in peer-review journals, presented at conferences, postgraduate thesis, unpublished data, etc.KEY VARIABLES: Only studies that give information about the key variables under study can be included. This information should be enough to calculate effect size (will discuss some other time).TIME FRAME: a particular start date, such as 1966, to include only modern studies.NUMBER OF SUBJECTS IN THE STUDY: Some have suggested including studies only with a large number of subjects, however, this is not generally accepted. This approach has not been evaluated fully.DURATION OF THE STUDY: in other words length of follow-up, such as at least 12 months of follow-up etc. This is important if one is interested in the long-term effects of a particular exposure (intervention).
