Performance of the Tariff Method: validation of a simple additive algorithm for analysis of verbal autopsies

The, James, Flaxman, Murray

Description

Verbal autopsies provide valuable information for studying mortality patterns in populations that lack reliable vital registration data. Methods for transforming verbal autopsy results into meaningful information for health workers and policymakers, however, are often costly or complicated to use. We present a simple additive algorithm, the Tariff Method (termed Tariff), which can be used for assigning individual cause of death and for determining cause-specific mortality fractions (CSMFs) from verbal autopsy data. Methods Tariff calculates a score, or "tariff," for each cause, for each sign/symptom, across a pool of validated verbal autopsy data. The tariffs are summed for a given response pattern in a verbal autopsy, and this sum (score) provides the basis for predicting the cause of death in a dataset. We implemented this algorithm and evaluated the method's predictive ability, both in terms of chance-corrected concordance at the individual cause assignment level and in terms of CSMF accuracy at the population level. The analysis was conducted separately for adult, child, and neonatal verbal autopsies across 500 pairs of train-test validation verbal autopsy data. Results Tariff is capable of outperforming physician-certified verbal autopsy in most cases. In terms of chance-corrected concordance, the method achieves 44.5% in adults, 39% in children, and 23.9% in neonates. CSMF accuracy was 0.745 in adults, 0.709 in children, and 0.679 in neonates. Conclusions Verbal autopsies can be an efficient means of obtaining cause of death data, and Tariff provides an intuitive, reliable method for generating individual cause assignment and CSMFs. The method is transparent and flexible and can be readily implemented by users without training in statistics or computer science.

Sujets

• Validation
• Gold standard
• Cause of Death
• Mortality

Informations

• Publié par: biomed
• Publié le: 01 janvier 2011
• Nombre de lectures: 6
• Langue: English
• Poids de l'ouvrage: 1 Mo

Extrait

James et al. Population Health Metrics 2011, 9:31
http://www.pophealthmetrics.com/content/9/1/31
RESEARCH Open Access
Performance of the Tariff Method: validation of a
simple additive algorithm for analysis of verbal
autopsies
1 1 1*Spencer L James , Abraham D Flaxman and Christopher JL Murray for
the Population Health Metrics Research Consortium (PHMRC)
Abstract
Background: Verbal autopsies provide valuable information for studying mortality patterns in populations that lack
reliable vital registration data. Methods for transforming verbal autopsy results into meaningful information for
health workers and policymakers, however, are often costly or complicated to use. We present a simple additive
algorithm, the Tariff Method (termed Tariff), which can be used for assigning individual cause of death and for
determining cause-specific mortality fractions (CSMFs) from verbal autopsy data.
Methods: Tariff calculates a score, or “tariff,” for each cause, for each sign/symptom, across a pool of validated
verbal autopsy data. The tariffs are summed for a given response pattern in a verbal autopsy, and this sum (score)
provides the basis for predicting the cause of death in a dataset. We implemented this algorithm and evaluated
the method’s predictive ability, both in terms of chance-corrected concordance at the individual cause assignment
level and in terms of CSMF accuracy at the population level. The analysis was conducted separately for adult, child,
and neonatal verbal autopsies across 500 pairs of train-test validation verbal autopsy data.
Results: Tariff is capable of outperforming physician-certified verbal autopsy in most cases. In terms of
chancecorrected concordance, the method achieves 44.5% in adults, 39% in children, and 23.9% in neonates. CSMF
accuracy was 0.745 in adults, 0.709 in children, and 0.679 in neonates.
Conclusions: Verbal autopsies can be an efficient means of obtaining cause of death data, and Tariff provides an
intuitive, reliable method for generating individual cause assignment and CSMFs. The method is transparent and
flexible and can be readily implemented by users without training in statistics or computer science.
Keywords: Verbal autopsy, validation, gold standard, Tariff Method, cause of death, mortality, cause-specific
mortality fractions
Background comparative performance needs to be evaluated.
LargeVerbal autopsies (VAs) are increasingly being used to scale validation studies, such as the Population Health
provide information on causes of death in demographic Metrics Research Consortium (PHMRC) [10], provide
surveillance sites (DSSs), national surveys, censuses, and objective information on the performance of these
difsample registration schemes [1-3]. Physician-certified ferent approaches.
The main limitation to date of PCVA is the cost andverbal autopsy (PCVA) is the primary method used to
assign cause once VA data are collected. Several alterna- feasibility of implementation. Finding and training
physitive expert-based algorithms [4-6], statistical methods cians to read VAs in resource-poor settings has proven
[7-9], and computational algorithms [7] have been challenging, leading in some cases to long delays in the
developed. These methods hold promise, but their analysis of data [1,11]. In some rural areas with marked
shortages of physicians, assigning the few available
physicianstoreadVAsmayhaveaveryhighopportunity* Correspondence: [e-mail]
1Institute for Health Metrics and Evaluation, University of Washington, 2301 cost in terms of health care delivery. Lozano et al. [12]
Fifth Ave., Suite 600, Seattle, WA 98121, USA
© 2011 James et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons
Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in
any medium, provided the original work is properly cited.James et al. Population Health Metrics 2011, 9:31 Page 2 of 16
http://www.pophealthmetrics.com/content/9/1/31
have also shown that there is a substantial idiosyncratic Methods
element to PCVA related to physician diagnostic perfor- Logic of the method
mance. In contrast, some automated methods (whether The premise behind the Tariff Method is to identify
statistical or computational in nature) have demon- signs or symptoms collected in a VA instrument that
strated performance similar to PCVA [7,8], but some are highly indicative of a particular cause of death. The
users may be uncomfortable with the “black box” nature general approach is as follows. A tariff is developed for
of these techniques. It is often very difficult for users to each sign and symptom for each cause of death to
unpack how decisions on a cause are reached. Further- reflecthowinformativethatsignandsymptomisfor
more, the actual statistics and mechanics that form the that cause. For a given death, based on the response
basis for cause assignments are difficult to access and pattern in the VA instrument, the tariffs are then
understand due to the myriad computations involved. summed yielding an item-specific tariff score for each
One method, the King-Lu method, is a direct cause-spe- death for each cause. The cause that claims the highest
cific mortality fraction (CSMF) estimation approach tariff score for a particular death is assigned as the
pre[13,14] that does not assign cause to specific deaths, dicted cause of death for that individual. The tariffs,
tarmaking it even harder for a user to understand how the iff scores, and ranks are easily observable at each step,
cause of death is being determined. and users can readily inspect the basis for any cause
Empirical methods that use the observed response decision.
pattern from VAs in a training dataset have an advan- Based on a training dataset in which the true cause is
tage over expert judgment-based methods in that they known and a full verbal autopsy has been collected, we
capturetherealitythatsomehouseholdrespondentsin can compute a tariff as a function of the fraction of
a VA interview may respond “yes” to some items even deaths for each variable or item that has a positive
when they would not be considered part of the classical response. The tariff can be thought of as a robust
esticlinical presentation for that cause. For example, 43% of mate of how different an item response pattern is for a
households report coughing as a symptom for patients cause compared to other causes, formally:
who died from a fall, and 58% of households report a
x −Median xij ijfever for patients who died from a road traffic accident. Tariff =ij
Interquartile Range xHowever, a limitation of many existing methods such as ij
Simplified Symptom Pattern and Random Forest is that
where tariff is the tariff for cause i, item j, x is theij ijthey may not give sufficient emphasis to pathognomonic
fraction of VAs for which there is a positive response to
signs and symptoms. For example, if 20% of patients
deaths from cause i for item j, median(x)isthemedianijdying of epilepsy report convulsions, and only 2% of
fraction with a positive response for item j across allnonepilepsy patients report c a statistical
causes, and interquartile range x is the interquartileijmodel will not assign this symptom as much
signifirange of positive response rates averaged across causes.cance as these data imply. Put another way, Bayesian
Note that as defined, tariffs can be positive or negativemethods such as InterVA and Symptom Pattern and
in value. As a final step, tariffs are rounded to the near-statistical methods such as King-Lu direct CSMF
estiest 0.5 to avoid overfitting and to improve predictivemation assume that the probability of signs and
sympvalidity.toms conditional on true cause is constant, but in reality
For each death, we compute summed tariff scores forit is not. There are subsets of patients who may have
each cause:signs and symptoms that are extremely informative, and
wother subsets with less clearly defined signs/symptoms.
Tariff Score = Tariff xIn this paper, we propose a simple additive approach ki ij jk
j=1using transparent, intuitive computations based on
responses to a VA instrument. Our premise is that there
where x is the response for death k on item j, takingjk
ought to be highly informative signs or symptoms for
on a value of 1 when the response is positive and 0
each cause. Our goal is to develop an approach to cause
when the response is negative, and w is the number of
of death estimation based on reported signs and
sympitems used for the cause prediction. It is key to note
toms that is simple enough to be implemented in a
that for each death, a different tariff score is computed
spreadsheet so that users can follow each step of cause
for each of the possible causes. In the adult module of
assignment. We illustrate the development of this
the PHMRC study, for example, there are 46 potential
approach and then use the PHMRC gold standard VA
causes and so there are 46 different tariff scores based
validation study dataset [10] to assess the performance
on the tariffs and the response pattern for that death.
of this approach compared to PCVA, which is current
For actual implementation, we use only the top 40 items
practice.James et al. Population Health Metrics 2011, 9:31 Page 3 of 16
http://www.pophealthmetrics.com/content/9/1/