# Measuring the Performance of Vaccination Programs Using Cross-Sectional Surveys: A Likelihood Framework and Retrospective Analysis

**Background:**

The performance of routine and supplemental immunization activities is usually measured by the administrative method: dividing the number of doses distributed by the size of the target population. This method leads to coverage estimates that are sometimes impossible (e.g., vaccination of 102% of the target population), and are generally inconsistent with the proportion found to be vaccinated in Demographic and Health Surveys (DHS). We describe a method that estimates the fraction of the population accessible to vaccination activities, as well as within-campaign inefficiencies, thus providing a consistent estimate of vaccination coverage.

**Methods and Findings:**

We developed a likelihood framework for estimating the effective coverage of vaccination programs using cross-sectional surveys of vaccine coverage combined with administrative data. We applied our method to measles vaccination in three African countries: Ghana, Madagascar, and Sierra Leone, using data from each country's most recent DHS survey and administrative coverage data reported to the World Health Organization. We estimate that 93% (95% CI: 91, 94) of the population in Ghana was ever covered by any measles vaccination activity, 77% (95% CI: 78, 81) in Madagascar, and 69% (95% CI: 67, 70) in Sierra Leone. “Within-activity” inefficiencies were estimated to be low in Ghana, and higher in Sierra Leone and Madagascar. Our model successfully fits age-specific vaccination coverage levels seen in DHS data, which differ markedly from those predicted by naïve extrapolation from country-reported and World Health Organization–adjusted vaccination coverage.

**Conclusions:**

Combining administrative data with survey data substantially improves estimates of vaccination coverage. Estimates of the inefficiency of past vaccination activities and the proportion not covered by any activity allow us to more accurately predict the results of future activities and provide insight into the ways in which vaccination programs are failing to meet their goals.

: *Please see later in the article for the Editors' Summary*

Published in the journal:
. PLoS Med 8(10): e32767. doi:10.1371/journal.pmed.1001110

Category:
Research Article

doi: 10.1371/journal.pmed.1001110

## Summary

**Background:**

The performance of routine and supplemental immunization activities is usually measured by the administrative method: dividing the number of doses distributed by the size of the target population. This method leads to coverage estimates that are sometimes impossible (e.g., vaccination of 102% of the target population), and are generally inconsistent with the proportion found to be vaccinated in Demographic and Health Surveys (DHS). We describe a method that estimates the fraction of the population accessible to vaccination activities, as well as within-campaign inefficiencies, thus providing a consistent estimate of vaccination coverage.

**Methods and Findings:**

We developed a likelihood framework for estimating the effective coverage of vaccination programs using cross-sectional surveys of vaccine coverage combined with administrative data. We applied our method to measles vaccination in three African countries: Ghana, Madagascar, and Sierra Leone, using data from each country's most recent DHS survey and administrative coverage data reported to the World Health Organization. We estimate that 93% (95% CI: 91, 94) of the population in Ghana was ever covered by any measles vaccination activity, 77% (95% CI: 78, 81) in Madagascar, and 69% (95% CI: 67, 70) in Sierra Leone. “Within-activity” inefficiencies were estimated to be low in Ghana, and higher in Sierra Leone and Madagascar. Our model successfully fits age-specific vaccination coverage levels seen in DHS data, which differ markedly from those predicted by naïve extrapolation from country-reported and World Health Organization–adjusted vaccination coverage.

**Conclusions:**

Combining administrative data with survey data substantially improves estimates of vaccination coverage. Estimates of the inefficiency of past vaccination activities and the proportion not covered by any activity allow us to more accurately predict the results of future activities and provide insight into the ways in which vaccination programs are failing to meet their goals.

: *Please see later in the article for the Editors' Summary*

## Introduction

Immunization is a proven and cost-effective tool for control of infectious disease. Two main types of immunization activities are used to deliver vaccinations to populations, routine immunization and mass campaigns such as Supplemental Immunization Activities (SIAs). Routine immunization occurs year-round, with the aim of providing coverage for all children, as part of the World Health Organization (WHO) Expanded Program on Immunization. In contexts where immunization goals are not met by routine activities, such as measles vaccination in sub-Saharan Africa, SIAs are used to increase vaccination coverage and provide the opportunity for a second dose of vaccine. SIAs occur via campaigns at intervals generally greater than 2 y, targeting a broader range of ages. Throughout this paper, we use the term “immunization activities” to refer to both types of vaccination efforts, and specifically refer to the former as “routine vaccination” and the latter as “campaigns.”

Establishing coverage attained via these immunization activities (i.e., routine coverage and campaigns) is of clear programmatic importance. The coverage of vaccination activities is usually determined by comparing the number of doses distributed during the activity by the size of the target population (the administrative method) *[1]*. This calculation ignores vaccine wastage and failure to vaccinate inaccessible populations *[2]*, and can sometimes lead to nonsensical results such as “we vaccinated 120% of children from 9 to 48 months of age” *[1]*. A more direct approach to assessing the success of a vaccination activity is to quantify outcomes (i.e., degree of coverage attained), rather than inputs (i.e., number of vaccines distributed). One measure of outcomes is provided by Demographic and Health Surveys (DHS), nationally representative household surveys undertaken globally and geared to provide comparable data for a wide range of monitoring and impact evaluation indicators for population health, including immunization status *[3]*.

Considering vaccination outcomes (e.g., age-specific vaccination rates) yields more accurate measures of coverage *[2]*, and may also allow identification of key correlates of vaccination (e.g., rural versus urban) *[4]*. However, considering outcomes alone cannot reveal whether poor coverage is predominantly due to a proportion of the population being inaccessible to vaccination, or predominantly due to distribution inefficiencies and wastage within campaigns and routine activities. Here we show how linking the input information (doses distributed) to the outcomes (age-specific vaccination coverage) allows us to quantify the relative importance of these two components, improving our operational understanding of vaccination activities.

The “inaccessible population” includes those who refuse vaccination (perhaps accounting for the majority of the “unreachable” group in highly developed nations *[5]*) and those who do not have physical access to vaccination, e.g., people living in remote areas with little access to health care services *[6]*. In addition to groups who are literally inaccessible, the inaccessible population includes individuals not covered because of overlaps between vaccination activities larger than would be expected by chance alone (i.e., correlations in coverage). For example, overlaps may occur if vaccination activities tend to reach some sub-populations more effectively than others. Hence, while particular immunization activities may have covered more or less of the accessible population, the size of the accessible population represents an upper limit on both the coverage attained by any particular activity and the coverage of all activities combined.

Individuals who are, in theory, targeted and reachable by vaccination activities may also be missed because of inefficiencies within immunization activities, such that not all nominally distributed doses (i.e., doses reported as distributed on country reports) result in an actual new vaccinee. Vaccine wastage may result from discarded doses (due to cold chain lapses or partial use of open vials), vaccination of individuals outside the target population, or revaccination of children already vaccinated within that activity *[7]*. Note that we consider revaccination an inefficiency only if a child receives two doses in the same activity (e.g., within the same campaign), not if they are vaccinated multiple times in separate activities (e.g., receiving one routine dose and one campaign dose), which may often be desirable. These within-activity inefficiencies dictate how many new vaccinees will be gained for each new dose of vaccine added to a single vaccination activity.

Here we introduce a likelihood formulation that can be used to estimate both the size of the population inaccessible to vaccination activities and the inefficiency in the distribution of vaccine within activities. Taken together, these two quantities dictate both the rate and upper limit of improvement achievable solely by introducing new doses of vaccine into a health system. The analysis provides a method to predict the performance of past vaccination activities and future activities if no systematic changes are made. Also, it may provide some insight into where the vaccine distribution system is failing (e.g., is there a large inaccessible population, or are large numbers of doses being wasted within activities?). Our framework requires only data from a cross-sectional survey measuring age and vaccination status, and information on vaccination activity timing, age range, and number of doses deployed in the years preceding the survey. We illustrate our technique using publicly available data on measles vaccination in Ghana, Madagascar, and Sierra Leone.

## Methods

## Estimating Effective Vaccine Coverage

## Data requirements

Two sources of data are needed to estimate effective vaccine coverage using our method. The first is administrative data on vaccination activities conducted during the period of interest, including both campaigns and routine vaccination. For each vaccination activity we need to know when the activity occurred, the target age range for the activity, the number of vaccine doses nominally distributed, and the size of the target population. Routine vaccination activities occur over a broad time frame, hence must be treated differently than campaigns in statistical procedures (see “Modeling Routine Vaccination” below). However, from a data standpoint, each year's routine vaccination activities can be represented as a pseudo-campaign occurring on January 1 of that year covering all ages. The first four columns of *Table 1* illustrate the data on vaccination activities required by our approach.

The second type of data required is a cross-sectional survey of age and vaccination status in the population. This survey may be an age-stratified survey aimed specifically at this question, or any cross-sectional survey where the vaccination status of children of differing ages is obtained (e.g., a DHS survey). Data from vaccination cards, indicating a child's age at the time of routine vaccination, are not necessary but can be used to improve estimates of the age distribution of routine coverage.

## Vaccination probability and coverage

Suppose that an individual has been in the target population for vaccination activities *V*_{1}, *V*_{2},…, *V _{m}*. The probability that this individual has been vaccinated is one minus the probability that they avoid vaccination in every activity:(1)

Let us assume that there exists some population of size 1−ρ that never has a chance of being vaccinated in any activity, which we term the inaccessible population. The probability that an individual is vaccinated is then:(2)

The probability that an individual in the accessible population is vaccinated during activity *V _{j}* is some function of the number of vaccine doses nominally distributed during that activity,

*v*, the size of the population targeted by that activity,

_{j}*N*, and the proportion of that targeted population that is accessible to vaccination activities in general, ρ. If vaccine doses were distributed with perfect efficiency, then this value would be

_{j}*v*/ρ

_{j}*N*. However, it seems reasonable to assume that there is some inefficiency in the distribution of vaccine to the accessible target population, and that this inefficiency has a larger effect as more nominal vaccinations occur during an activity. That is, the first nominally distributed dose will nearly assuredly result in an additional vaccinee, but the 1,000th nominally distributed dose has a smaller chance of resulting in an additional vaccinee, and the 100,000th nominally distributed dose has a still smaller chance. We denote this inefficiency ψ, where ψ = 0 denotes an activity with perfect efficiency, i.e., every dose results in an additional vaccinee, and ψ = 1denotes a campaign that is effectively at random, i.e., your chance of receiving any vaccine dose is independent of your chance of receiving a dose previously in that activity (though unlikely, values of ψ>1 are possible, and represent activities worse than at random). Hence, the probability of an individual in the accessible target population remaining unvaccinated during activity

_{j}*V*is

_{j}*f*(

*v*, ρ

_{j}*N*, ψ) where (see

_{j}*Text S1*for derivation):(3)

We can now formalize and simplify Equation 2 above to an expression of the probability that any individual has been vaccinated:(4)Where *V*_{1}, *V*_{2},…, *V _{m}* now denotes all vaccination activities that anyone in the population has been exposed to, and

*z*is an indicator of whether person

_{ij}*i*was in the target population for vaccine activity

*j*, which is fully determined by the child's age (

*x*).

_{i}The coverage of a particular vaccination activity, *c _{j}*, is the same as the probability that an individual who is in the target population for only that activity (i.e.,

*z*= 1 and

_{ij}*z*= 0 for all

_{ik}*k*≠

*j*) is vaccinated. Hence, the expected coverage of activity

*j*is:(5)

*Figures 1* and *2* illustrate how ρ and ψ affect actual coverage. The accessible population, ρ, represents the upper limit of coverage, while the efficiency parameter, ψ, dictates the expected improvement in coverage from the introduction of additional vaccine doses.

## Modeling routine vaccination

In the simplest formulation, routine vaccination activities can be treated as a single pseudo-campaign occurring when a child reaches the age of recommended vaccination (9 mo for measles), with the routine vaccination coverage reported during the year of that event. However, this approach ignores the fact that different children receive their first dose of vaccine from routine immunization activities at different ages. We can account for this by creating a series of pseudo-campaigns representing the routine vaccination coverage of each year when child *i* was alive. By weighting each pseudo-campaign by the probability of a child having the “opportunity” to be vaccinated in a year's routine activities (*w _{ij}*), these pseudo-campaigns can be combined to obtain a child's probability of routine vaccination in their lifetime up to that point (see

*Text S1*):(6)where

*j*= 1…

*R*are the years of routine vaccination activities and represents the probability that the opportunity for routine vaccination occurs after a child's current age. These weights are calculated as (see

*Text S1*):(7)(8)where

*F*(

_{R}*x*) is the probability of having the opportunity for vaccination by age

*x*,

*x*is child

_{ij}*i*'s age at the beginning of routine vaccination year

*j*, and

*l*is the length of exposure to a year's routine activities (12 mo for most years, but truncated in the year of the survey). This new combination of pseudo-campaigns can be included in Equation 4 above as a single vaccination activity.

_{j}Sophisticated distributions and forms of estimation for *F _{R}*(

*x*) are possible, but here we make the simplifying assumption that there is a constant “hazard” λ of routine vaccination after 8.5 mo of age and estimate this hazard in conjunction with the other model parameters. That is:(9)

## Estimation

In a cross-sectional survey we observe a set of individuals with ages ** x** = {

*x*

_{1},

*x*

_{2},…,

*x*} and a set of corresponding vaccination statuses,

_{n}**= {**

*y**y*

_{1},

*y*

_{2},…,

*y*}, where

_{n}*y*= 0 indicates that an individual has never been vaccinated, and

_{i}*y*= 1 indicates that he has. Using the formulation from Equation 4, the probability of observing

_{i}*y*= 1 is

_{i}*g*(

*x*, ρ, ψ), and the probability of observing

_{i}*y*= 0 is 1−

_{i}*g*(

*x*, ρ, ψ). Assuming that

_{i}*y*

_{1},

*y*

_{2},…,

*y*are independent stochastic variables, the likelihood of the parameters ρ and ψ given these observations can be expressed as the product of the probability of each observation:(10)Numeric optimization (e.g., Nelder-Mead) or Markov chain Monte Carlo (MCMC) methods can be used to estimate these parameters.

_{n}We extend Equation 10 to use data on the age at time of vaccination for those with vaccine cards to better fit λ, optimizing:(11)where *n*′ is the number of children with a vaccination card, *r _{i}* is the age of routine vaccination on that card, and

*h*(

*r*, λ) is the probability distribution function for Equation 9. This formulation assumes that the age of routine vaccination is independent of the probability of vaccination given that a child has a vaccination card and that vaccinations recorded on vaccination cards only represent routine vaccination.

_{i}## Application to Measles Vaccination

We used data from WHO and country DHS to estimate the effectiveness of measles vaccination activities in Ghana, Madagascar, and Sierra Leone. Countries were selected with the criteria that at least one campaign (i.e., an SIA) occurred within the 5 y prior to the most recent DHS survey, that no campaign occurred in the same year as the most recent DHS survey, and that countries with differing reported vaccination coverage were represented.

Values for the number of doses nominally administered during routine vaccination and the size of the target population are based on the values reported by countries to WHO (data provided by WHO). WHO–United Nations Children's Fund (UNICEF) estimates of national immunization coverage for comparison were obtained from WHO *[8]*. Information on when campaigns occurred, doses deployed during campaigns, and the age range targeted were obtained from WHO.

Measles vaccination status for children from 9 to 59 mo of age was obtained from country-specific DHS surveys *[9]*–*[11]*. Children were considered to have been vaccinated for measles if vaccination was recorded on the child's health card or the child's mother reported that the child had been vaccinated (*Table 2*). Vaccination status, age, and timing of interview were obtained for 2,304 children from Ghana (DHS survey in 2008), *n* = 9,747 for Madagascar (DHS survey in 2008–2009), and *n* = 3,966 for Sierra Leone (DHS survey in 2008). Age at time of routine vaccination was calculated for those children where the DHS data indicated a vaccination card had been seen and a date of vaccination was recorded (*n* = 1,550 for Ghana, 3,787 for Madagascar, and 1,216 for Sierra Leone).

Parameter estimates and 95% confidence intervals were obtained using MCMC methods (the Metropolis-Hasting algorithm). All MCMC chains were started from ρ = 0.5, ψ = 1, and λ = 1. Model convergence was checked by examination of MCMC chains, comparison of the posterior distributions estimated from different chains, calculation of *[12]*, and comparison with results from numerical fitting procedures (Nelder-Mead). Five chains of length 5,000 were run for each country, and the posterior distribution was estimated to be the empirical distribution of the combined set of the last 2,500 iterations from all chains. Reported parameter estimates are the medians of the posterior distributions (posteriors were normally distributed on the scale used in estimation), and 95% confidence intervals from quantiles of the posterior distribution. The estimated coverage for each campaign and confidence intervals were similarly obtained from the posterior distribution created by applying Equation 5.

Confidence intervals on the age distribution of vaccination reported in DHS data were obtained by bootstrapping (500 iterations). Confidence intervals for model estimates of the age distribution of vaccination were obtained by performing 500 parametric bootstrap iterations where (a) parameters were sampled from the posterior distribution, (b) a bootstrap population is created based on the DHS data, and (c) each individual in the bootstrap population is randomly assigned a vaccination status based on the parameters selected in step a. This procedure accounts for (a) uncertainty in parameter estimates, (b) uncertainty in population structure, and (c) uncertainty from the Bernoulli process; confidence intervals are thus comparable with those obtained from the DHS data.

The age distribution that would have resulted from naïve use of WHO-corrected estimates of coverage was calculated assuming independence between vaccination activities and routine vaccination at 9 mo of age. Campaigns were assumed to have coverage that differed from that reported by the same proportion as the routine vaccination activities occurring that year. Confidence intervals were calculated using steps b and c above.

All statistical analyses were done in the R statistical package, version 2.11 (http://cran.r-project.org).

## Results

We estimate that 7% (95% CI: 6, 9) of children in Ghana, 23% (95% CI: 24, 22) of children in Madagascar, and 31% (95% CI: 33, 30) of children in Sierra Leone were never accessible by routine measles vaccination or campaigns (*Table 3*). The estimated inefficiency within vaccination activities was highest in Madagascar (ψ = 0.34, 95% CI: 0.28, 0.41), followed by Sierra Leone (ψ = 0.33, 95% CI: 0.31, 0.39), then Ghana (ψ = 0.03, 95% CI: 0.02, 0.04). Hence, our estimates of routine and SIA coverage are substantially lower than administrative estimates for Madagascar and Sierra Leone, and only slightly lower for Ghana (*Table 1*). Our estimates of routine coverage are in general lower than the WHO-UNICEF estimates generated by a heuristic method.

Based on our estimated routine vaccination distribution (λ; *Table 3*), children in all three countries who receive routine vaccination do so near their 9-mo birthday. In Ghana, children receive routine vaccination at a slightly younger age (mean age = 10.0 mo) than in Madagascar (mean age = 10.2 mo) or Sierra Leone (mean age = 10.6 mo). These estimates are slightly lower than estimates obtained from the empirical distribution of ages reported on vaccination cards (mean = 10.4, 10.8, and 12.2 mo respectively), reflecting that the constant rate assumption is not precisely correct.

A comparison of the age-specific proportion vaccinated predicted by our approach with the proportion vaccinated observed in the DHS data shows substantial agreement (*Figure 3*). For Madagascar and Sierra Leone, our predictions show substantially better agreement with the DHS data than what is predicted by naïve application of WHO-adjusted coverage estimates. For Ghana, where vaccination coverage is relatively high, the unreachable population is small, and wastage very low, projections from WHO generally agree with our estimates and the proportion vaccinated observed in the DHS.

With the exception of Ghana, we find that the models including both ρ and ψ as free parameters model the data substantially better, as measured by Akaike information criterion, than models where we assume a fully accessible population (ρ = 1) or perfect efficiency (ψ = 0) (*Table S1*). In the case of Ghana, where ψ is estimated to be nearly zero, assuming perfect efficiency leads to a slightly improved but qualitatively equivalent model of the DHS data.

## Discussion

Reliable estimates of vaccination coverage are key to managing population immunization status. Estimates of coverage are usually based on administrative measures (i.e., the ratio of distributed doses to the size of the target population). Refinements of this approach do exist; however, current WHO adjustments of estimates of coverage employ a heuristic method in which “no statistical or mathematical models are used.” *[1]*. Data quality audits *[13]* provide a verification factor that as yet has not proved sufficiently stable to be useful *[14]*. Here we have introduced a method by which administrative coverage estimates can be combined with a cross-sectional survey to estimate the effectiveness of vaccination programs. This method not only attempts to correct coverage estimates, but also distinguishes between issues of overall coverage and vaccine wastage within vaccination activities.

The causes of variation between country-specific levels of inefficiency and overlap between campaigns (*Table 3*) may be diverse, but general conclusions about immunization activities emerge. While a combination of routine immunization via the WHO Expanded Program on Immunization and campaigns can successfully maintain high coverage in countries like Ghana, the analysis shows how in contexts like that of post-conflict Sierra Leone, or Madagascar, the strategy itself may be inadequate. For example, in Sierra Leone, despite investment in two campaigns in 3 y, both these campaigns and the underlying routine program failed to reach 30% of children; and in 2010, there was a large outbreak in children aged greater than 5 y in Freetown. More generally, wherever such a large proportion of the population remains inaccessible, e.g., when there is high overlap between campaigns, the combined strategy of vaccination via the Expanded Program on Immunization and via SIAs is unlikely to succeed, and refocusing effort into the design of vaccination strategy (e.g., improving the vaccine distribution structure and finding novel ways to target unreached populations) should yield considerable rewards in terms of improved coverage.

For our technique to be useful, countries must have cross-sectional data on vaccine coverage for children across a range of ages, some of an age where they have been exposed to multiple vaccination activities (e.g., one or more campaigns and routine vaccination). As illustrated here, a DHS survey provides sufficient information, but surveys aimed specifically at measuring coverage that target a wider age range (ideally paired with serosurveys) could improve estimates. Once countries have estimates of the accessible population and within-campaign inefficiencies, they can predict the coverage of activities occurring after the cross-sectional survey (using Equation 5) and the age-specific coverage obtained after these activities (using Equation 4). With these estimates in hand, countries can better understand how susceptibility may be building up in their population, perhaps enabling them to avoid outbreaks of the type and scale observed recently in Sierra Leone, Malawi, Zambia, and South Africa.

It may seem surprising that both ψ and ρ are identifiable using only age-stratified vaccination prevalence. However, simulations show that differences in these values lead to significant differences in the age profile of vaccination coverage when children of differing ages have differing exposure to multiple vaccination activities (*Figures S1* and *S2*). If children are not exposed to multiple activities (e.g., there is only routine vaccination), it will not necessarily be possible to distinguish between these two sources of program inefficiency. A critical way estimates could be improved is by conducting cross-sectional surveys of vaccination coverage covering ages greater than 5 y (the current upper limit in DHS surveys). Such surveys would include individuals who had been potentially covered by more vaccination campaigns, improving estimates and allowing for separate analysis of inefficiencies in routine vaccination and SIAs.

Limitations of our approach include required assumptions that may not always be justified. The assumption of constant inefficiencies across the study period may not be appropriate, especially in the context of a global measles elimination campaign. A potential extension to account for this variation would be to allow estimates of ρ and ψ to vary smoothly over time. The assumption that the rate at which children have the opportunity to receive routine vaccination is constant after 8.5 mo of age is clearly an oversimplification. While more sophisticated techniques can be used to fit this age distribution, as in *[15]*, the simplified approach still fits the age distribution seen in the DHS data well (*Figure 3*).

Our sample estimates are also subject to the quality of the available data. We assume that the target population is accurately estimated. A sensitivity analysis of the effect of over- or underestimates of the target population indicates that such misspecifications do not much bias estimates of the size of the unreachable population, but do impact estimates of within-campaign efficiency (*Table S2*). Underestimates of the size of the target population lead to underestimates of within-campaign efficiency (i.e., overestimates of ψ), and overestimates lead to overestimates of within-campaign efficiency (i.e., underestimates of ψ). Hence, the high within-campaign inefficiency estimates for Sierra Leone and Madagascar could be partially the result of poorly specified denominators, particularly as in both cases at least some of the immunization activities were performed long after the last census (*Table S3*).

We also assume that the DHS data are representative of each country. If DHS surveys in fact missed populations that were also missed by immunization activities, the size of the population inaccessible to vaccination would be underestimated. Another potential source of bias is that a high proportion of the vaccination data come from reporting by the mother (*Table 2*), particularly in Sierra Leone. If mothers report more children have been vaccinated than is in fact the case, both the size of the accessible population and the efficiency of campaigns may be overestimated, and the reverse if mothers report fewer children vaccinated than there really are. Additionally, in all countries, the proportion of reports from the mother increases with child age, reflecting the fact that vaccination cards are more rarely distributed during SIAs. This could lead to either over- or underreporting of vaccination occurring during SIAs. If underreporting of campaign coverage is occurring, the model will tend to both underestimate overall coverage and overestimate within-campaign inefficiency. Conversely, if campaign coverage is overreported, our technique will underestimate within-campaign inefficiencies. Age-specific serosurveys would provide a valuable benchmark by which to evaluate the coverage estimates, and could perhaps be paired with existing research, monitoring, or vaccination activities.

The estimates obtained by our method bear an inconsistent relationship to the WHO-UNICEF adjusted estimates (*Table 1*). In some places we estimate significantly lower coverage (e.g., Sierra Leone in 2004) and in others we estimate higher coverage (e.g., Ghana in 2005). In all cases our estimates are the result of the reported administrative coverage and the estimated model parameters. Where we estimate a large accessible population and low inefficiency, our estimates will be high relative to administrative coverage; where we estimate a small accessible population and high inefficiency, our estimates will be relatively low. We assume model parameters are constant over the 5 y considered, while the heuristics used in the WHO-UNICEF estimates may capture specific short-term factors of which we are unaware *[1]*, and hence may be more accurate for individual years. However, without further information, it is unclear how to combine yearly WHO-UNICEF estimates to the get full age distribution of vaccine coverage (assuming independence clearly performs poorly; *Figure 3*). Since the assessment of coverage from multiple activities is integral to our approach, our approach has some clear advantages despite its limitations.

The method presented here provides a way in which the performance of vaccination activities can be more accurately measured (and can be extended to consider, e.g., the problem of access to a second dose; *Figure S3*). As illustrated by our examples, this approach can be used to produce estimates of effective vaccine coverage that are more consistent with the proportion of the population reporting vaccination than current approaches are. These estimates go beyond mere measures of cross-sectional coverage obtained directly from a DHS survey, characterizing the performance of the activities leading to that coverage, and helping to predict the effect of future vaccination activities. Our estimates can be used to identify those countries where problems in vaccine delivery may exist (e.g., Madagascar and Sierra Leone), thereby providing important operational guidance as to how vaccine coverage may be improved. Such guidance is essential if measles control goals are to be met.

## Supporting Information

##### Zdroje

1. BurtonAMonaschRLautenbachBGacic-DoboMNeillM 2009 WHO and UNICEF estimates of national infant immunization coverage: methods and processes. Bull World Health Organ 87 535 541

2. MurrayCJLShengeliaBGuptaNMoussaviSTandonA 2003 Validity of reported vaccination coverage in 45 countries. Lancet 362 1022 1027

3. BrownJMonaschRBicegoGBurtonABoermaJT 2002 Assessment of the quality of estimates of child immunization coverage from population-based surveys: MEASURE Evaluation Working Paper series Chapel Hill (North Carolina) University of North Carolina at Chapel Hill

4. MunthaliAC 2007 Determinants of vaccination coverage in Malawi: evidence from the demographic and health surveys. Malawi Med J 19 79 82

5. WallingaJHeijneJCMKretzschmarM 2005 A measles epidemic threshold in a highly vaccinated population. PLoS Medicine 2 e316 doi:10.1371/journal.pmed.0020316

6. GraisRFDubrayCGerstlSGuthmannJPDjiboA 2007 Unacceptably high mortality related to measles epidemics in Niger, Nigeria, and Chad. PLoS Med 4 e16 doi:10.1371/journal.pmed.0040016

7. McBeanAMFosterSOHerrmannKLGateffC 1976 Evaluation of a mass measles immunization campaign in Yaoundé Cameroun. Trans R Soc Trop Med Hyg 70 206 212

8. World Health Organization 2010 WHO-UNICEF estimates of MCV coverage Geneva World Health Organization Available: http://apps.who.int/immunization_monitoring/en/globalsummary/timeseries/tswucoveragemcv.htm. Accessed 15 September 2010

9. Ghana Statistical Service, Ghana Health Service, ICF Macro 2009 Ghana demographic and health survey 2008 Accra (Ghana) Ghana Statistical Service

10. Institut National de la Statistique, ICF Macro 2010 Enquête démographique et de santé de Madagascar 2008–2009 Antananarivo (Madagascar) Institut National de la Statistique

11. Statistics Sierra Leone, Ministry of Health and Sanitation, ICF Macro 2009 Sierra Leone demographic and health survey 2008 Freetown (Sierra Leone) Statistics Sierra Leone

12. GelmanARubinDB 1992 Inference from iterative simulation using multiple sequences. Stat Sci 7 457 472

13. World Health Organization 2003 The immunization data quality audit (DQA) procedure. WHO/V&B/03.19 Geneva World Health Organization

14. RonveauxORickertDHadlerSGroomHLloydJ 2005 The immunization data quality audit: verifying the quality and consistency of immunization monitoring systems. Bull World Health Organ 83 504 510

15. LesslerJMossWJLowtherSACummingsDAT 2010 Maintaining high rates of measles immunization in Africa. Epidemiol and Infect E-pub ahead of print. doi: 10.1017/S0950268810002232

16. United Nations Economic Commission for Africa 2007 Reference regional strategic framework for statistical capacity building in Africa: baseline information—data development [database]. Available: http://www.uneca.org/eca_programmes/policy_analysis/rrsf/Baseline_datadev.htm. Accessed 4 November 2010

##### Štítky

Interní lékařstvíČlánek vyšel v časopise

### PLOS Medicine

2011 Číslo 10

Nejčtenější v tomto čísle

- STrengthening the Reporting of OBservational studies in Epidemiology – Molecular Epidemiology (STROBE-ME): An Extension of the STROBE Statement
- A Statistical Model of the International Spread of Wild Poliovirus in Africa Used to Predict and Prevent Outbreaks
- The Effect of Chromosome 9p21 Variants on Cardiovascular Disease May Be Modified by Dietary Intake: Evidence from a Case/Control and a Prospective Study
- Measuring the Performance of Vaccination Programs Using Cross-Sectional Surveys: A Likelihood Framework and Retrospective Analysis