Reducing Missed Opportunities to Vaccinate During ...

ORIGINAL ARTICLE

Reducing Missed Opportunities to Vaccinate During Child Health Visits How Effective Are Parent Education and Case Management? David Wood, MD, MPH; Mark Schuster, MD, PhD; Cathy Donald-Sherbourne, PhD; Naihua Duan, PhD; Rabecca Mazel, MA; Neal Halfon, MD, MPH

Background: At child health visits, immunizations that are due are frequently not given. Increased parent understanding of and demand for immunizations may influence providers to not miss these opportunities. Objective: To assess, as part of a larger study of effec-

tiveness of parent education and case management (CM) in raising immunization rates, the intervention’s effectiveness at reducing missed opportunities to vaccinate during child health visits. Methods: A representative sample of African Ameri-

can newborns and their families from south central Los Angeles, Calif, were randomly assigned to a control or a CM group and observed during the first year of life. Case managers visited and telephoned parents, educating them on the benefits and safety of immunizations, and encouraging them to request immunizations from providers. When the children were at least 1 year of age, parents were interviewed and provider records were abstracted. Results: Complete records were abstracted for 126 con-

trols and 129 CM group children. For these children, 1092 visits were documented where immunizations were due. Missed opportunities to vaccinate occurred at more than 50% of visits. Case management was associated with a modest reduction in the percentage of visits with missed opportunities in the bivariate analysis but not after adjustment for other covariates. In a logistic regression model, missed opportunities were more frequent at visits with private than public physicians and at acute illness than wellchild visits. Missed opportunities were less frequent among children with a history of at least 1 cancelled appointment, and for visits of children with mothers who smoked. Conclusions: Missed opportunities were minimally in-

fluenced by a home visitation and parent education program. They are primarily determined by issues under the control of the provider. Family- and child-related characteristics, however, do influence the likelihood of a missed opportunity occurring independent of provider factors. Arch Pediatr Adolesc Med. 1998;152:238-243

Editor’s Note: So much for the effectiveness of case manage-

ment—at least as it relates to immunization. Any other ideas? Catherine D. DeAngelis, MD

From RAND, Santa Monica, Calif (Drs Wood, Schuster, Donald-Sherbourne, Duan, Mazel, and Halfon); Shriners Hospitals for Children, Tampa, Fla (Dr Wood); Department of Pediatrics, School of Medicine, University of California, Los Angeles (Dr Schuster); and Department of Community Health, School of Public Health, University of California, Los Angeles (Dr Halfon).

I

MMUNIZATION RATES for certain groups of preschool children remain below the national target of 90% up to date by 2 years of age.1 A number of studies have documented that, during almost half of child health visits, immunizations are due but not given. These missed opportunities to vaccinate (MOs) significantly contribute to the problem of underimmunization of young children.2-7 Studies have associated MOs with provider knowledge of the appropriate contraindications to immunization and the immunization schedule.8-11 A number of studies have shown

that a provider’s compliance with preventive health screenings can be improved through interventions to educate and empower patients.12-14 Similarly, it has been demonstrated that parental perception of and demand for immunizations can influence the child’s receipt of immunizations.15-18 Misconceptions held by both professionals and lay people (eg, that immunizations are dangerous or ineffective when given to a mildly ill child) inhibit the delivery of immunizations. In a previous study, we found that the majority of African American mothers believed that it was unsafe to give immunizations to a mildly ill child.19,20 More accurate parental knowledge of immunization safety, effectiveness, and timing may increase demand for timely immunizations. Home visitation has been shown to be effective at providing parents with ac-

ARCH PEDIATR ADOLESC MED/ VOL 152, MAR 1998 238

©1998 American Medical Association. All rights reserved.

SUBJECTS AND METHODS SAMPLE We enrolled a representative sample of 419 African American infants born into a 10–ZIP code area in inner-city Los Angeles, Calif, during a 3-month period (February through early May) in 1994 (more detail on the sample selection process is available elsewhere24,25). They were randomly assigned at entry into a CM or a control group, and they received follow- up for more than 1 year. We successfully followed up and interviewed 185 members of the intervention group and 180 members of the control group. During the interview, we asked them to identify all providers the child had seen in the first year of life. We were able to obtain and abstract the complete medical records from all reported providers for 255 children (70%)—129 intervention children and 126 control children. We compared the 255 families and children used in this analysis with the 110 for whom we did not have complete provider records. They were comparable on all demographic and health utilization characteristics except for health insurance (10% of the abstracted group vs 5.4% of others had private insurance), knowledge of contraindications scale (P=.005), and history of being breast-fed (P=.02). These factors are controlled for when the main effect of group (intervention effect) on MOs is compared in the multivariate analysis. INTERVENTION Both groups were given health passports produced by the State of California, which contain information on the recommended visits for preventive health care and the childhood immunization schedule approved by the Centers for

curate information and supporting them in their effort to seek health services.21-23 In a previous article, we showed that case management (CM) intervention was effective at raising immunization levels among inner-city, African American infants.24 We hypothesized that a CM program could increase parental demand for immunizations during child health visits, thereby reducing MOs. In this article, we focus our analysis on the effect of the CM intervention on MOs during child health visits. RESULTS

The 2 experimental groups (CM and control) were comparable at baseline on all demographic and health care system utilization characteristics in Table 1. The CM intervention was associated with a trend toward a lower rate of MOs (47.7% vs 53.3%; P=.07) (Table 2). Comparing the MO rates between the control and CM group across independent variable categories, the control group had MO rates higher by approximately 5 to 6 percentage points in almost every category (range, 3.4-9.3 percentage points). For within–intervention group comparisons, MOs were (1) more frequent for visits at private physicians’ offices and health maintenance organization offices than for visits at public clinics (P,.001); (2) less frequent for children of women who smoked (P=.004);

Disease Control and Prevention (CDC).26 Case managers conducted an in-depth assessment in the home of the client generally by 6 weeks of age. Subsequent visits were scheduled for 2 weeks before the next preventive health visit when immunizations were due. Case managers also followed up with clients by telephone and mail after scheduled appointments. The mean (±SD) numbers of contacts of case managers with clients were 4.0±1.9 for home visits, 7.5±3.9 for telephone contacts, and 4.4±2.5 for mailings. At each visit, the case manager documented that the client understood which immunizations were due at the next visit, and had a preventive health visit appointment at the appropriate interval. If barriers to care were expressed, the case manager helped the mother overcome them. The case managers specifically sought to assure parents that it was safe to give immunizations if the child had a cold, mild ear infection, or other mild illness. The case manager also encouraged the mothers to be proactive and request immunizations from their providers. DATA COLLECTION At the end of the intervention, we interviewed the mothers and collected demographic information (eg, family structure, birth order, maternal education, maternal smoking history), information on health care system utilization (type of health insurance, months of health insurance in the child’s first year of life, and type of regular provider), and parental knowledge of the immunization schedule (2-item scale) and of contraindications to vaccination (3-item scale). Each provider’s chart was coded for the presence or absence of a separate immunization record. For children with multiple providers, visits were merged. For each visit, the following Continued on next page

(3) more frequent for children who had been breast-fed (P=.001); and (4) more frequent for visits occurring from 6 to 12 months of age than 0 to 6 months of age (P,.001). There was a trend for lower MO rates among those continuously receiving Medicaid insurance vs the other insurance groups. There was no significant within-group relationship between MOs and maternal education or months of insurance coverage. For the control group, greater knowledge of the immunization schedule was significantly associated with a higher rate of MOs (P,.01). For children in the CM group, greater knowledge of immunization contraindications was associated with a lower MO rate (P,.01). Missed opportunities were much more common at sick visits than well visits (P,.001) (Table 3). For the control group only, MOs were more frequent when the chart lacked an immunization record (P,.05). Within the control group, the rate of MOs was similar for visits of children with and without a history of cancelled appointments. However, within the CM group, MO rates were lower by 14.4 percentage points for children with a history of cancelled appointments compared with those with no history of cancelled appointments (P,.01). In both the control and CM groups, MOs tended to be more common during visits of children who reported having had more than 1 provider during the year (P=.09). Missed



information was abstracted: whether it was a preventive health or an illness visit, date of visit, all diagnoses recorded, temperature if recorded, and immunizations administered. DEFINITION OF MO As defined by the CDC, an MO occurs at a visit at which “immunizations were due but not given and no contraindications were present.”27 According to the CDC definition, a contraindication to vaccination can be justified on the basis of either (1) a vaccine-specific contraindication (eg, temperature higher than 40.6°C with a previous diphtheria, tetanus, and pertussis vaccination or allergy to eggs for measles, mumps, and rubella vaccination) or (2) “the child being too sick to be immunized at that visit.” No vaccine-specific contraindications were recorded during the 1092 visits in this sample at which immunizations were due. To assess whether the diagnoses recorded provided evidence that the child was “too sick to immunize,” each diagnosis was reviewed and rated by a panel of 3 pediatricians. The interrater reliability (k) for the pediatrician ratings was 93%. In cases where the raters did not agree, the majority rating was assigned. The following diagnoses were deemed sufficient to conclude that the child was too sick to immunize: acute chickenpox, acute croup, pneumonia, possible pneumonia, meningitis, viral meningitis, possible bacteremia, sepsis, and possible sepsis. In addition, the pediatric panel agreed that a recorded temperature of 38.3°C or greater should qualify as a valid contraindication under the “too sick to immunize” clause. By combining the data on dates of visits, dates of immunizations, and contraindications, we could determine for each visit which, if any, immunizations were due and whether an MO occurred.

opportunities were more common at visits with any of the illness diagnoses recorded compared with visits with no illness diagnosis recorded. With the exception of pharyngitis, MO rates at visits with illness diagnoses were similar between the control and MO groups. The total number of visits in the illness categories exceeds 1092 because multiple diagnoses were recorded at some visits. The logistic regression model is presented in Table 4. The model explains more than one third of the variance in the occurrence of MOs (pseudo R2=0.37). After adjusting for the other covariates, neither the intervention group nor parental knowledge variables had significant predictive effect on the likelihood of an MO occurring. Provider, visit, and child/family characteristics were significantly associated with the likelihood of an MO occurring. Missed opportunities were more likely at visits (1) by children 7 to 12 months of age (vs 0 to 6 months of age); (2) by children with older mothers; (3) by children who were breast-fed as infants; (4) by children with more continuous health insurance coverage; (5) in private physicians’ offices or health maintenance organizations (vs public clinics); and (6) in physicians’ offices that did not maintain separate immunization records. Missed opportunities were less common at visits of children whose mothers smoked cigarettes or if the child had a history of cancelled appointments. Characteristics of the visit itself were strongly as-

DATA ANALYSIS In the abstracted records, a total of 1938 visits were documented in the providers’ charts for the study children as having occurred through the first 12 months of life. Of the 1938 recorded visits, 846 were eliminated from the analysis because immunizations had not been due or contraindications to vaccination had been present (eg, temperature .38.3°C or a diagnosis that indicated the child was too sick to immunize). The remaining 1092 visits were assigned the family, child, provider/chart, or visit characteristics with which they were associated. Statistically significant bivariate associations between independent variables (health visit, child/family, and provider/chart characteristics) and the dependent variable (the occurrence of an MO) were tested within each intervention group. We tested the association between the rate of MOs and continuous independent variables by means of the test of significance for correlation coefficients. For categorical independent variables, we tested the association with the occurrence of an MO by means of the x2 statistic. All bivariate associations with P,.10 were included in a visit-based logistic regression model predicting visits with MOs vs visits with no MOs (all immunizations due were given). To achieve a more parsimonious model, we used a stepwise, backward elimination approach to remove nonsignificant variables. To test the hypotheses central to this analysis, we forced the CM and immunization knowledge variables into the model regardless of their level of significance. Furthermore, child health insurance and history of breast-feeding were also included in the model to adjust for potential biases in the sample selection. We adjusted for the clustering of visits within provider by means of a Huber logistic regression model.28

sociated with the likelihood of an MO. Missed opportunities were much less likely to occur at preventive health visits than illness visits, even after adjusting for the presence of 1 or more illness diagnoses. Missed opportunities were more likely to occur when illness diagnoses of otitis media or “other” were listed. The illness diagnoses of pharyngitis, diarrhea, and cold were not independently associated with MOs and fell out of the model. COMMENT

Our study is one of the first to examine predictors of MOs taking into account detailed parent/child characteristics from a parent interview as well as provider/visit characteristics derived from chart abstractions. We found that parent-directed education and CM with an emphasis on immunizations and preventive health visits, after adjustment for other factors, did not reduce MOs during child health visits. While the intervention did not have the desired effect, our study does provide important new information on issues relating to MOs to vaccinate in young, inner-city children. At approximately 50% of the 1092 visits we analyzed, immunization opportunities were missed. This is particularly disturbing given that almost half of these children were behind in their immunizations at 1 year of age. Moreover, this inner-city area was a focus of the



Table 1. Characteristics of the Case Management and Control Groups*

Characteristic Demographic characteristics Mother’s education, % Less than HS HS graduate Some college Mean±SD maternal age, y Mother currently smokes, % Health care utilization characteristics Mean±SD No. of canceled appointments Mean±SD No. of missed opportunities Type of provider, % Private MD HMO Public provider/ED Provider charts with separate immunization record, % Mean±SD months of health insurance

Control Group (n=126)

Table 2. Association Between Missed Opportunities and Child/Family Demographic and Health Care System Utilization Characteristics*

Case Management Group† (n=129)

37 42 21 25.7±6.4 32

28 49 23 24.9±6.0 24

0.73±1.6 2.3±2.6

0.87±1.34 2.1±2.4

45 30 25 71

41 26 33 73

10.9±2.32

11.3±1.9

*HS indicates high school; MD, doctor of medicine; HMO, health maintenance organization; and ED, emergency department. †P values for all comparisons were greater than .15.

1989 to 1990 measles epidemic, which resulted in considerable media attention and additional immunization outreach efforts targeted at parents and providers in our study area. Despite this context, the MOs at child health visits remained high, consistent with MO rates in other studies.2,3,7,8 Low immunization rates have been attributed, in large part, to MOs. However, quantifying the impact of MOs on immunization completion rates is not straightforward. The relationship is complicated by the number and timing of health visits in relation to the immunization schedule. As we described elsewhere,24 while this CM did not lower the rate of MOs, it did raise immunization completion rates by 13.2 percentage points. Some of the children who received CM failed to receive more than 3 well-child visits in the first year. Nonetheless, this intervention subgroup experienced a remarkable 28 percentage point rise in immunization completion (from 30% up to date in the control group to 58% up to date in the CM group). In this same group (for whom we have physician records), the CM intervention only modestly reduced the frequency of MOs. Our interpretation of these findings is that, for children with few well-child visits, MOs may have a much greater negative effect on immunization completion than for children with ample numbers of visits to physicians. While the latter children may even experience more MOs, additional visits can overcome the effect of MOs and allow them to meet the immunization completion milestones. For children with few visits, 1 MO may be adequate to cause them to fail to meet the immunization up-to-date milestone. The most powerful predictor of an MO is the visit type; MOs were much more common at acute illness visits than at preventive health visits. These findings are consistent with previous research showing that MOs are, in

Children or Family Characteristics

Case Management Group

Categorical Variables† Source of preventive health care Private physician’s office (n=535) 54.3 Public clinic/ED/other (n=229) 32.7 HMO clinic (n=305) 44.4§ Type of health insurance Medicaid, discontinuous (n=633) 47.6 Medicaid, continuous (n=311) 45.2 Private (n=137) 50.5 Maternal education level Less than HS (n=330) 44.4 HS graduate (n=493) 49.1 At least some college (n=267) 48.6 Maternal smoking Yes (n=250) 38.8 No (n=825) 50.2§ History of child being breast-fed Yes (n=436) 54.8 No (n=651) 43.3\ Age at visit ,6 mo (n=567) 36.4‡ 6-12 mo (n=531) 59.6¶ Intervention group total 47.7

Control Group

60.3 39.4 53.7¶ 56.3 45.1 58.7‡ 52.0 53.2 55.2 46.6 55.9‡ 57.6 50.0‡ 39.7 64.6¶ 53.3

Continuous Variables# Knowledge of immunization schedule, −0.01 scale score Knowledge of contraindications to −0.13\ vaccination, scale score Months of insurance coverage 0.05

0.13\ 0.03 0.080

*ED indicates emergency department; HMO, health maintenance organization; and HS, high school. All tests of significance are for comparisons within intervention groups. †Values are percentages of visits with missed opportunities. ‡P,.10. §P,.05. \P,.01. ¶P,.001. #Values are correlation coefficients with missed opportunity rate.

large part, caused by the provider misconception that minor illnesses are contraindications to vaccination.8,9,29,30 In our model, the illness visit had a strong independent association with MOs even after accounting for the effect of the illness diagnoses listed. A possible explanation for these findings may be that most physician offices reserve well-child visits for the labor-intensive task of administering immunizations and do not give immunizations at sick visits regardless of the degree or nature of the illness. We also found that private providers and health maintenance organization providers were more likely to miss opportunities to vaccinate than were public health providers. A number of factors may explain this finding. First, private providers in inner-city areas may concentrate more on illness care than preventive care, which is more time consuming and less well reimbursed.31 Second, private and health maintenance organization providers saw their patients more frequently than the public providers in this sample of patients, which may have



Table 3. Percentage of Visits With Missed Opportunity by Visit Characteristics*

Table 4. Logistic Regression Predicting Occurrence of a Missed Opportunity Within a Health Visit

% of Visits With MO Case Management Control Group Group

Visit Characteristics Visit type Preventive health visit (n=631) Sick/other visit (n=459) Immunization record present in chart Yes (n=832) No (n=258) History of any canceled appointment(s) Yes (n=401) No (n=691) No. of providers for well-child care in first year 1 (n=766) .1 (n=301) Diagnosis at visit\ No illness diagnosis (n=373) Upper respiratory tract infection (n=255) Otitis media without fever (n=163) Pharyngitis (n=35) Acute gastroenteritis (n=51) Other (n=399)

24.6 79.8¶

27.5 84.0¶

46.2 53.3

50.0 62.3§

38.9 53.3¶

55.0 52.3

45.4 53.3‡

50.9 59.3‡

20.2\ 62.4¶ 39.9¶ 87.5\ 85.3¶ 69.6¶

26.9¶ 65.2¶ 42.5¶ 73.7‡ 83.3¶ 68.3¶

*All tests of significance are for comparisons within intervention groups. †More than 1 diagnosis could be recorded. ‡P,.10. §P,.05. \P,.01. ¶P,.001.

led them to be more relaxed about delaying immunizations to future well-child visits, thereby missing opportunities to vaccinate. The CDC Standards for Pediatric Immunization Practice, published in 1993, strongly promote screening for and administering immunizations at both sick and well visits.27 The standards also clearly delineate the true contraindications for each vaccine, which are rare.26 Despite having been published more than 2 years before our study, the standards were not widely adopted by the providers serving children in inner-city Los Angeles. The CDC immunization standards need to be disseminated more widely, especially in high-risk, poor areas where immunization levels are low. In addition, for high-risk areas, public health departments should identify physicians for education, audits of their records, and constructive feedback on their performance.32-34 A number of child and family characteristics in our model were associated with either a greater or lower likelihood of MOs occurring. Lower rates of MOs (ie, higher compliance by physicians) were associated with the negative characteristics of maternal smoking and a history of broken appointments. Higher rates of MOs (ie, lower compliance by physicians) were associated with the relatively positive characteristics of insurance continuity and history of breast-feeding. We hypothesize that these family characteristics are perceived by providers as indicators of risk and that providers adjust their immunization decision making in response to these perceptions. That is, provider perceptions of increased risk may prompt

Parameter Home visit/case management group (vs control group) (n=1047, pseudo R 2=0.37) Demographic characteristics Age of child 6-12 mo (vs 0-6 mo) Mother smokes (yes vs no) Mother’s age (in years) History of being breast-fed (yes) Knowledge of immunization schedule scale score Knowledge of contraindications to vaccination scale score Health care utilization characteristics Source of preventive health care Private MD office Health maintenance organization office (vs public/emergency department) Months of health insurance coverage History of canceled appointment(s) Chart/visit characteristics Presence of immunization record Preventive health visit (vs sick visit) Specific diagnoses at visit (vs no illness diagnosis recorded) Otitis media without fever Other illness diagnosis

Odds Ratio

95% Confidence Interval

1.0

0.73-1.4

2.73 0.50 1.05* 1.5 0.92*

1.49-5.0 0.29-0.85 1.01-1.10 1.06-2.12 0.72-1.18

0.98*

0.80-1.20

3.5 1.8

1.9-6.2 1.1-2.9

1.13 0.62

1.03-1.24 0.42-0.94

0.49 0.06

0.27-0.90 0.04-0.11

3.1 2.5

1.93-5.0 1.58-3.83

*Odds ratios for continuous variables can be interpreted as the odds of being up-to-date associated with a 1-point increase in the continuous variable. The odds associated with a multiunit increase in a variable or scale is calculated by taking the odds ratio to the nth power, with n being the number of unit changes in the variable. For example, the odds of an MO occurring at a visit for a child with a mother who is 30 years old vs a child with a mother who is 20 years old is 1.05(30−20)=1.05(10)=1.63.

more aggressive ordering of immunizations, hence the lower MO rate in children of smoking mothers. For children and families perceived to be at low risk, the provider may be more willing to defer immunizations, confident that the child will return for immunizations at a later well-child visit. The limitations of this study should be appreciated. First, we were able to obtain and abstract the complete records from all reported providers for 255 children, which is 69.7% of the 366 successfully followed up in the randomized controlled trial. However, it represents only 61% of the 419 children originally enrolled in the trial. We compared important demographic characteristics between both the 419 originally enrolled and 366 followed up in the randomized controlled trial with the 255 families/children used in this analysis; they were comparable on all demographic and health utilization characteristics except for health insurance, knowledge of contraindications scale, and history of being breast-fed. These factors are controlled for in the multivariate analysis. However, the possibility that attrition of 39% of the sample from our analyses may have biased our results cannot be completely ruled out. Second, while we were largely successful at abstracting records from all providers reported by the parents, the potential for bias resulting from inadequate provider record keeping remains. Because we were equally successful at



provider record abstraction across the 2 intervention groups, we doubt that this bias affected our results. Finally, this study was conducted on a sample of inner-city, predominantly poor, African American families. The results, therefore, may not be generalizable to other populations of children. In summary, we found that family CM was not effective at reducing MOs at child health visits. In this study, MOs were found to be associated with a number of provider-related factors (ie, presence of false contraindications, record-keeping quality) that are under the providers’ control and should be addressed. Family- and childrelated factors (eg, maternal smoking, cancelled appointments), however, do influence the provision of immunizations, indicating that immunization provision, like most medical decisions, is an outcome of a complex interaction between the provider, child, and family. Accepted for publication October 30, 1997. This study was supported by contract 200-91-0942 from the Centers for Disease Control and Prevention, Atlanta, Ga. We thank the Los Angeles Research and Education Institute–Women-Infant-Children staff and the study case managers for their enthusiastic and dedicated efforts to improve the care of children and families in inner-city Los Angeles. Reprints: David Wood, MD, MPH, Shriners Hospitals, 2900 Rocky Point Dr, Tampa, FL 33607-1460 (e-mail: [email protected]). REFERENCES 1. Centers for Disease Control and Prevention. Vaccination coverage of 2- year-old children—United States, 1995. MMWR Morb Mortal Wkly Rep. 1996;43:705709. 2. Grabowsky M, Orenstein WA. The critical role of provider practices in undervaccination. Pediatrics. 1996;97:729-732. 3. Wood DL, Pereyra M, Halfon N, et al. Vaccination levels in public health centers: missed opportunities and other contributing factors. Am J Public Health. 1995; 85:850-853. 4. Mustin HD, Holt VL, Connell FA. Adequacy of well-child care and immunization in US infants born in 1988. JAMA. 1994;272:1111-1115. 5. Williams IT, Milton JD, Farrell JB, Graham NMH. Interaction of socioeconomic status and provider practices as predictors of immunization coverage in Virginia children. Pediatrics. 1995;96:439-446. 6. Centers for Disease Control and Prevention. Impact of missed opportunities to vaccinate preschool-aged children on vaccination coverage levels—selected US sites, 1991-1992. MMWR Morb Mortal Wkly Rep. 1994;43:709-711, 717-718. 7. Guyer B, Hugart N, Hold E, et al. Immunization coverage and its relationship to preventive health care visits among inner-city children in Baltimore. Pediatrics. 1994;94:53-58. 8. Szilagyi PG, Rodewald LE, Humiston SG, et al. Missed opportunities for childhood vaccinations in office practices and the effect on vaccination status. Pediatrics. 1993;91:1-7. 9. Wood DL, Halfon N, Hamlin J, Grabowsky M. Knowledge of child immunization schedule and contraindications to vaccinate by private and public providers in Los Angeles. J Pediatr Infect Dis. 1996;15:140-145. 10. Hughart N, Guyer B, Stanton B, et al. Do provider practices conform to the new pediatric immunization standards? Arch Pediatr Adolesc Med. 1994;148:930935.

11. Lieu TA, Black SB, Sorel ME, Ray P, Shinefield HR. Would better adherence to guidelines improve childhood immunization rates? Pediatrics. 1996;98:1062-1068. 12. Davis DA, Thomson MA, Oxman AD, Haynes RB. Changing physician performance: a systematic review of the effect of continuing medical education strategies. JAMA. 1995;274:700-705. 13. McPhee SJ, Bird JA, Fordham D, Rodnick JE, Osborn EH. Promoting cancer prevention activities by primary care physicians. JAMA. 1991;266:538-544. 14. Rosser WW, McDowell I, Newell C. Use of reminders for preventive procedures in family medicine. Can Med Assoc J. 1991;145:807-814. 15. Bates AS, Fitzgerald JF, Dittus RS, Wolinsky FD. Risk factors for underimmunization in poor urban infants. JAMA. 1994;272:1105-1110. 16. Szilagyi PG, Rodewald LE, Savageau J, Yoos L, Doane C. Improving influenza vaccination rates in children with asthma: a test of a computerized reminder system and an analysis of factors predicting vaccination compliance. Pediatrics. 1992; 90:871-875. 17. Bennett P, Smith C. Parents’ attitudinal and social influences on childhood vaccination. Health Educ Res. 1992;7:341-348. 18. Ball TM, Serwint JR. Missed opportunities for vaccination and the delivery of preventive care. Arch Pediatr Adolesc Med. 1996;150:858-861. 19. Wood DL, Halfon N, Sherbourne C, et al. Increasing Immunizations for African American and Latino Preschool Children in the Inner City of Los Angeles. Atlanta, Ga: Centers for Disease Control and Prevention; 1993. Contract 200-910942. 20. Wood DL, Sherbourne CD, Halfon N, et al. Factors related to immunization status among Latino and African American inner city preschoolers. Pediatrics. 1995; 96:295-301. 21. Olds DL, Kitzman H. Review of research on home visiting for pregnant women and parents of young children. Future Child. 1993;3:53-92. 22. Selby-Harrington M, Sorenson JR, Quade D, Stearns SC, Tesh AS, Donat PLN. Increasing Medicaid child health screenings: the effectiveness of mailed pamphlets, phone calls and home visits. Am J Public Health. 1995;85:1412-1417. 23. Colombo TJ, Freeborn KD, Mullooly JP, Burnham VR. The effect of outreach workers’ educational efforts on disadvantaged preschool children’s use of preventive services. Am J Public Health. 1979;69:465-468. 24. Wood DL, Halfon N, Donald-Sherbourne C, et al. A randomized trial of case management to raise immunization rates among African American children in innercity Los Angeles. JAMA. 1998;279:29-34. 25. Wood DL, Halfon N, Sherbourne C, et al. A Randomized Trial of Case Management to Raise Immunization Rates Among African American Children in InnerCity Los Angeles. Atlanta, Ga: Centers for Disease Control and Prevention; 1996. Contract 200-91-0942. 26. Centers for Disease Control and Prevention. General recommendations on immunization: recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Morb Mortal Wkly Rep. 1994;43(RR-1):1-38. 27. Centers for Disease Control and Prevention. Standards for pediatric immunization practices. JAMA. 1993;269:1817-1822. 28. STATA 4.0. College Station, Tex: Stata Corp; 1994/1995. 29. Hutchins SS, Jansen HAFM, Robertson SE: Missed Opportunities for Immunization: The Magnitude, Reasons, Strategies and Recommendations. Geneva, Switzerland: Expanded Programme on Immunization, World Health Organization; October 1991. 30. Farizo KM, Stehr-Green TA, Markowitz LE, Patriarca PA. Vaccination levels and missed opportunities for measles vaccination: a record audit in a public pediatric clinic. Pediatrics. 1992;89:589-592. 31. Fairbrother G, Friedman S, DuMont KA, Lobach KS. Markers for primary care: missed opportunities to immunize and screen for lead and tuberculosis by private physicians serving large numbers of inner-city Medicaid-eligible children. Pediatrics. 1996;97:785-790. 32. Batalden PB, Stoltz PK. A framework for the continual improvement of health care. J Qual Improv. 1993;19:424-446. 33. Kibbe DC, Kaluzny AD, McLaughlin CP. Integrating guidelines with continuous quality improvement: doing the right thing the right way to achieve the right goals. J Qual Improv. 1994;20:181-191. 34. Kanouse DE, Kallich J, Kahan JP. Dissemination of practice-relevant information to health care providers and payers. Presented at the National Agenda Setting Conference in Outcomes and Effectiveness Research; April 15, 1991; Washington, DC.

