Skip

Statistical Process Control:Possible Uses to Monitor and Evaluate Patient-Centered Medical Home Models

March 2013
AHRQ Publication No: 13-0031-EF
Prepared For:
Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services, 540 Gaither Road, Rockville, MD 20850, http://www.ahrq.gov/

Prepared by: Jill A. Marsteller, Ph.D. (John Hopkins University School of Public Health, Baltimore, MD) (jmarstel@jhsph.edu), Mary Margaret Huizinga, M.D. (John Hopkins University School of Medicine, Baltimore, MD and LifePoint Hospitals, Inc., Brentwood TN), and Lisa A. Cooper, M.D. (John Hopkins University School of Medicine, Baltimore, MD)

Abstract

This brief focuses on using statistical process control in studies of patient-centered medical home (PCMH) models. It is part of a series commissioned by the Agency for Healthcare Research and Quality (AHRQ) and developed by Mathematica Policy Research under contract, with input from other nationally recognized thought leaders in research methods and PCMH models. The series is designed to expand the toolbox of methods used to evaluate and refine PCMH models. The PCMH is a primary care approach that aims to improve quality, cost, and patient and provider experience. PCMH models emphasize patient-centered, comprehensive, coordinated, accessible care, and a systematic focus on quality and safety.

Suggested Citation

Marsteller JA, Huizinga MM, Cooper LA. Statistical Process Control: Possible Uses to Monitor and Evaluate Patient-Centered Medical Home Models. Rockville, MD: Agency for Healthcare Research and Quality. February 2013. AHRQ Publication No. 13-0031-EF.

This brief and companion briefs in this series are available for download from pcmh.ahrq.gov.

I. Statistical Process Control

One issue in evaluating PCMH models is that reporting of changes in process and outcome measures is typically infrequent and often lags significantly after the start of the intervention. This is due to several factors: the burden of frequent data collection, the fact that outcome metrics often require an extended time to show the impact of the intervention, a lack of good short-term process metrics, and a lack of knowledge regarding tools to differentiate true change from random noise.

Statistical process control (SPC) is a set of statistical methods based on the theory of variation that can be used to make sense of any process or outcome measured over time, usually with the intention of detecting improvement or maintaining a high level of performance. SPC combines rigorous time series analysis with graphical presentation of data, and can provide early insights into the data in a manner understandable to a wide range of audiences (Benneyan, Lloyd, and Plsek, 2003). Using frequently measured data, SPC can be used to detect, early on, whether any change has taken place since the start of the intervention, long before results from a larger, summative evaluation are available. Such information can be used for purposes ranging from forming hypotheses about changes in outcomes to adapting elements of the intervention to increase the likelihood of success. In the case of PCMH evaluations, findings from SPC may be included in ongoing status reports to stakeholders including frontline clinicians, staff members, office managers, evaluators, funders, and policymakers (Thor, Lundberg, Ask, et al., 2007; Benneyan, Lloyd and Plsek, 2003; Carey, 2003).

There are several SPC tools that can be applied to the evaluation of PCMH and other health services and programs. This brief focuses on the control chart. 1 Measurement of the same variable over time often yields slightly different values at each point in time. Walter Shewhart, a physicist at the Bell Telephone Laboratories, introduced the control chart in the 1920s to distinguish between this inherent or normal variability, known as common cause variation, and variation due to a special cause (Shewhart, 1931; Moresteam.com, 2012). Special causes are changes in the pattern of data that can be assigned to a specific cause. These special causes can be bad or good, intentional or unintentional. An example of a positive intentional special cause variation would be adding a care manager to bring in patients for tests, which could increase the percentage of patients with diabetes who have received appropriate blood tests. If the care manager’s cell phone breaks, his or her contact rate of eligible patients might decline, an example of a negative unintentional special cause variation. One can observe process or outcome variables over time in the absence of an intervention to establish the normal variation that exists in a stable period. When the intervention is introduced, the data can be monitored for evidence of special cause variation, which would indicate meaningful change.

The key steps for constructing and using a control chart are:

  • Identify the process(es) of interest, such as use of specialists by a primary care practice’s patients.
  • Identify measurable attributes of the process, such as referrals to endocrinologists for patients with diabetes who do not have well-controlled blood glucose levels. Measures appropriate for control chart analysis are those that change frequently over time. They may be based on continuous, count, proportion, or ratio data. Select the type of control chart that is recommended for the type of data under study (see the Resources section of this brief for further information). For example, the “p-chart” (p is for proportion or percentage) is used to study proportion data. 2 Collect the measure at regular intervals over time (20 to 25 points for adequate statistical power [Carey, 2003]) and plot the measurements on a chart that has time on the x-axis and levels of the measure own the y-axis.
  • Use SPC computer software to generate the correct chart type, compute the mean value over the time period of interest, and display a line on the chart showing this value.
  • Characterize natural variation in the measure by using the software to calculate the upper and lower control limits on either side of the mean based on appropriate formulae for the chart type, and display lines on the chart for these values.
  • Track the variable going forward and observe patterns in the new values.
  • If variation in the measure meets certain criteria indicating special cause, then:
    • Identify the assignable cause
    • Remove the assignable cause if negative or celebrate it if it is an intentional change to improve care
    • Continue to track the variable and look for Special Cause Variation

Figure 1. Example of p-Charts

Figure 1. Stages of Formative Evaluation

As an example, imagine that one goal of a PCMH program is to improve blood pressure (BP) control and to eradicate disparities by race in BP control. 3 First, data on patients with hypertension who are and are not in control would be retrospectively pulled from their electronic medical records, claims, or a paper registry of patients with hypertension. To establish normal variation, the evaluator would plot the proportion of patients with hypertension in control each week for 24 weeks before the start of the intervention using a p-chart. In addition, the p-chart would include lines indicating the mean and the upper and lower control limits (defined as three standard deviations [SDs] above and below the mean) over the 24 weeks (Carey, 2003; Lee and McGreevy, 2002). Control charts could be used for patients in all participating PCMH practices, in an individual practice, or of individual clinicians. The upper portion of Figure 1 is a p-chart with hypothetical aggregate data on BP control at the practice level showing a normal variation pattern (no special causes).

After the practice begins an intervention to improve BP control, the p-chart would be examined on an ongoing basis looking for special cause variation, defined by convention as one or more of the following:

  • One value outside the control limits.
  • Two of three consecutive values above or below the mean and more than two SDs away from the mean.
  • Eight or more values falling above or below the mean.
  • Six or more values in a row steadily increasing or decreasing (that is, showing a trend) (Carey, 2003).

The lower portion of the figure shows hypothetical examples of special cause variation. These low values would be examples of (probably) unintentional negative influences on the variable that were only temporary.

If the interventions are successful, the p-chart would show special cause variation such as an increase in the average number of patients with controlled BP. If the improvement is sustained over time, we would expect to see a new, higher average level of success (a new mean line) and stability or only normal variation around this new mean line. The control limits may also become narrower, indicating reduced random variation in the proportion with controlled BP due to more standardized BP intervention. Reducing variation in delivery of services is a core tenet of providing highly reliable care.

SPC methods can be supplemented by multivariate time series regression analysis to test change over time or to associate process measures with outcomes.

back to top

II. Uses of the Method

In addition to monitoring changes in outcome measures such as the proportion of patients with controlled BP described above, the control chart has several other uses, including to:

Monitor process measures. PCMH evaluators can use the control chart to examine successful implementation and sustainment of different aspects of the intervention. In the example above, a control chart could also monitor the number of patients with hypertension contacted by the care manager, the number of appointments missed by patients, or the number of patients with up-to-date care plans.

Identify early signs of correlation between processes and outcomes. Trends in outcomes can be superimposed on a chart showing trends in process measures to create a visual display of correlation (this is most useful when correlations are supported by statistical association). Such a display can be useful in engaging clinicians and staff in the intervention, a key element of successful change management. Further, this example extends easily to the kind of quality improvement process and outcomes indicators commonly measured in PCMH evaluations (such as chronic illness management implementation and outcome metrics).

Identify differences across groups. Change over time for more than one group can also be graphed on the same chart and tested for differences. Examples of this include monitoring and comparing BP control for blacks and whites, for multiple practices, or for intervention and comparison groups.

Aid self-management interventions. Providers have used SPC to monitor change in individual patients (Staker, 2003) as well as for self-monitoring by patients (Thor, Lundberg, Ask, et al., 2007), making SPC a powerful tool for self-management interventions.

A review of the literature between 1990 and 2004 showed SPC had been used to analyze 97 different variables measuring health care improvement (Thor, Lundberg, Ask, et al., 2007). Seven articles addressed primary care topics. Measures they examined that are relevant to the PCMH include the following:

  • Clinical outcomes: Hemoglobin A1C (A1C) level in patients with diabetes; average A1C levels in patients with diabetes cared for by each clinician; and percentage of all acute bronchitis visits during which an antibiotic was prescribed per month.
  • Process of care measures: Number of patients with diabetes seen in office visits with A1C measurements per month; number of patients seen in office visits at a department of family medicine with a recorded diagnosis of tobacco use per month; and number of referrals per month for patients with diabetes from primary care to an endocrinologist.
  • Patient experience: Physician care scale score (an average of patient survey responses to 10 questions).
  • Financial measures:Net patient revenue per relative value unit; provider cost as a percentage of net revenue; and non-provider cost as a percentage of net revenue.

back to top

III. Advantages

Thor and colleagues (2007) noted that 22 articles considered the utility of SPC approaches themselves, in addition to using the method, and found that SPC can be “a powerful and versatile tool for managing changes in healthcare through quality improvement” (p. 389). They grouped the benefits cited in the articles into three major categories: (1) SPC facilitated and documented health care process improvement, especially in helping determine the effects of interventions intended to improve care; (2) SPC was a useful tool for health care management, most commonly in deciding what variation in processes was significant and what was not, its ease of use, and cost-effectiveness and timeliness of feedback; and (3) SPC helped stakeholders, such as enabling patients to participate more in their own care decisions, improving communication among managers and clinicians, and improving managerial decisionmaking.

In addition, SPC methods:

Provide early information. SPC methods may permit early reporting on process performance, and may be used to assess the need for a summative evaluation before such an evaluation is undertaken. This could be especially informative for processes that are known to contribute to positive outcomes, or where processes are valued in and of themselves.

Avoid lack of power issues from small sample sizes. In many efforts to bring about system change, such as transformation into PCMHs, some outcomes are best measured at the system level (such as within a practice), but only a small number of practices may participate. This sample size issue makes other kinds of evaluation challenging because of a lack of statistical power (Peikes, Dale, Lundquist, et al., 2011). SPC uses the power of frequent measurement to provide information on whether real change is occurring.

Measure rare events. SPC can also be used to follow occurrence of adverse events that tend to be rare, such as failure to address a test result. Using a special type of control chart (a g-chart, where g stands for geometric distribution), evaluators can track the days between events to provide a visual representation of the current status and progress for avoiding rare events (Wall, Ely, Elasy, et al., 2005).

Simplify interpretation of data to avoid mistaken conclusions. Visual representation of change using SPC tools is particularly useful for some stakeholders who may not prioritize statistical significance when drawing conclusions about an intervention. SPC provides a graphical alternative to other methods that aids assessment of whether the pre- and post-intervention means of a variable represent stable measurements. If either measurement is not stable, evaluators and stakeholders can be alerted that the difference in the two means may not represent real change (Carey, 2003). Additionally, because statistical significance of a change may not always translate into clinical significance, visual representation of the magnitude of the change using SPC tools can help evaluators and stakeholders recognize this distinction and avoid making mistaken conclusions.

back to top

IV. Limitations

Like all methods, SPC methods have some limitations that should be considered when applying them to PCMH evaluation. In particular, they:

Require “smart application” and expert consultation.. SPC methods are best applied in consultation with an SPC expert; while they seem simple, proper use requires sophistication. As Thor, Lundberg, Ask, et al. (2007) indicate, “SPC cannot solve all problems and must be applied wisely. There are many opportunities to ‘go wrong’…. Its power hinges on correct and smart application, which is not necessarily a trivial task.” The authors note that observers should not equate statistical control in a process or outcome to clinical control or desired performance, and that having documented change does not make it obvious what exactly brought about the change. Finally, it is important to recognize the need to risk adjust for patient characteristics and comorbidities or choose homogenous subgroups when aggregating patient-level data in order to best understand the practical implications of the control charts (Thor, Lundberg, Ask, et al., 2007).

Rely on many data points. Control charts require a large number of units on which data are collected over many time periods in order to be useful. For example, p-charts, such as the example shown above, are unreliable if there are fewer than 20 time points and fewer than 25 units in the denominator at each time point. For example, the method would need at least 25 patients with hypertension in each time period to calculate the proportion of patients with controlled BP. Also, control charts are not well suited to measuring change in infrequently collected data such as annual surveys.

Involve some degree of autocorrelation. 4 Thor, Lundberg, Ask, et al. (2007) also highlight the problem of autocorrelation in repeated measurements on the same subject. This is a larger problem in more frequent measurement, such as hourly. One method to reduce autocorrelation is to use measurements that are three to four periods apart, rather than those that are only one period apart. While this approach may require more data and a longer observation period, it could be used as a sensitivity test when additional data are not available. Wheeler (1995), however, argues that a small amount of autocorrelation will have little impact on control limits and can be ignored.

In addition to its many uses for clinical quality improvement and practice management, statistical process control holds promise as a statistically sound, easily interpretable approach for evaluations of PCMH interventions. It has the potential to accelerate learning about possible effects on processes and outcomes by providing early information on change, before results from a full summative evaluation are available. Because control charts rely on graphical representation of data, they can be readily understood by stakeholders of many different backgrounds, which may open up avenues to inform policy at the patient, clinician, practice, health care system, local, State, and national levels.

back to top

VI. References

  • Benneyan JC, Lloyd RC, Plsek PE. Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care 2003;12:458–64.
  • Carey R. Improving healthcare with control charts: basic and advanced SPC methods and case studies. Milwaukee: ASQ Quality Press; 2003.
  • Lee K, McGreevey C. Using control charts to assess performance measurement data. Jt Comm J Qual Improv. 2002;28(2):90-101.
  • Moresteam.com. Statistical Process Control (SPC). http://www.moresteam.com/toolbox/StatisticalProcesss- control-spc.cfm.
  • National Institutes of Health. Current Centers for Population Health and Health Disparities, Johns Hopkins University Center to Eliminate Cardiovascular Health Disparities. http://cancercontrol. cancer.gov/populationhealthcenters/cphhd/centers-jhu.html,
  • Peikes D, Dale S, Lundquist E, et al. Building the evidence base for the medical home: what sample and sample size do studies need? Rockville (MD): Agency for Healthcare Research and Quality; 2011. AHRQ Publication No. 11-0100-EF.
  • Shewhart WA. Economic control of quality of manufactured product. New York: D. Van Nostrand Company; 1931.
  • Staker L. The use of run and control charts in the improvement of clinical practice. In: Carey R. Improving healthcare with control charts: basic and advanced SPC methods and case studies. Milwaukee: ASQ Quality Press; 2003.
  • Thor J, Lundberg J, Ask J, et al. Application of statistical process control in healthcare improvement: systematic review. Qual Saf Health Care 2007;16:387–99.
  • Wall RJ, Ely EW, Elasy TA, et al. Using real time process measurements to reduce catheter related bloodstream infections in the intensive care unit. Qual Saf Health Care 2005;14:295–302.
  • Wheeler DJ. Advanced topics in statistical process control. Knoxville (TN): SPC Press; 1995.
back to top

VII. Resources

Applying Statistical Process Control in Health Care Research

  • Carey R. Improving healthcare with control charts: basic and advanced SPC methods and case studies. Milwaukee: ASQ Quality Press; 2003.
  • Thor J, Lundberg J, Ask J, et al. Application of statistical process control in healthcare improvement: systematic review. Qual Saf Health Care 2007;16:387–99.

How to Select and Generate Control and Run Charts

back to top

Footnotes

  1. 1. Other tools include check sheets, histograms, scatterplots, and pareto diagrams.
  2. 2. It should be noted that some types of control charts are more powerful than others, so the evaluator should seek to collect data in a form that permits the use of the most powerful chart (Carey, 2003).
  3. 3. We are currently conducting such an investigation in one of the Centers for Population Health and Health Disparities sponsored by the National Heart, Lung and Blood Institute, though not all of the sites are PCMHs (NIH, 2012).
  4. 4. Autocorrelation is the presence of correlation among consecutive measurements over time on the same subject. This is a problem because the data points are not independent. This tends to increase the number of Type-I errors (more frequent detection of special cause when there is none).