Interpretation of Results
The State Snapshots are produced annually by the Agency for Healthcare Research and Quality. They provide State-specific health care quality information, including strengths, weaknesses, and opportunities for improvement. The goal is to help State officials and their public- and private-sector partners better understand health care quality and disparities in their State. This section presents additional information for users interested in the basis of the data and performance measures used in the State Snapshots. It also discusses issues to consider in evaluating the utility and appropriateness of health care data.
Health care performance measures and public health data are valuable decisionmaking aids for State agency staff. Yet interpreting performance information in the right context can sometimes be challenging when creating a performance improvement plan.
Ensuring that performance information is credible and meaningful is essential for State agencies to succeed in engaging all the necessary stakeholders—physicians and other health care providers, hospitals, nursing homes, health plans, and the public health system—to improve quality. Understanding the detail behind the statistics can help States consider systematic ways to examine State data when planning and developing quality improvement initiatives.
This section presents additional information on issues to consider when evaluating data for a quality improvement program, including:
The information in this section can be applied to the State Snapshots, as well as to other State data to identify areas where supplemental information is needed.Top of Page
Examination of Data Sources
A successful quality improvement initiative should be built on robust data systems, such as those used to create the State Snapshots. Before embarking on any effort to improve health care, staff will need to understand the strengths and limitations of the data, as well as the need to supplement these data with other available information.
Ensuring the credibility of data is critical to any quality improvement program, particularly when multiple stakeholders are involved. Before State policymakers can implement changes based on problems identified in the data, stakeholders usually have questions about the data that may include the following:
Agency staff will need to be able to respond to these and similar questions in order to gain support from other stakeholders in the quality improvement process.
Data Source Inventory Examples
Described below are several sources of State-level data available to States. Additional sources States may consider include Medicaid health provider reimbursement claims, State employee health benefits claims, and data from regional health information organizations.
Agency for Healthcare Research and Quality – Healthcare Cost and Utilization Project (HCUP). The HCUP family of health care databases and related software tools and products includes the largest collection of longitudinal hospital care data in the United States, with all-payer, encounter-level information starting in 1988. These databases enable research on a broad range of issues, including cost and quality of health services, medical practice patterns, access to health care programs, and outcomes at national, State, and local market levels. For more information, go to http://www.hcup-us.ahrq.gov/.
Centers for Disease Control and Prevention (CDC) – Behavioral Risk Factor Surveillance System (BRFSS). The world's largest, ongoing telephone health survey system, the Behavioral Risk Factor Surveillance System has been tracking health conditions and risk behaviors in the United States yearly since 1984. BRFSS provides State-specific information about issues including asthma, diabetes, health care access, alcohol use, hypertension, obesity, cancer screening, nutrition and physical activity, tobacco use, and more. For more information, go to http://www.cdc.gov/brfss/.
CDC National Center for Health Statistics (NCHS) – State and Local Area Integrated Telephone Survey (SLAITS). SLAITS collects health care data at State and local levels to supplement national data collection efforts. One SLAITS survey is the National Asthma Survey, which examines the various predictors that relate to better control of asthma and helps to characterize the content of care and health care experiences of people with asthma. More information on SLAITS is available at http://www.cdc.gov/nchs/slaits.htm. For a list of other NCHS surveys on additional clinical topics and health care delivery systems, go to http://www.cdc.gov/nchs/express.htm.
National Vital Statistics System (NVSS) and State vital statistics registries. NCHS collects and disseminates the Nation's official vital statistics through contracts between NCHS and vital registration systems operated in 50 States, 2 cities (New York City and Washington, DC), and 5 trust territories. These jurisdictions are legally responsible for maintaining registries of vital events, including births, deaths, marriages, divorces, and fetal deaths. For more information on NVSS, go to http://www.cdc.gov/nchs/nvss.htm. For State vital statistics, go to http://www.cdc.gov/nchs/about/major/natality/sites.htm.
Bureau of the Census – population and household data. The U.S. Bureau of the Census has tracked the Nation's population since the first decennial census in 1790. Data can be accessed across a variety of demographic, socioeconomic, and other dimensions. These include age, insurance, disability, income, occupation, voting and registration, school enrollment, migration, and language use. For more information, go to http://www.census.gov/population/www/index.html.
Health Resources and Services Administration – Area Resource File. The Area Resource File facilitates analysis of health care access at the county level. The database contains more than 7,000 variables for each U.S. county. Records include geographic codes and classifications, health professions supply and detailed demographics, health facility numbers and types, hospital utilization, population characteristics, economic data, and health professions training resources. For more information, go to http://bhpr.hrsa.gov/healthworkforce/data/arf.htm.
State disease registries. Many States have created disease registries to document the names of individuals with a specific illness and to track disease outcomes in order to provide information on incidence and prevalence of various diseases. Many registries can be accessed through the State's health department. CDC provides a list of information networks and other sources, including links to State health departments. For more information, go to http://www.cdc.gov/other.htm.
State hospital discharge data. Several States maintain a statewide uniform system of records on hospital discharges. These records can assist hospitals and health care organizations and agencies with financial planning and monitoring of patient services and costs. They can also be used for monitoring disease and injury rates through ongoing collection and interpretation of data and for studies of specific diseases. (State discharge data are also included in the HCUP database.)
National Committee for Quality Assurance – Health Plan Employer Data and Information Set (HEDIS®). HEDIS® measures performance on 71 specifically defined measures across 8 domains of health care. Used by more than 90 percent of U.S. health plans to measure performance, HEDIS® facilitates plan-to-plan comparisons. For more information, go to http://web.ncqa.org/tabid/59/Default.aspx.
Kaiser Family Foundation – statehealthfacts.org. A project of the Henry J. Kaiser Family Foundation, statehealthfacts.org is designed to provide free, up-to-date, and easy-to-use State health data on more than 500 topics. For more information, go to http://statehealthfacts.org/.Top of Page
Types of Measures
Several types of measures—process, outcome, and context—are reflected in the State Snapshots. Taken together, the measures provide pieces of a puzzle depicting the health of the population, the system's readiness to care for patients, the quality of care given by providers, and the context or environment in which the system operates.
Using various types of measures provides a broader and more complete picture of performance. Choice of measures is important to any quality improvement initiative. Of prime concern is choosing measures that are valid and important indicators of performance for the health care condition or problem being addressed.
Process measures examine whether a certain process was carried out. In health care, a variety of functions are performed because the scientific evidence shows that these processes result in better outcomes. Many process measures have been developed to evaluate the use of evidence-based practices. However, many processes contribute to achieving a certain result (outcome), so it is often difficult to attribute a good outcome to a single process.
For example, an immunization rate is a process measure indicating the proportion of individuals who were immunized. Immunization rate (process) correlates closely with the absence of vaccine-preventable disease (outcome).
Other process measures are important but not as directly linked to outcomes. For example, process measures that are, in large part, under the control of the physician for diabetes care include HbA1c measures, rates of eye exams, and rates of foot exams. The physician can carry out the exams, but other factors, such as access to care, contribute to how much improvement can be made in the results. Many processes, including patient adherence to the appropriate regimen, must occur before reductions in diabetes complications (outcome) will occur. Therefore, process measures shown to lead to better outcomes are more indirect for diabetes care than for immunizations.
Examples of process measures in the State Snapshots include providing pneumonia vaccinations to the elderly and prescribing aspirin to heart attack patients when they are discharged from the hospital.
Outcome measures examine an end point of care. Outcomes may include death, cost of care, satisfaction, or absence of disease.
For chronic conditions, outcomes are often measured as absence of complications of the chronic disease (such as no heart attack for people with heart disease) or an end result (such as the survival rate for dialysis patients).
When the outcome desired is something that does not happen, it can be difficult to measure. The length of time between some outcomes and events in the system that might influence them is another measurement challenge. In these cases, longer periods of measurement are necessary, because, for example, a heart attack could occur at any time in a person's life. Also, outcome measures may not provide as much focused information about what went wrong in the system and possibly contributed to a bad outcome. For example, it is difficult to know whether a diabetes-related death resulted from a failure to prevent, a failure to diagnose, or a failure of the patient and clinical staff to manage the condition.
Outcome measures in the State Snapshots include infant mortality rates, cancer deaths, and survey results related to getting appointments for care. Trends in these measures generally indicate that the system may be improving or deteriorating, but the reasons are usually more difficult to discern.
Summary Versus Individual Measures
Users of the State Snapshots will often observe multiple measures related to a single topic. A State may perform well on one measure and poorly on another similar measure.
In home health care, for example, measures are included that relate to pressure sores, weight loss, bathing, mental clarity, and other aspects of care and patient status. Many States score well on some of the measures in this set but poorly on others. In many cases it is useful to consider related measures as a group, rather than individually. Examining “summary” measures—a single score that results from combining multiple measures—may be more informative if there is interest in a broader view of a particular dimension of quality or clinical area that is represented by a set of related measures.
Contextual factors provide information on the environments in which State health care systems operate. These factors may have a direct or indirect influence on the State's measures of health care quality. Examples of contextual factors include the characteristics of State populations, availability of health care resources, and ways systems are organized and operated. More specifically, contextual factors can include availability of insurance or access to services, measures of staffing, and aging or disease prevalence of the population.
Access to services is a critical consideration for a State seeking to improve the health care status of its residents. For example, if a high percentage of the State's population does not have health insurance, a high percentage of the State's population might not use preventive services. Other contextual factors include the prevalence of a particular condition or risk factor, such as the portion of the population at risk of heart disease. The State should also consider measures that assess capacities of the health care system to handle increased volume resulting from a quality improvement effort.
The 2007 State Snapshots include “dials” that represent 13 separate contextual factors. The dials can be accessed from the left navigation bar (see “Focus on Contextual Factors”) or from the performance meter pages. The same set of factors is linked to each meter, although different factors may be more or less relevant to the interpretation of each specific performance meter. The contextual factors include seven demographic factors, three health status factors, and three health care resource factors.Top of Page
Attributes of Data
To use data to guide quality improvement projects, it is important to understand certain underlying aspects of the data. Several factors must be considered to ensure that differences observed in the data reflect actual performance differences, rather than differences in capturing data or in the underlying health of the population under study.
In today's health information technology environment, performance measurement is limited by the data collection systems and approaches available. The State Snapshots offer a wide variety of measures, collected from a variety of data sources. Each, however, reflects a population of patients, such as hospitalized patients, older adults covered by Medicare, or a sample interviewed by the source collecting the data. The people being measured make up the “denominator” population.
Knowing who is in the population is vital in order to understand comparisons. For example, when comparing a health plan's breast cancer screening rate with a public health department's screening rate, it is important to acknowledge the differences in their populations. The health plan covers a known, insured group of women, while the health department may serve as a public safety net that is accountable for an entire community, many of whom may lack insurance.
For a quality improvement strategy to achieve its goal, policymakers must determine whether the population for which they have data is the population they want to reach.
Adjustments to Data
Sometimes data need to be assessed to determine whether factors such as age or gender affect the results. For example, the prevalence of disease increases as the population ages, as does the risk of falls and nursing home admissions. It is important for a State to evaluate measures to determine whether they are reported according to age, gender, and disease prevalence, if needed.
Another example involves hospitalization rates, which are affected by the prevalence of disease in a community. When there are more people with asthma in a community, that community is likely to experience more hospitalizations for asthma. However, many hospitalization rates are reported relative to the total population in a community because the rate of asthma among the population may not be known. In these cases, it is important to understand the limits of the measures.
Cancer deaths will follow a similar pattern. If a State has a high rate of colorectal cancer deaths, for example, analysts would need to understand how the measure is calculated. The calculations would provide insight into whether the problem is an unusually high rate of cancer or greater cancer mortality (possibly due to gaps in treatment quality).
In addition, it is important to understand what adjustments may be needed to produce valid outcome measures. Although outcome measures are the ultimate measure of quality, they must account for differences in patient characteristics. Statistical adjustments should take into account patient-specific variables that are beyond the control of the clinician, hospital, or nursing home.
Timeliness of Data
Data collection can be time consuming and cumbersome. Data sources include surveys, health care reimbursement claims, and data submitted by health care institutions. There is usually a time lag between data collection and analysis and reporting. Even the most timely data at the State level are often a year old. (Health care institutions can often examine their own data every quarter, or more frequently, depending on the capabilities of their information systems.)
A State engaged in a quality improvement program should consider whether more recent data are available. If not, they should examine the likelihood that the reported data are still a valid reflection of the current status of the population. For example, an immunization campaign based on childhood immunization rates from 5 years ago would ignore potential results from an education and immunization campaign by health plans implemented in the past 3 years.Top of Page
Other Factors Affecting Performance Rates
It is important for users of performance measures to understand how those measures fit within the context of the larger health care system and what other factors may affect performance.
Health Care Disparities
Sometimes a performance measure may mask important disparities in health care. For example, State-level data may show that the overall rates of prenatal care in the State are very good. However, a State interested in reducing health care disparities might further examine the data by race, ethnicity, income, and geographic location to determine whether any specific subgroup has a disparate rate of insufficient prenatal care. This would be the group to target with a quality improvement strategy.
Levels of Care
Health care takes place at a variety of levels:
When using measures for a quality improvement campaign, it is important to define the appropriate outcome and target the right level to achieve the intended result. For example, an effective strategy for a program aimed at increasing rates of HbA1c testing among patients with diabetes depends on engaging both patients and clinicians. Patients need to seek care and comply with care regimens to effectively manage their condition, and clinicians need to offer the test and educate the patient. For the quality improvement strategy to make a difference, it must be implemented at the appropriate levels for the desired outcome.Top of Page
Socioeconomic and Environmental Context
State leaders should understand the socioeconomic and other environmental factors that may affect the health of the State's population. In addition to individual patient characteristics, each State has a different health care infrastructure and set of contextual factors over which policymakers may or may not have control. Examples of these factors include:
Many of these factors may affect performance rates in the State Snapshots and should be considered as the State develops a quality improvement strategy.
To aid this assessment, the State Snapshots provide contextual factors that may be related to each State's performance. These factors include demographics, health status, and health care resources. States may need to use the State Snapshots together with information on these environmental factors to determine whether additional data are needed to support the quality improvement effort.Top of Page
Estimating Potential for Improvement
Many of the measures reported in the State Snapshots reflect a variety of perspectives on the same or similar and related problems. Examination of the entire measure set will help to identify measures that stand alone or are part of a group of measures linked to a particular area of care.
For example, a standalone measure such as the percentage of nursing home residents who need help with daily activities may not provide a great deal of insight into quality of care in the nursing home setting. However, the entire set of nursing home measures taken together may give policymakers much broader insight into the severity of illness or condition of the population residing in nursing homes in the State. The measures can also indicate the overall effectiveness of nursing homes in meeting their residents' needs.
Important issues for States considering using measures in different ways to understand the extent of a health problem and the potential for improvement are discussed below.
Relationship Between Measure and Outcome
Some measures more directly identify potential steps for improvement than others. For example, if health plans perform poorly on a measure of beta-blocker administration after a heart attack, the suggested improvement is to increase physician prescribing rates for beta blockers. However, poor performance indicated by other measures, such as suicide rates, is a more complex challenge. A variety of factors and approaches might contribute to a successful quality improvement strategy.
Appropriate Targets for Improvement
For any quality improvement effort, it is important to set reasonable goals. For example, a quality improvement initiative focused on reducing the occurrence of a particular disease or condition is likely to have an impact on preventable cases only. Depending on the condition of interest, some cases may continue to occur, regardless of any quality improvement initiative. Thus, “perfect performance” may not be achievable. The “best possible rate” is probably just lower than the current rate.
It is important to understand what percentage of cases is preventable and to set goals that take into consideration realistic estimates of the number of preventable cases. Once States have identified measures and acquired relevant data, analysts must develop estimates that gauge State performance.
Benchmarks are external markers or values against which States can measure performance. The benchmark can be based on the national average or best performers. How a State fares depends on where the State estimate falls compared with the benchmark. The National Healthcare Quality Report (NHQR) provides a set of national and State estimates for quality measures that can be used as benchmarks for quality improvements.
Several types of metrics or benchmarks can be used for assessing a State. From more to less stringent, they include:
Some of these benchmarks can be found in the NHQR—the national and regional averages. The State Snapshots include individual estimates of the top performing States. A best-in-class average can be calculated from the individual State estimates shown in the data tables. Healthy People 2010 benchmarks are also provided for applicable measures via the Web site.Top of Page
Conclusion: Getting Started on Quality Improvement
Getting started on quality improvement is not an easy task. One strategy a State may find helpful is to identify other States with populations similar to those targeted for a quality improvement effort. For example, a State seeking to improve rates of pneumonia vaccination for people discharged from hospitals may want to model its efforts on those of a State that has previously implemented an improvement program in this area and demonstrated success.
In many cases, the greatest value in comparison may lie in identifying States that have started from relatively low performance and made incremental improvements. The State with the greatest improvements may have the most to contribute in demonstrating to other States how to encourage delivery system change that improves quality of care.Top of Page