Cross-national efficiency comparisons of health systems, subsectors and disease areas (2024)

7.3.1. Health system-based approaches

System-based health care efficiency studies use aggregate country data to construct measures of efficiency. Often, studies have made use of the aforementioned OECD data to conduct these analyses. Many analytic approaches have been taken, with no consensus on the correct methodological approach. The majority of studies use DEA to estimate a production possibilities frontier and incorporate multiple inputs and outputs into the estimate. Studies occasionally take more simplistic approaches; one such study calculated ratios of mortality rate reductions relative to the health care share of gross domestic product for 19 countries (Pritchard & Wallace, 2011). However, this type of analysis is problematic because mortality is not directly attributable to health care spending without efforts to control for factors outside the health system that are also likely to affect mortality rates. Second, in this particular study there was no effort to adjust for differences in scale, for example, even the same health spending as a share of gross domestic product will reflect very different levels of resources dedicated to health depending on the size of the economy.

One of the first large studies to compare the efficiency of health systems is also one of the most well-known and often criticized studies of health system efficiency. An analysis of panel data from 191 countries was conducted by the WHO to compare per capita health expenditure to life expectancy (adjusted to account for disability), after controlling for educational attainment (Evans et al., 2001). The models used country-fixed effects, which take advantage of variations within each country over time to estimate parameters; only one country was deemed to be efficient using this method. An efficiency index was then constructed, where the expected level of health if there was no health care expenditure was compared to the expected level of health if all health systems were as efficient as the best performer. Based on this analysis, Oman was ranked the most efficient country, with Zimbabwe the least efficient.

The WHO efficiency study (Evans et al., 2001) and related study of overall performance in the 2000 World Health Report (WHO, 2000) have been heavily criticized both on methodological and data quality grounds. For example, a study by Hollingsworth & Wildman (2003) uses parametric and non-parametric approaches and found that their results varied from the method used by the WHO. Using DEA and SFA, they demonstrated how the panel data approach used by the WHO did not permit assessment of changes over time, but rather, assumed that efficiency within a country remains constant. They also suggested that analysis in the future should compare similar countries, rather than attempt to estimate efficiency across a range of countries with very different characteristics that cannot be appropriately accounted for using modelling techniques. Other research into the robustness of the WHO methodology has also revealed how sensitive the rankings are to how efficiency is defined and how models are specified (Gravelle et al., 2003). Gravelle et al. suggested that the use of country-fixed effects to estimate the models is inappropriate, particularly because 50 of the 191 countries in the study had only one year of data, and therefore had no variation over time. They found that alternative approaches, including using a model that exploits variation between countries (rather than relying on variation within countries over time), and changing the units of the variables so that they are not logarithmic, changed the results considerably. For example, in a model that used between-country effects instead of country-fixed effects, Oman changes from the most efficient country to being ranked 169th. Following widespread criticism of this analysis, the WHO has not attempted any further performance ranking.

Similar research using DEA and panel data regressions has been prepared using the OECD data (Joumard, André & Nicq, 2010). Joumard, André & Nicq (2010) estimated the contribution of health spending to life expectancy, accounting for lifestyle and socioeconomic determinants. The authors concluded that if health spending in all countries were as efficient as the best performing countries, life expectancy would increase by two years without a need to increase the actual level of spending. In contrast, increasing health expenditure levels in countries where health expenditure is not high-performing to begin with does little to increase life expectancy. The results suggested that Australia, Japan, South Korea and Switzerland are among the countries that make the most efficient use of health care expenditure. The study also found negligible relationships between output measures of efficiency (such as average length of stay) and outcome measures (such as life expectancy). This indicated that countries that most efficiently produce health system outputs might not necessarily also produce actual health gains most efficiently.

Rather than control for lifestyle factors at an aggregate country level, one recent study attempted to compare health system efficiency using data on life expectancy and health care expenditure that are adjusted a priori for individual-level differences in lifestyle factors, such as smoking, alcohol consumption and body mass index (European Commission, 2015). Unsurprisingly, the research concluded that healthier lifestyles would lead to longer life expectancy at per-person curative health care spending levels. However, despite wide variation in health behaviours across the 30 European countries in the study, the lifestyle-adjusted country efficiency estimates did not differ considerably from those that were unadjusted; most countries appeared to be positioned similarly relative to the efficiency frontier in both adjusted and unadjusted analyses. Additionally, the effects of changes in lifestyle were difficult to infer from this analysis, since the estimates were based on cross-sectional data. As noted by the authors, interventions to actually improve health behaviours may themselves be costly, and may also not have short-term health benefits that are comparable in magnitude to those reported.

Using life expectancy as an outcome measure also only tells part of the story. Other studies using OECD data adopted other health outcomes, such as reductions in infant mortality as a measure of health system outcomes. One such study used a DEA approach, where health outcomes, life expectancy and infant mortality are dependent on a number of inputs (Retzlaff-Roberts, Chang & Rubin, 2004). Rather than only focus on health expenditure as an input, this study accounted for health care resources such as the number of beds, MRI units and physicians, as well as social factors such as schooling, the Gini coefficient and tobacco use. For example, using an input oriented model of infant mortality, the authors found that the efficiency frontier is formed by Ireland, Mexico, Sweden and Spain. Based on this, the authors concluded that the USA should be able to reduce its health care inputs by 9.3% and, if it were more efficient, would still be able to maintain its level of infant mortality. However, there is no clear rationale for including all inputs together in the model; including health expenditure in addition to human and capital resources, which are purchased by the health sector, would seem to double count inputs and could invalidate the findings of this study and others that take a similar approach.

Indeed, there is no clear agreement on how to identify appropriate inputs or to control for non-health system factors that influence health. A study that used both OECD and WHO panel data demonstrated how complex the health production process is by considering socioeconomic determinants of health that are outside the health system as inputs to producing life expectancy (Spinks & Hollingsworth, 2009). The authors suggested that using both macro level socioeconomic factors, such as government policies, housing or working conditions, in addition to intermediate-level socioeconomic factors like psychosocial characteristics and health behaviours in the same model could be problematic, as the inputs are inherently interlinked. The study instead used a single measure each for education (school expectancy), employment (total unemployment rate), income (gross domestic product per capita) and health expenditure as inputs to producing either life expectancy or disability-adjusted life expectancy. Using DEA, the authors found that countries have generally moved away from the efficiency frontier over time, implying that on average, efficiency decreased slightly. The countries that formed the efficiency frontier and were deemed efficient for all model specifications were Greece, Japan, Mexico, South Korea, Spain and Turkey.

Other studies have illustrated the complexity of health production and also highlighted that inputs other than medical care play a large role in producing health. One such study used OECD data to investigate the differential effects of health system and non-health system inputs on DEA estimates of efficiency (Hadad, Hadad & Simon-Tuval, 2013). In two separate models, in addition to total per capita health expenditure as an input, the other inputs included either health system characteristics such as beds and physician density, or factors arguably outside the control of the health system such as gross domestic product and consumption of fruits and vegetables; life expectancy and infant survival were the chosen health outcomes. The study found that many countries that were efficient in the model accounting for health system inputs were not efficient when accounting for factors outside the health system. For example, using both models, the Czech Republic, Estonia, Iceland, Japan, Poland, Portugal, Slovenia and South Korea were efficient; Australia, Canada, Israel, Italy, Luxembourg, Spain, Sweden, Switzerland and the United Kingdom were efficient when using health system inputs, but not when using gross domestic product and consumption of healthy food as inputs instead. The authors allowed for super efficiency (where inputs and outputs in each country are weighted to maximize the efficiency score without being constrained to a maximum possible score of 1) to calculate rankings and found that the most efficient country using the health system input model was Iceland, whereas the most efficient country using non-health system inputs was Japan. The study also calculated rankings using cross-efficiency, where all countries shared the same weights; this is a potential measure of AE if we assume that the weights used represent the optimal mix of inputs and outputs. Using the cross-efficiency approach, Canada was the most efficient country using the health system inputs model, while using non-health system inputs, the Czech Republic was most efficient.

An earlier study also used the OECD Health Statistics data and similarly found that features referred to as environmental factors play a large part in observed variations in efficiency estimates (Puig-Junoy, 1998). With male and female life expectancy as the outcomes and five health system inputs (numbers of physicians, non-physician personnel and hospital beds, as well as tobacco and alcohol consumption, all relative to population size), various model specifications found that the most efficient countries are most consistently Canada, Greece, Italy, Japan, Portugal and the USA. The study then employed a two-stage approach, where after DEA was used to calculate country efficiency scores, regression techniques assessed the relationship between observed scores and environmental factors: human capital (that is, average years of schooling), the private share of total health expenditure and the presence of primary care gatekeeping. Nevertheless, much of the variation in efficiency scores remains unexplained by their regression models.

While the majority of studies employ DEA in a traditional sense to assess inputs relative to outputs, one study used DEA in a unique way to construct composite indicators of health system efficiency based on the set of efficiency indicators available from the OECD Health Statistics data (Cylus, Papanicolas & Smith, 2015). In this study, each individual efficiency indicator was treated as an output in a DEA model and inputs were held constant. DEA then attached weights to each efficiency indicator separately for each country so that each country’s composite score was maximized, casting it in the best possible light. By combining several efficiency indicators into a single measure, the objective was to see if there was evidence of system-wide efficiency effects. Using all partial efficiency measures as outputs, five countries – Estonia, Hungary, Slovak Republic, Slovenia and the United Kingdom – formed the efficiency frontier. The study found that Hungary was the only country that was efficient in all model specifications presented.

Finally, although most studies used publicly available international databases like the OECD Health Statistics database, the Commonwealth Fund has assessed health system performance based largely on its own International Health Policy Survey (IHPS). This data set differs substantially from the OECD and WHO data, because it is a telephone-based survey of a random sample of individuals from a set of high-income countries, rather than national-level data from official sources. Caution should be exercised as the samples in each country are small, and the data are self-reported and may thus be subject to bias. The survey captures some variables that could be considered as indicative of efficiency, such as whether an individual had a duplicate medical test, rehospitalization and timely access to records, and whether a physician used information technology. In their 2014 report, the Commonwealth Fund assessed health system efficiency based on these data in addition to data on expenditure levels (Davis et al., 2014). The authors found the USA to be the least efficient of the countries analysed – a consistent finding across all waves of their survey. The USA spends high shares of total health spending on administrative costs, and its doctors report that they spend too much time on paperwork, clearly indicating that there are administrative inefficiencies in the USA. Yet, there are important discrepancies in the report that warrant further analysis; overall, the United Kingdom was the most efficient country based on its low level of expenditure and high scores according to the process measures collected in the IHPS. However, the United Kingdom performs second from last in terms of healthy lives, which raises questions of how a health system that fails to achieve good health outcomes can be considered the most efficient.

Overall, many system-level studies have taken advantage of access to international harmonized data sets to compare efficiency, with their added value generally being the use of analytic techniques. Despite efforts to account for other inputs that have an effect on health outcomes, such as lifestyle, education or institutional characteristics, much of the variability in efficiency scores appears to be unexplained by health system characteristics or other factors. It is unclear how successfully confounders can be controlled for. Additionally, most studies took a very narrow perspective on the outputs of health system, with the main products of the health system being life expectancy and infant mortality. Of note, there seems to be little consistency across studies in the countries that are found to perform most efficiently, despite studies frequently relying on the same data sets.

7.3.2. Subsector-based approach

While the aforementioned studies use country-level data to compare health systems, similar international comparisons have been done at the subsector level, the most common being to compare hospital sectors across countries. At this less aggregated level, because patient characteristics are often more hom*ogeneous than population characteristics, variations in outcomes are likely due to unobserved confounding factors to a lesser degree. There are also a number of outputs, such as hospital discharges or physician visits, which can be assessed that are not possible at the health system level. Common frontier-based analytic techniques, DEA and SFA, are also employed.

For example, using the OECD panel data between 2000 and 2009, a recent study employed both DEA and SFA methods to explore efficiency in hospitals, adjusting for differences in case severity and environmental factors (Varabyova & Schreyögg, 2013). Using discharges weighted on the basis of case severity as an activity-based output and in-hospital mortality rates for AMI, haemorrhagic stroke and ischaemic stroke as additional outcome measures, the authors assessed the efficiency of the number of beds and hospital workers, and a number of other factors, such as health care spending, length of stay, education and patient mix. Importantly, the authors found that countries that demonstrated good health outcomes, like Japan, might be technically inefficient based on their use of health care resources. The authors also found that countries with longer length of stay are less technically efficient using their methodology, implying that length of stay may be a reasonable proxy measure for efficiency of the hospital sector.

To more appropriately compare the prices and volumes of health care services across countries, there have been joint efforts by the OECD and Eurostat to develop output-based purchasing power parity (PPP) price indices (Koechlin et al., 2014). This involves estimation of quasi-prices based on the reimbursem*nt levels paid for comparable medical services (for example, payments covering direct, capital and overhead costs), as opposed to being based on the prices of inputs to care (for example, wages), which can be used to compare hospital prices and volumes across countries. This is important for the measurement of health care efficiency, as the input-based PPP price index methodology unrealistically assumes that health care productivity is identical across countries, because hypothetical countries with the same input prices (for example, wages) and health care expenditure would implicitly be assumed to have produced the same volumes of health care goods and services. In addition to improving the estimation of hospital volumes, in an earlier iteration of this study, the methodology was used to calculate novel comparisons of inpatient care productivity through metrics such as the cost of an inpatient care day (Koechlin, Lorenzoni & Schreyer, 2010). Per-day costs in 2007 were highest overall in the USA and lowest in South Korea.

Researchers have also compared efficiency for specific types of care provided within a hospital. A recent study using the OECD data investigated inpatient mental health care, where mental health-specific inputs included the number of psychiatrists, psychiatric beds and length of stay, and the outputs were the discharges per 1000 population (Moran & Jacobs, 2013). Factors external to the health system that could potentially play a role involved alcohol consumption, income, education and unemployment rates, and were included in some of the DEA model specifications presented. Unlike in many other studies using DEA, the authors used a bootstrapping approach that allowed them to calculate confidence intervals so that they could ascertain how certain they were of the rankings. They found that countries with greater efficiency, including Denmark, Hungary, Italy, Poland, Slovenia and South Korea, also had wider confidence intervals, suggesting less certainty; in general, however, the countries deemed efficient in one model were also reasonably efficient in other specifications.

Not all studies review large numbers of countries. Often, international comparisons have been limited to smaller numbers of health systems, which could potentially allow for more detailed comparisons. For example, a study comparing the NHS in England to the Kaiser Permanente integrated managed care consortium in California, a private health maintenance organization that integrates both financing and delivery, concluded that Kaiser Permanente performed more efficiently than the NHS (Feachem et al., 2002). This study informally compared costs to quality of care, showing that per capita costs were roughly the same but that there were notable variations in quality and responsiveness across the two systems. Another study comparing many hospitals in Norway and California used DEA to estimate a production frontier to investigate whether privatization and competition lead to greater efficiency (Mobley & Magnussen, 1998). The authors matched hospitals based on a variety of criteria to ensure that they were comparing similar types of hospitals and concluded that private competition among hospitals in California does not lead to greater efficiency in the long run.

Another study compared hospitals in Germany (Saxony federal state) and Switzerland using DEA, finding in all instances the German hospitals were more efficient. Unlike in most studies, the analysis considered the number of patient days as an input rather than an output (Steinmann et al., 2004). The justification for treating days as an input is that in Saxony, hospital financing is based on pre-approved patient days; hospital managers are incentivized to meet this pre-approved level of patient days by attracting less complex cases. Likewise, in Saxony the number of beds per hospital is fixed so in some analyses beds are not included as an input since they are non-discretionary. This underscores how important it is to understand institutional arrangements within countries before conducting any analysis, as incentive structures or financing mechanisms could potentially drive results. Importantly, the authors tested whether their sample of hospitals was hom*ogeneous and found that it was not, which required that they limited the usable sample substantially. This highlights another important issue: most studies that employ frontier-based analyses do not effectively ensure that the decision-making units (for example, countries or providers) are comparable and exist as part of the same production possibilities frontier.

It is essential to note the large number of studies that have been done in Scandinavian countries because of the wide availability of registry data, which historically has not been readily available in many other countries (see Chapter 3). Using such data, it is easier to make sure that inputs and outputs are well defined and comparable, and to control for confounding factors. For example, a DEA study comparing Finnish and Norwegian hospitals used data on hospital operating costs as inputs and DRG-weighted admissions, and weighted visits and days of care based on National Discharge Registry data (Linna, Häkkinen & Magnussen, 2006). Registry data allowed the authors to cluster cases by NordDRG grouper and use cost weights based on actual patient-level costs. The authors also adjusted input prices based on a hospital-specific input price index comprising hospital-specific wage and operating cost data. The study results were reasonably robust across multiple models and indicated that Finnish hospitals were more efficient than Norwegian hospitals. A more recent analysis of university hospitals in Nordic countries (Denmark, Finland, Norway and Sweden) used similar data and also found Finnish hospitals to be the most efficient. The study also employed a bootstrapping DEA approach that allowed for estimation of confidence intervals (Medin et al., 2011). This gives not only an idea of the range of certainty, but also corrects for bias associated with having only a small number of hospitals.

7.3.3. Disease-based approaches

Health system efficiency can also be explored by examining the costs, resources, outputs and outcomes associated with treating specific diseases. The advantage is that patients treated for certain diseases are likely to be more hom*ogeneous. Additionally, it may be possible to more accurately observe the processes leading to differences in efficiency if the data are detailed enough.

For example, the McKinsey health care productivity study examined variations in inputs and outcomes for treating breast cancer, lung cancer, gallstones and diabetes in Germany, the United Kingdom and the USA (Garber, 2003). Data on the levels of inputs, such as physician hours, nursing hours, medications and capital, were used as opposed to the level of spending, because spending could lead to erroneous efficiency estimates because of differences in input costs across countries. These data were linked to outcome measures. Although spending levels in the USA are higher than in Germany or the United Kingdom, the USA was largely found to perform efficiently using this method, suggesting that US providers may use resources efficiently but that their input prices are notably higher.

Additionally, while the OECD Health Statistics database contains primarily aggregate, system-level data, the Health at a Glance report based on these data contains some disease-based efficiency indicators. These data are still aggregated at the country level, but can shed light on the efficiency of treating specific conditions. For example, while Finland and Turkey had the longest and shortest average length of acute care hospital stay in 2012 overall among European countries, the average length of stay for AMI was longest in Germany (10.3 days) and shortest in Denmark (3.9 days). Additionally, there were comparisons of the percentage of cataract surgeries carried out as day cases (Figure 7.5). A higher share is indicative of greater efficiency, as cataract surgeries are a high-volume surgical procedure that can potentially be done using fewer resources as a day case rather than as an inpatient admission. However, caution is advised when comparing across countries for a variety of reasons; for example, some countries do not consider outpatient cases in hospitals or surgeries outside of hospitals when reporting these figures.

Other research also compared costs and outcomes for selected diseases. For example, the OECD Ageing-Related Diseases study included some indicators of efficiency, such as length of stay and unit costs of treating diseases including stroke and cancer; however, in many instances data were not available or fully comparable across countries. More recently, the OECD Cancer Care study investigated variations across 35 countries in the resources allocated to cancer care, as well as variations in care delivery and outcomes (OECD, 2013b). For example, the report compares average referral times between GP and specialist visits, finding that referral times are shortest in Denmark (typically only a few days) while they can be a month in Israel or Norway. Comparing waiting time between diagnosis and initial treatment shows even wider variation, from under 3 days on average in Luxembourg to over a month in Poland or the Netherlands. For both of these indicators, in some instances the data are estimates based on expert opinion. In exploratory analysis, the OECD compared resources for cancer care, such as oncologists per million population and use of imaging technology and found a significant association between resources for cancer care and survival. However, the study did not include explicit cross-country comparisons of the efficiency by which cancer care is delivered. Nevertheless, the study suggested that some countries have higher survival rates at given levels of total per capita health spending than others, which could suggest greater efficiency despite per capita health spending being only a weak proxy for health system inputs specific to cancer care. While there appear to be some diminishing returns for cancer survival given greater health spending, Iceland, Israel and Turkey have the highest five-year survival rates for breast cancer relative to their levels of total per capita health spending (Figure 7.6). The OECD has also conducted similar research that links health care quality indicators to expenditure for cardiovascular diseases and diabetes (OECD, 2015).

DRGs and other methods that group similar cases have been used for efficiency measurement, not only to weight discharges and days as described previously, but also to group cases so that similar types of care are compared across health systems (see Chapter 2). Because of their design as patient classification systems that group patients with similar characteristics and resource consumption, they can be effective at ensuring that similar types of patients are matched. Three large studies, the HealthBASKET, EuroDRG and EuroHOPE have made major strides in this domain.

The HealthBASKET project reviewed the costs of care for nine European countries (Busse, Schreyögg & Smith, 2008). Using case vignettes that described particular types of patients (that is, based on age, gender and comorbidities), the study compared and attempted to explain variations in costs within and between countries. The advantage of this approach is that specific services for comparable patients could be costed and compared across countries. Vignettes were developed for inpatient, outpatient, elective and emergency care. Using a sample of providers, the researchers collected information on typical usage patterns and costs. However, despite successfully averting the need to risk adjust by comparing standardized patients, there were limitations. For example, the samples were small and therefore often reflected normative cases rather than actual patient experiences. Additionally, data were not always comparable across countries because providers in some countries do not own their assets. Likewise, patient outcomes were assumed to be identical, which is not realistic. Nevertheless, the approach revealed how low costs in southern and eastern Europe were largely due to low wages and did not necessarily reveal greater efficiency after adjusting for episode-specific PPPs. Some of the most important reasons for variations in costs were the differences in the types of technologies used to provide care.

EuroDRG used an episode-of-care approach to compare costs across countries (Busse, 2012). This study investigated the classification variables used by different country DRG systems, such as diagnosis, procedure, patient age, length of stay, death and the level of reimbursem*nt for a selection of similarly defined patients based on episodes of care. Examples of the types of care reviewed included child care (Bellanger & Or, 2008), stroke (Epstein, Mason & Manca, 2008) and cataract care (Fattore & Torbica, 2008).

The theory behind the EuroDRG project is based on the fact that most analyses of efficiency cannot properly control for differences in case mix. As a result, the study takes advantage of a different unit of measurement, episodes of care, which are essentially meta-DRGs that are uniformly defined based on a number of diagnosis and procedure codes. Patients are observed from the time of diagnosis until the end of treatment including follow-up. The advantage is that patient characteristics are not standardized, as in the HealthBASKET study, but rather, similar types of patient care are compared. The study subsequently estimated how well the DRG systems could explain variation in resource consumption, particularly how well DRGs explained variations in costs or length of stay for each episode of care (Street et al., 2012). Nevertheless, most of the EuroDRG analyses could not identify variations in quality of care.

A recent paper within the EuroDRG project compared costs and quality (measured as being discharged alive) for AMI and stroke patients in hospitals in Finland, France, Germany, Spain and Sweden (Häkkinen et al., 2014). The study used patient-level data and separate models to predict costs and survival across around 100 hospitals. Though the purpose of the study was to evaluate trade-offs between cost and quality (that is, explicit ratios of survival to cost were not reported) the hospital fixed effects of both equations were plotted against each other to give an idea of whether hospitals that spent more on patients achieved better outcomes, though no conclusive cost–quality relationship was found.

Another recent project, the EuroHOPE, has made important advances in disease-based efficiency comparisons across countries (Häkkinen et al., 2013). This study used linkable patient-level data, which allowed for the measurement of both outcomes (including follow-up) and the use of health care resources (costs, days of care, procedures and drugs) for comparable patient groups. The diseases investigated were AMI, stroke, hip fracture, breast cancer and low birth weight. EuroHOPE could evaluate entire treatment pathways and identify the extent to which a health care system produces better outcomes. Key strengths of the project include: detailed data on patients and their comorbidities; identification of the beginning and end of episodes of care; and reliable data on health care costs; however, such high-quality data is not available for all countries or many types of care.

7.4.1. Improving data availability and consistency is a key challenge

Differences in data availability and consistency are important challenges to creating comparable health care efficiency indicators. There are few longitudinal efficiency metrics currently in the public domain; while efforts have been made to harmonize data, there are still issues due to differences in definitions, clinical practices and reporting. Additionally, available national-level data allow for only a limited number of efficiency indicators to be constructed manually based on ratios of outcomes or outputs to inputs (for example, consultations per physician). At a subsector level, aggregate national-level data are most readily available to assess resource usage for hospitals, but often not for other types of providers. From a disease-based perspective, conditions for which survival is likely in the presence of timely access to quality health services, such as some types of cancer, are promising areas for efficiency comparison, although episodic data are not always available in many countries.

While international databases like the OECD Health Statistics do not contain many ready-made efficiency comparisons, a large number of the studies reviewed in this chapter make use of the OECD data to measure efficiency. Yet, there are questions as to whether the use of the OECD panel data is entirely appropriate for calculating efficiency indicators, as the data are sometimes estimates (Spinks & Hollingsworth, 2009). In general, there are inconsistencies in measurement and reporting standards across countries, which researchers have limited capacity to control, in spite of the headway made by resources such as the SHA. Even within the United Kingdom, a National Audit Office report concluded that it was not possible to compare efficiency successfully across the four countries (National Audit Office, 2012), primarily because of a lack of data availability and consistency. For example, differences across the United Kingdom in how countries categorize types of expenditure make spending comparisons nearly impossible.

Therefore, while significant efforts are being made to improve the quality and consistency of expenditure and non-expenditure data, this remains an important challenge to improving efficiency measurement. On the expenditures side, the SHA is an excellent example of the advances that can be made coordinating data reporting across countries. However, efforts are needed to improve the comparability of input data other than expenditure, and particularly, to increase the availability of comparable output and outcome data. While data for many different types of inputs – from expenditure, to beds, to the number of doctors, to drugs – are available, there is also a need to expand the types of outcome data that are available for analysis. Studies that use health outcomes often rely on life expectancy or infant mortality, but these broad indicators of health status are very distant from the activities of health care systems. Outcome data for conditions that are known to be amenable to health care should be more readily available. There is a need to develop more PROMs, following the example of countries such as Sweden and the United Kingdom, to monitor more closely the health outcomes of different health care interventions.

It would be prudent for countries to focus more on harmonizing and improving access to registry or hospital discharge level data. Not only would better micro level data be preferable from a methodological perspective because it would be easier to control for potential confounders such as case severity and compare like-with-like, it could also be more useful to end users, such as hospital managers or policymakers, who require detailed information that allows them to take action. While it is difficult to harmonize registry or discharge data because they are used to meet administrative needs, which often differ across countries, these data are exceedingly useful and allow researchers to determine their own levels of aggregation. Data that allow researchers to follow patients throughout treatment across different providers are essential to understand the efficiency of care pathways. Similarly, longitudinal data need to be available to track changes in efficiency across time.

7.4.2. Researchers should make use of multiple methodological approaches in the absence of the correct methodological approach

There are numerous methodological approaches that can be useful for measuring efficiency, including frontier-based methods like DEA and SFA. Although DEA seems to be preferred based on this chapter’s assessment of the international literature, it still remains unclear which methodological approach to use in general for estimating efficiency, regardless of whether comparisons are to be done across countries. While some studies confirm that DEA results are similar to other non-parametric methods (Afonso & St Aubyn, 2005), results are not always the same across methods and can be sensitive to model specification (Gravelle et al., 2003). Even if data are comparable, the question remains of which are the appropriate inputs and outputs to be compared across countries. As demonstrated in this chapter, variations in the selection of these data can lead to very different results. For example, despite high spending and generally poor health outcomes, when input prices are not taken into account, the USA might appear to perform efficiently (Garber, 2003).

As a result, all studies should make use of multiple techniques to ensure robustness and the results of many different studies should be used together to inform conclusions. Likewise, more studies should make use of methods that quantify the level of uncertainty of an estimate, including bootstrapping for DEA to estimate confidence intervals. Ultimately, when deciding which methods to use, researchers must consider the extent to which methods are helpful to policymakers. Given that arbitrary decisions regarding inclusion or exclusion of inputs and outputs can have a significant effect on results, methods like DEA may not be useful for policy purposes (Spinks & Hollingsworth, 2009).

Importantly, methods like DEA and SFA assume that all entities exist in the same production possibilities frontier. A perhaps underappreciated issue with international efficiency comparisons is that this requirement may not be met. Differences in system designs may mean that providers do not have the same production possibilities, leading methods like DEA to inaccurately estimate production frontiers. For example, smaller countries or countries with more geographically dispersed populations may require greater spending on inputs, such as medical imaging technology, to make care available to the entire population. This issue of entity comparability was also elucidated in a study comparing Switzerland and Germany, which found that many hospitals in the two countries were not sufficiently hom*ogeneous to be compared using frontier-based methods (Steinmann et al., 2004).

Other related methodological issues include the need for improved risk-adjustment techniques because of substantial heterogeneity across and within countries. However, to do this well, adjustment needs to be made at the individual level, as national-level risk adjustment does not properly account for variations within population groups. This also supports greater use of patient-level data. Using individual-level data, it is easier to ensure that patients are comparable and to subsequently identify the characteristics of the health system that contribute to differences in efficiency.

7.4.3. More work is needed to properly attribute health outcomes to inputs

One reason why some types of performance metrics are fairly common (for example, population health) while efficiency is not, may also be partially due to the well-known difficulties attributing outcomes and outputs to inputs. The production of health is influenced by many factors that lie outside the health care system. While the challenges associated with attributing health system characteristics to health outcomes do not only apply to international comparisons, it is perhaps an even more salient challenge when comparing across countries because factors that influence health in some countries may vary to such a large extent that they are nearly impossible to control for. For example, factors such as genetic differences, geographical ancestry (Diez Roux, 2011) and cultural lifestyle differences play a role in health. Because the effect of the health system as an input to health is interrelated to and dependent on many country- or context-specific characteristics, it is difficult to accurately isolate the contribution of the health system itself in different country contexts.

Reasons for variations in efficiency across countries or over time, such as changes in the way care is delivered, changes in case mix, economies of scale and determinants outside of the health system are not consistently accounted for, or in many instances, cannot be properly accounted for. Although there have been attempts to include non-health system factors in analyses, it is not precisely clear which are the right ones and whether including all factors makes sense since so many are interrelated. To some extent, all policies and environmental factors are likely to play a role in determining health, making comparisons across dissimilar countries increasingly complex. Likewise, lifestyle behaviours, including healthy eating (Hadad et al., 2013), social class and welfare, or even occupation, may be as important, if not more important than health care in determining population-level health outcomes, highlighting the potential difficulties with attribution across countries. One solution might be to match patients not only based on DRGs or episodes of care, nor on simple characteristics such as age or sex, but also on more detailed observable characteristics including genetics.

On a related front, attribution issues may be one reason why it is difficult to measure efficiency for many subsectors of the health system. For example, there are very few estimates of long-term care efficiency because of the difficulties attributing changes in individual outcomes to care services.