In the United States, "Rigorous evaluations of health policy are exceedingly rare," leaving the government without the solid evidence it needs to determine "which health policies work and which don't," Austin Frakt, a health economist, writes for the New York Times' "The Upshot."

What CEOs need to know about health policy today

A few influential studies

"The United States spends a tremendous amount on health care," Frakt writes, but "less than 0.1% of total spending on American health care is devoted to evaluating them."

There are even fewer proper health policy evaluations than there are evaluations for medical interventions, according to Frakt who holds positions at the Department of Veterans Affairs, Boston University's School of Public Health, and the Harvard T.H. Chan School of Public Health, . While almost 80% of studies of medical interventions are randomized trials, only 18% of U.S. health policy trials are randomized.

As a result, there isn't a lot of evidence the government can use to inform its decisions on programs like Medicaid or Medicare, meaning the "small minority of policy evaluations" that "have had randomized designs…are widely regarded as the gold standard of evidence and commonplace in clinical science," according to Frakt.

For instance, a "classic example" is the RAND health insurance experiment from the 1970s, according to Frakt. By randomly assigning families to different levels of health care cost sharing, the experiment found that the family member responsible for the highest amount of the care cost use far less of it.

The results of the study have influenced health care insurance design every since. "In large part, you can thank (or curse) this randomized study and its interpretation for your health care deductible and co-payments," Frakt writes.

Another example is the recent Oregon study through which researchers planned a state lottery that randomly extended Medicaid coverage to some low-income adults. The findings revealed that Medicaid increases access to care and reduces financial hardship and rates of depression. The study has "been influential in the debate over Medicaid expansion," Frakt writes.

The problem with most health policy trials

There are other examples of randomized studies of health policies, but "the vast majority have far less rigorous designs, according to Frakt.  

For instance, the Center for Medicare and Medicaid Innovation, sends about $1 billion a year on pilot programs for Medicare and Medicaid services, but most of the pilots lack randomized designs, according to Frakt.

"Also potentially problematic: Most of its programs rely on voluntary participation by health care organizations," Frakt writes. "There might be crucial differences between those that opt in and those that don't."

On the other hand, requiring mandatory participation comes with its own set of challenges, Frakt writes.
If you force a hospital to join a new program, but not its competitor down the street, you might put the hospital at an unfair financial disadvantage,” said Nicholas Bagley, a health law professor at University of Michigan.

But randomization can also be "complex and hard to maintain," according to Frakt, and while the design may be desirable for evaluation, it's not always the most realistic design.

And just like in clinical settings, there are methods that come out of low-rigor studies that are later overturned by more rigorous trials, Frakt writes.  "For example, many low-quality studies suggest that wellness programs reduce employers' health care costs as they improve health outcomes. But when the programs have been subject to randomized controlled trials, none of these findings hold up."

An issue that goes beyond health care

But the lack of policy evaluations is not unique to health care, according to Frakt.

Governments across all levels routinely implement policies without any plans to evaluate their effectiveness, or "how to unwind them if they don’t, or how to build on them if they do," according to Frakt. For instance, a 2017 Government Accountability Office report found the vast majority of managers of federal programs didn't know if their programs were recently evaluated. "In most cases, none had been done. In others, none had been done in the past five years."

But, Frakt writes that it will be "hard to rid ourselves of ideas that are little more than wishful thinking," but "The first step would be to do more rigorous policy evaluations. The next would be to heed them" (Frakt, New York Times, "The Upshot," 9/9).