The Stew BLOG
The Sense—and Nonsense—of Using ROI in Population Health
I had a conversation last week with the executive director of a free clinic. The clinic provides roughly $3 million a year in healthcare services to people who are uninsured. The clinic’s board of directors hopes to move beyond a business model reliant solely on grants, but is being confronted with demands to “prove savings” as a condition of financial support.
Compare the clinic’s experience with that of an entrepreneur I recently sat next to on a plane. His company had developed an app to remind patients of their doctors’ appointments or to pick up their medications. For this service, his company was paid a fee for every patient transaction (e.g., each doctor’s appointment attended or every prescription picked up) as well as a per patient, per month fee by insurers. Curious, I asked him what he had been required to provide by way of proof that his app would positively impact their bottom lines. His response: Nobody asked.
These two examples, although anecdotal, demonstrate opposite ends of the spectrum in the application of Return on Investment (ROI). The entire healthcare field is wrestling with the concept of ROI, and for the moment, we have very uneven standards, even as we move toward value-based payments.
In my view, ROI is an important concept. However, it is often inappropriately applied to population health. Those in the field of population health are routinely challenged to “show me the ROI.” But how often are health care providers — or the purveyors of new pharmaceuticals or medical devices — required to “prove savings” (or, in the stories above, even effectiveness) as a condition of being paid for the good or service being offered?
Much of the discussion in the field centers on how to measure ROI. In this post I’d like to back things up to a more basic question. When and how is it appropriate to apply ROI to population health? Here are three common applications and my provisional thoughts:
- Evidence of expected yield. Is it important to develop and act on evidence about which population health investments can be expected to yield positive financial returns and the source of those returns? Absolutely.
- Decision-making consideration. Should financial ROI be the sole deciding factor in making investments? Questionable.
- Conditional payments. Should payment for services be conditional on actual ROIs? Fraught with all sorts of problems.
A few definitions are in order before we start with an example. ROI is a very simple calculation: investment returns minus investment costs divided by investment costs. I like this because it shows how efficiently a dollar invested is at producing “profit.” That is, it shows which investments produce the greatest bang for the buck. Alternative measures include benefit-cost ratios (investment returns divided by investment costs) and pay-back periods (how many years before investment costs are recouped by investment returns). However, you can see that the simplicity of all of these formulas is deceptive. What exactly counts as a cost and as revenue? Over what time period? And even more complicated for population health, to which stakeholders in which sectors do the savings accrue?
An example. Suppose you’re interested in energy efficient windows for your house—a sizeable investment. The window manufacturer provides you evidence of expected yield by quoting you the payback period. So far, so good. You understand the potential returns. You can decide whether three or seven or 10 years is acceptable, and make the decision on whether or not to invest.
Maybe the payback period is seven years—which seems a bit long in your mind, but climate change is important to you. So your decision-making criteria includes both environmental benefits plus financial ROI.
Note that the window manufacturer makes no guarantees about your returns, and you owe the money up front at a fixed price, no matter what transpires over the next seven years. Imagine instead that we made payment conditional on actual savings. It would create layers of complexity. Sophisticated and costly controlled trials would be needed to measure your actual savings because so many variables influence actual savings, and these would need to be teased out and accounted for. Suppose for example, that the next three winters are unseasonably mild. Or suppose that your nephew decides to live in your basement for a protracted period and he loves to crank the heat in the winter and the air conditioning in the summer. There’s no way the window manufacturer would agree to be paid based solely on actual savings over time—unless you sweetened the deal considerably (which, of course, would change your ROI).
Complexities. ROI is nothing more than a measurement tool. While ROI is a simple formula, the actual calculation can become quite complex (see for example, this guide from Rutgers on calculating savings from ACOs). ROI can be manipulated — and it doesn’t tell the whole story — even in the private sector where it was born. See this article in Harvard Business Review, for example, discussing the weaknesses of using ROI as a management control device, or this in a Forbes article, pointing out the weaknesses of applying ROI to complex interactions (i.e., what really caused what).
And when the subject matter is especially complex (say, in measuring the impact of a population health intervention as opposed to the returns from a new capital investment), ROI gets squishier.
Greater care must be taken in using it as a guide to investments. For example:
- Social return may be greater, or more important, than financial return. Sometimes preventive efforts result in people living longer, which in turn drives up health care costs. Should we value financial ROI over life? This dynamic played out, for example, in the Certification of the Medicare Diabetes Prevention Program (DPP), by the Centers for Medicare and Medicaid Services (CMS) Office of the Actuary:
It is unclear whether the program would break even over the participants’ lifetimes. These results appear to stem from improvements in mortality experienced by the participants. Mortality improvement would result in additional years of coverage under the Medicare program, which would offset a portion of the near-term savings. Since the cost effects of the potential mortality improvement shall not be considered for the purposes of certification, the modeling supports this certification that the considered DPP expansion is expected to reduce Medicare expenditures.
- The ROI of preventive measures is a moving target because the health care costs of treating a condition change over time. Treatment costs may go up (or down) as new and more expensive (or cost-effective) treatments are developed. Therefore, the ROI of a single preventive measure will change as treatment costs change. Should we invest in the preventive measure only when the ROI is high, and not when it’s low?
For example, the ROI of preventing obesity has increased significantly over the past thirty years for this very reason, as Charles Roehrig and George Miller wrote in Health Affairs. “Since the 1980s, expensive new treatments for obesity and its associated medical conditions (such as new drugs to treat diabetes, hypertension, and hyperlipidemia, and new surgical procedures such as bariatric surgery) have driven up obesity-related health care costs… As a result, reductions in obesity save more in health care costs today than in the 1980s.”
Reasonable standards. So what standard should we use to apply ROI to population health? I think the answer is it depends, as unsatisfactory as this answer may be. This is by way of suggesting that while ROI might be a simple calculation, it is not a straightforward measure and the conditions under which we apply ROI and the assumptions we use to measure it are laden with value. If you are inclined to dispute this, revisit the two scenarios at the top of this post, and ask yourself what the use of ROI, or lack thereof, suggests about what is valued in our current health care system. Even the requirement of a certification of actuarial value for the diabetes program emphasizes savings over health as a precondition for preventive services (i.e., the prevention program had to demonstrate not just that people are healthier, but that short-term savings offset the cost of the program).
Together we need to shape new norms around what constitutes reasonable evidence, and for what purpose. To start, I’d like to make two suppositions:
- In the case of evidence-based practices, that is, where robust evidence exists about ROI, no further demonstration of ROI should be required of those seeking funding. To require further “proof” will hamstring the take up of evidence-based population health measures, and thus delay advances in health. And there is plenty of such evidence. Check out this fabulous compilation from the Washington State Institute for Public Policy (WSIPP) that quantifies the costs and benefits of numerous interventions. Or see for example, Sonoma County’s ambitious Upstream Investments, County Roadmaps and Rankings What Works for Health, or the CDC’s Community Health Improvement Navigator. After all, if an investor refuses to be convinced by the robust evidence produced by trained researchers, then there’s probably no one-off ROI calculation that would suffice, either.
- We should strive to hold non-clinical population health efforts and clinical treatments to the same burden of proof. For example, the DPP studied the impacts of both lifestyle interventions and pharmaceuticals. The research found the non-clinical interventions to be more effective: participants in the lifestyle intervention group reduced their risk of developing diabetes by 58 percent, compared to those taking Metformin, which reduced their risk of developing diabetes by 31 percent. Furthermore, DPP participants in the lifestyle intervention group without metabolic syndrome at the beginning of the study were less likely to develop it than those on Metformin. Additionally, high blood pressure decreased in the lifestyle intervention group but increased in the Metformin group. Yet the DPP program of lifestyle interventions had to demonstrate both health impact and cost savings — pharmaceuticals need only demonstrate “no harm” and some level of effectiveness. The double standard means that we are probably failing to optimize our investments in health. It also favors the status quo (i.e., drugs and devices) over alternative methods to improve health.
At this point, the little voice in your head may be saying (or screaming), “but, but, but . . .” Well, so is the voice in my head; there’s so much to think about. ROI is a topic that RTH will continue to work on in the coming months.
 For example, as is the case with social impact bonds, or shared savings with ACOs.