The cost-benefit revolution is far from over

Want to get smarter about biotech clinical trials, R&D strategy, and financial modeling? Join the thousands of biopharma execs and professionals who subscribe to our free email newsletter. Read a sample issue, and then sign up here.

(This post originally appeared on Forbes.com on September 16, 2018.)

If you’re wondering what legal scholar and former “regulatory czar” Cass Sunstein thinks about cost-benefit analysis, look no further than the bold-type teaser on the back of his new book, The Cost-Benefit Revolution (MIT Press, 2018): “Why policies should be based on careful consideration of their costs and benefits rather than intuition, popular opinion, interest groups, and anecdotes.”

It’s hard to argue against the idea that reason should trump rhetoric in informing complex decisions – but despite its title, this clear, well-argued book is far from a victory lap. Yes, Sunstein believes in the power and benefits of cost-benefit analysis, and proudly makes the case for their positive contribution to government rule-making. But he’s also clear-eyed about many of the persistent challenges. In Sunstein’s analysis, the cost-benefit revolution has made great strides, but there’s much more work to be done.

The history of Federal cost-benefit studies begins in 1981, when President Reagan established their key principles: the requirement to conduct data-driven analyses of proposed new rules, the need for the benefits to outweigh the costs, and the requirement for transparent documentation. These tenets have persisted largely unchanged since then, notwithstanding subtle modifications by Presidents Clinton and Obama, and somewhat more substantive ones by President Trump. (At several points, Sunstein discusses President Trump’s Executive Order establishing the “one in, two out” dictum and prohibiting new rules from imposing additional net costs, but contemporary politics is not his main focus.)

As a statement of policy and philosophy, Sunstein argues, cost-benefit analysis has endured because of the core principles it reflects: Federal agencies should only be able to implement new regulations if they improve people’s lives, and the best way to determine if they’ll do so is to quantify their likely effects. Paraphrasing and interpreting comments of President Obama, Sunstein says that Americans generally “want their government to make its decisions on the basis of what regulations and policies would actually achieve. Cost-benefit analysis is a way of finding out.”

So, almost 40 years later, how well is cost-benefit analysis performing? It’s harder to answer this question than one might think. Part of the challenge is these studies are generally not intended to act as go/no-go gates for proposed rules; per Sunstein, they should serve as “a spur and a prod, not merely a check and a veto.” Thus, despite Sunstein’s assertion that cost-benefit analysis has “often” been a key driver in “significant cases”, it’s almost impossible to find clear success stories where one of these studies tipped the balance on a new rule from approval to rejection or vice versa.

On the other hand, there’s no doubt that cost-benefit analysis has forced rule-makers to be much more rigorous and transparent about articulating who we think will gain and lose, and by how much, from proposed regulations. When we consider imposing new regulations on companies to reduce death and illness from pollution, cost-benefit analysis helps us calculate the “value” of those rules by considering the amount that individuals would pay to avoid, say, a 1 in 100,000 risk of death from chemical-laden water. It also helps us combat what Sunstein refers to as “systems neglect,” in which we tend to focus on isolated parts of problems without considering their further-reaching effects; an individual may be willing to pay some cost to avoid the adverse effects of polluted water, for example, but her friends and families might pay even more.

And in fact, maybe it’s a good thing that cost-benefit analysis is rarely decisive in decision-making – because it still suffers from significant shortcomings, despite decades of refinement. Sunstein notes that standard approaches fail to adequately measure effects on social welfare in many scenarios, such as those that selectively and disproportionately affect the elderly or the very young, or those that cause significant hedonistic benefits (increased ease or convenience) or negative emotional consequences (grief or anguish). We are also still learning how to best apply cost-benefit analysis in many complex contemporary areas of regulation such as mandatory labeling, privacy and national security, and free speech, which Sunstein highlights as examples of “frontiers” for further study and methodologic improvement.

In addition, although cost-benefit analysis helps combat systems neglect, it still leaves open how widely to define the “system”. (What of the owners, employees, customers, and investors of companies that might be affected by a new pollution control rule? How much weight should their interests receive compared to those of individuals most directly affected – or should they even be counted at all?) Sunstein observes repeatedly that cost-benefit analysis shows its limitations when one tries to apply it to “hard cases” – but many (most?) real-world scenarios are hard when one looks slightly beneath the surface.

In considering ways to improve cost-benefit analysis, Sunstein devotes particular attention to what he calls the “knowledge problem” – the challenge of obtaining adequate data to establish if a planned rule would increase social welfare across all affected parties. He suggests four reforms: improving the “notice-and-comment” process to obtain broader views on the likely impact of proposed regulations; retrospectively analyzing rules to determine how well reality agrees with intentions; conducting randomized controlled trials in policy-making to prospectively test the effects of new rules; and instituting “measure and react” protocols to obtain real-time data on the consequences of new regulations and adjust accordingly. These are all thought-provoking ideas, albeit with varying degrees of complexity and implementation challenges, which warrant consideration and wider implementation.

But these technical solutions may not solve a larger problem rooted in human psychology: how can we best help decision-makers use these complex models to make what are often binary decisions? As noted by risk modeling expert Sam Savage, in most organizations, “[e]xecutives’ desire to work with “a number,” to plug in an average figure, is legendary,” particularly in high-uncertainty situations. Sunstein would likely argue that in all but the simplest cases, the “number” is far less important than the insights gleaned from the analytic approach. But what should a harried decision-maker suffering from information overload do with the information that the projected benefits of a proposed pollution control rule just barely offset its projected costs, or vice versa? Given his vast experience, it would be helpful to hear more details from Sunstein about his successes and lessons learned in helping senior leaders look past top-line results as they incorporate cost-benefit studies into policy decisions.

Cost-benefit analysis faces tough challenges, without easy solutions – but Sunstein is appropriate to indulge in some celebration, even while acknowledging the work that lies ahead. The late British statistician George Box’s observation that “all models are wrong, but some are useful” certainly applies to these studies that, despite their limitations, have brought a much-needed dose of analytic rigor to regulatory decisions that would otherwise be ruled by emotion and anecdotes. And as Sunstein notes, cost-benefit models have become progressively less wrong and more useful over time, and will surely continue to improve. The cost-benefit revolution may be incomplete and its pace of progress uncertain, but it’s far from over. ¡Viva la revolución!

Thanks to MIT Press for providing me with a complimentary review copy of this book.