PrintIn my last two posts, about data points that really matter, an implicit assumption is that the data in question (house sales and auto sales) actually drives decisions. Surprisingly, this hasn’t always been the case in many fields. Until fairly recently, many decisions have been largely based on expectations, plausibility, and bias.
Medicine is a great example. I don’t mean to pick on health care—after all, it’s the source of the current gold standard for effectiveness studies, the double-blind clinical trial—but think about the current debate on medical costs. One of the principal arguments for the availability of cost reductions is that no one really knows which treatments are most effective for many problems. Surgery rates for back pain, for example, vary widely among regions—which wouldn’t be the case, I hope, if there were one clearly superior solution. An entire industry, pharmaceuticals, is built around rolling out new treatments that, in many cases, offer little measurable benefit over existing treatments, at a much higher cost. Doctors hold onto their right to practice as they please, without regard for studies of industry best practices. The fact is, in many areas of medicine, we really don’t know what works best and why.
Education is another great example. Over the past several decades, multiple pedagogical models have been rolled out in different subjects, with little objective evidence to back them up. My own state, Massachusetts, introduced a set of reforms 20 years ago, to much controversy. Here, the reforms worked; Massachusetts is now first in the nation in math, and second in the world. But even two decades later, no one really agrees on why. The debate, then and now, is driven by politics more than by data.
My own fields certainly aren’t immune from this. In investing, analysts such as Ed Easterling of Crestmont Research focus on trying to determine what works and when. Jim O’Shaughnessy wrote a groundbreaking book, What Works on Wall Street, which I recommend. My own work has focused on this area as well, but this type of analysis has only really started to hit in the past decade or so. What makes analysts like Easterling and O’Shaughnessy stand out is that they’re the exceptions, not the norm.
As for economics, there’s tremendous debate on almost every topic that touches everyday life. Something as simple as the effect of tax increases or changes in the minimum wage generates massive disagreements. Despite huge amounts of data, we really don’t know yet what works and why.
Two things are needed for this to change. The first is that the data needs to be available, which is happening more and more. Data collection in medicine, economics, education, and many other areas is advancing on a daily basis, and that data is being used in decision making. Availability of data is driving improvements.
Second, the data needs to be parsed in a way that generates useful results, and this is where we are right now. There’s an analytical component to this, but, ultimately, it’s value driven. A good example is climate science. Whether you agree that global warming exists or not—and even many of the critics are coming around as the evidence mounts—what to do about it remains value driven. Per point one, we have been and are developing the data set needed. Per point two, we’re now arguing about what it means and what to do. Medicine is also in stage two in many areas.
It will be at the intersection of interpretation and action that the biggest fights occur. If you want to understand the debates of the future in a meaningful way, study statistics. That is also good advice right now. Increasingly, policy decisions will be based on interpretation of data sets, meaning that political fights will become increasingly data based. This won’t necessarily equate to better decisions, but it should mean we can better evaluate the results of those decisions and correct them as needed.
