Does a CEO’s gut instinct matter in a world that runs on digital data-crunching?

Over the last decade, big data analytics has risen to the forefront as a tool to facilitate smarter, faster, and more effective decision-making in business. Its meteoric ascension into corporate boardrooms and executive offices has displaced our conventional ideas about the value of an individual leader’s savvy. Today, a leader who so much as mentions intuition during a strategy meeting is likely to prompt a few skeptical looks and a polite variation of: “Okay, but what does the data say?”

Gut instinct is an outdated faux pas in modern business culture — an idea that might have backed CEO icons in the past, but hardly suits our understanding of leadership craft now.

As one writer for the Harvard Business School Online summarizes in an article, “The concept of intuition has become so romanticized in modern life that it’s now a part of how many people talk about and understand the “geniuses” of our generation. Though intuition can be a helpful tool, it would be a mistake to base all decisions around a mere gut feeling. While intuition can provide a hunch or spark that starts you down a particular path, it’s through data that you verify, understand, and quantify.”

And, in the writer’s defense, data is a crucial aspect of business strategy. With big data analytics, organizations can use the troves of information that they collect to identify new business opportunities, better understand their customers, improve marketing and sales, improve operational efficiency, and boost profits, among other gains.

According to a 2019 whitepaper published by New Vantage, a full 62 percent of surveyed businesses have already experienced measurable results from big data and AI investments. Analysts for the International Institute for Analytics take these findings a step further; they estimate that businesses using big data will see a collective $430 billion in productivity benefits over their non-data-reliant competitors by the close of 2020.

Given these findings, it’s no surprise that 88 percent of surveyed organizations feel an urgency to invest in big data and AI, nor that 92 percent of those who do are motivated by a desire for digital transformation, agility, and competitive advantage. According to New Vantage data, 55 percent of surveyed businesses spend over $50 million on big data and AI initiatives, and 21 percent are spending over half a billion on them. 

But is our appreciation of big data at the expense of intuition poisoning our decisions with blind faith?

As leaders, we need to ensure that we aren’t allowing ourselves to be steamrolled by raw data. The truth is, data applied blindly or without context is worse than useless — it can be outright misleading.

Consider the issue of cherry-picking data as an example. With large data sets, it is all too common to find links that appear to be legitimate conclusions but are, in fact, due to fake statistical relationships. As tech writer Nassim Taleb once concluded in an article for Wired, “in large data sets, large deviations are vastly more attributable to variance (or noise) than to information (or signal).”

For a leader who is both untrained in statistics and overly confident in the power of data, a glance over gathered data could inadvertently lead to cherry-picking (i.e., selectively choosing) these apparent relationships to build faulty conclusions. The consequences of following this statistical misdirection can be, as you might imagine, disastrous. 

Then, we have the issue of bias in algorithms. Research has repeatedly demonstrated that, for all our insistence that data is “objective” and thus immune to faulty human opinions, algorithms can be just as biased — if not more so — than human analysts.

As Dr. Nicol Turner-Lee once explained in an article for the Brookings Institute, “In machine learning, algorithms rely on multiple data sets or training data, that specifies what the correct outputs are for some people or objects. From that training data, it then learns a model that can be applied to other people or objects and make predictions about what the correct outputs should be for them.”

The result, she writes, is that “some algorithms run the risk of replicating and even amplifying human biases, particularly those affecting protected groups.” For all that the business world puts data on a pedestal, these points make it abundantly clear that we cannot blindly rely on its findings to guide our decision-making. 

This isn’t to say that data-driven decision-making isn’t useful — it is. However, data analytics tools must be implemented within a context of training and thoughtful consideration. To borrow a quote from New York Times contributor Robert J. Moore, “Obsessing over tests and metrics can be counterproductive if it prevents you from thinking about aspects of your vision that can’t be quantified.”

We seem to have forgotten in the hype that data is meant to inform our human intuition, not overpower it. 

In recent years, many executives have realized that the most significant barrier they face in creating data-driven organizations isn’t the technology, but the people who should be using it. One recent study published in Sustainability found that nearly half (48.5 percent) of U.S. executives polled in 2018 cited “people challenges” as their foremost concern in creating a data culture.

The problem appears to be a lack of personnel training and support. 

“Managers need to wake up to the fact that their data investments are providing limited returns because their organization is underinvested in understanding the information,” business researchers Shvetank Shah, Andrew Horne, and Jaime Capellá wrote in an article for the Harvard Business Review. “Companies that want to make better use of the data they gather should focus on two things: training workers to increase their data literacy and more efficiently incorporate information into decision making, and giving those workers the right tools.”

As business decision-makers, we need to stop dismissing intuition as an outdated strategic compass and take a more informed approach to our data strategy. Leaders need to seek balance at work and remember that while data should guide their path, it should not quash their instincts.