Why Focusing on Answering Questions Dooms Data Teams

The real purpose of analytics

I wrote a post recently on LinkedIn and X (aka Twitter) that got a lot of engagement. I was both surprised and delighted, because the inspiration for that post came seemingly out of nowhere. It was a synthesis of ideas I had been thinking about for some time.

So for today’s newsletter, I decided to polish up that post and send it. I’ll add more nuance here as the post was a bit of a “hot take.”

Ready? Here we go!

Any and all analytics focused on just answering questions or building dashboards is doomed to becoming a low value cost center in the organization, which eventually leads to the dreaded "what's the ROI of the data team?" question.

Why? Let me show you.

Let’s assume you are a smart data leader. You've learned that to show value, you must fulfill the needs of the organization. Once you walk in, you sense a huge need for "data" which comes in the form of lots of different questions. They could be questions for specific numbers, questions about overall performance or demand for insights.

Just about any smart data person will do one of the following:

1. They will start directly answering questions by writing SQL queries on top of operational databases. Eventually they’ll build a few dashboards that answer the most common questions and showcase key metrics. (KPIs) Eventually they’ll demand more resources to build a data warehouse and staff a larger team capable of answering more complicated questions.

2. They will start by demanding resources to build (or buy) a system capable of answering these questions (such as a data warehouse) with simpler queries by modeling raw data into simple structures and then passing on the query writing to the question askers. This is the so-called "self-service" model.

Both these projects take time, but you can use agile methods and deliver value incrementally. Great! You think this will help solve the initial problems, and to some extent they do, but soon enough something interesting (and quite inevitable) happens.

What seemed like an insatiable need for answers and insights never quite stopped, in fact it got worse. The questions keep coming but the quicker you answer them, the more complicated they become. Meanwhile you’re several hundred thousand dollars into the project to build a system and staff a team of data analysts.

What the hell is going on here?

Are you familiar with the term "induced demand?" It's a phenomenon that's well known in transportation whereby widening a highway by a lane or more to solve traffic congestion inevitably leads to more cars on the road which leads to more traffic congestion making the problem worse.

Here’s a quote by Robert Caro in the Power Broker:

During the last two or three years before [the entrance of the United States into World War II], a few planners had ... begun to understand that, without a balanced system [of transportation], roads would not only not alleviate transportation congestion but would aggravate it. Watching Moses open the Triborough Bridge to ease congestion on the Queensborough Bridge, open the Bronx-Whitestone Bridge to ease congestion on the Triborough Bridge and then watching traffic counts on all three bridges mount until all three were as congested as one had been before,

That's exactly what's happening.

Now that you have spent 9 months and hundreds of thousands of dollars to build this beautiful system for answering questions, you’ll not only get more questions but even more exotic questions.

Questions you couldn't have predicted. Questions that your intricate system cannot answer directly. So to answer them what do you do? You load even more data in your data warehouse, add more tools, build more data marts and dashboards.

More data will lead to more complex and more fragile data systems which will lead to increased cost. Meanwhile the questions never stopped and soon enough you’ll be face with the dreaded "what's the ROI of the data team?" question.

You might argue that this was “pent up demand” that people always had those questions in mind and once you were able to answer the simple ones, there was finall room to ask the complicated ones, but I beg to differ.

The problem isn’t the questions. In fact, questions are essential. The problem is a few key fundamental incorrect assumption about the needs of the organization.

The first incorrect assumption is that the purpose of the data team lies in answering questions and providing insights. It seems really obvious and could even be in your job description, but as soon as you make this assumption, you lose.

Why?

Not all questions are created equally. There’s a huge difference between a question that leads to a decision that improves the company’s bottom line and a question that’s a mere curiosity. But if you can’t tell the difference you’re in for a world of pain.

The second incorrect assumption is that stakeholders know exactly what they want when they ask these questions. After all, they might have years of experience in their domain, so you have no reason to believe otherwise.

But that’s not always the case. Oftentimes these questions arise from incomplete or even incorrect mental models about what really works to drive performance. It doesn’t mean they don’t know what they’re doing. Just think about the baseball scouts in Moneyball.

So what's the alternative?

The correct assumption is that the purpose of analytics isn't to answer questions but to improve operational performance. Questions will definitely be a part of that, but the change in focus will make all the difference.

As a data leader, you have quite the luxury of being simultaneously outside many of the key functions in the organization while also working closely with all of them. This gives you unparalleled “birds eye view” of the entire value chain.

As such your contributions are vital to improving operational performance but you have to start with the right aim and understand the goal of that organization. If you have the wrong focus, you'll end up dooming your efforts and your team.

I'll write more about this in the future, so stay tuned. Until next time.