The disillusionment of data careers

Why analytics can be a frustrating career and what we can do about it.

Welcome to the latest issue of the data patterns newsletter. If this is your first one, you’ll find all previous issues in the substack archive.

In this edition we’ll talk more about the metrics playbook and how it can help revive the careers of disillusioned data scientists.

Yesterday I posted a tweet that got an unexpected level of engagement.

Ergest Xheblati @ergestx

This is the real crux of the problem in the data field. We can about data accuracy, data monitoring, data modeling, ELT vs ETL, etc. till the cows come home. At the end of the day, all those problems are symptoms of this one.

This isn’t the first time I’ve seen this sentiment; data scientists frustrated that executives often ignore their findings if they go against their intuition.

There’s even a trend of disillusioned data scientists turning to data engineering as an alternative career.

Joe Reis is the author of the book “Principles of Data Engineering” which has proven to be quite popular with people on the field. His phrase “recovering data scientist” succinctly captures the heart of the abandonment zeitgeist.

So what’s happening here? How did data science go from “the sexiest job of the 21st century” to disillusionment?

To fully understand the problem we need to deconstruct how data driven decisions are made and see if there’s a way to improve them. But first let’s dive into a bit of cognitive science.

Confirmation bias or expert intuition?

Confirmation bias is our tendency to choose only information that validates our existing beliefs and disregard information that disproves them. It would be very easy to blame executives for being biased, and say that decisions are political vs data driven. To be fair, that does happen but that’s not where the story ends.

Expert intuition is the ability to recognize patterns instantly and know exactly what’s going on. An expert can immediately tell if your insights make sense or not. It’s a hard-earned skill that requires a lot of exposure to a large variety of situations, problems and solutions.

On the surface it looks very much like confirmation bias, because it is! Here’s the rub though.

We cannot tell the difference between the intuition of an expert or that of a novice. Both will have the same exact feelings of them being right and you being wrong. Both will have an intuition.

The difference is experience.

You as a data professional cannot match the vast experience of an executive with a decade of experience in their field. So it’s quite natural why they cannot trust your insights if they go against their intuition.

Insights vs Metrics

That’s why the focus on insights to drive decisions on how to improve the business is misguided. Insights are best thought of as the output of cutting edge research. They are useful for exploring new markets and ideas but are often unreliable as directives for what to do to improve the business.

Why? Just like any cutting edge research the findings are unproven. They would require additional investment in experimentation to see if they hold any water, and in a climate of cost cutting that’s out of the question. Once they’re proven they become best practices but until then they’re still untested hypotheses.

So what should you focus on instead?

Imagine for a second that there existed a playbook of already proven insights and best practices that identified the key metrics each area of business should focus on. For example let’s say that in marketing the key metrics to focus on were customer acquisition cost (CAC) and conversion rate (CVR).

As a data professional you could walk into a business and use the playbook as a checklist to figure out what metrics should be tracked, get buy-in from the executive board and then build them. That’s level one.

Next you can start doing research into what factors influence these metrics and come up with levers executives can pull to drive CAC down and CVR up. I bet that would be a very satisfying experience for any data scientist. That’s level two.

Furthermore you can do research into the metrics themselves and create new ones that better reflect the natural evolution of the business processes. That would also make for a satisfying career.

There are a few prerequisites. It needs to start from the top. The executive board has to establish a culture of reviewing metrics weekly or monthly. This will give executives a “fingertip feel” for the ebbs and flows of the business. It will sharpen their intuition and improve their decisions.

This will necessitate consistent and accurate metrics which will drive the need for accurate data. Eventually this will trickle down to engineering teams and ensure everything is tracked correctly.

That’s my vision for the metrics playbook project and why I’m so excited to build it.

I’ll write more about this topic in upcoming newsletters.

Until next time.