Very nicely laid out. Though I must admit I'm not convinced that the root is data vs intuition. But I'm also not quite ready to refute it either.
This conclusion seems to find its ultimate focus on a tension internal to an individual and presumes that the individual has full agency and autonomy. That may hold in small organizations (where I'm often employed) but may not in larger organizations (where I often consult).
The DevOps example is a nice contrast because it easily roles up to the organizational level, even though several of its parameters can be applied at the individual level (work, time). So while data has the flexibility of application to both individual and organization, I'm not sure that intuition does.
As a final thought, I think there may be another dimension, or alternative expression of intuition. Perhaps it is rigor, or structure, of decision-making. The most structured expression of decision-making would be a coded algorithm which is completely deterministic in its results. The less rigorous end of the spectrum would be decisions based on emotional or other whims, nearly indistinguishable random dice roles or monkeys with keyboards.
I find that over time organizations tend to get more rigorous in their approach to decision-making, as they layer on processes and requirements in an effort to better manage risks and variations. Perhaps this is the basic contract with the concept of intuition.
I've battled with this for months and I'm also not 100% convinced it's intuition vs data analysis. I think it's part of it for sure but I haven't found anything else that comes this close. I keep wondering: "do organizations even want to make rigorous data driven decisions?" I know it will benefit them for sure but I don't know they don't do it.
I find it a reasonable simplification to make after the main argument of root cause analysis and how you've approached it.
Intuition and culture are somewhat interchangeable in this context, as both derive action from a set of inherited practices, symbols and social roles, rather than some logical or analytical effort.
As we've seen in the past decade or so, some of the most relevant paradigm shifts attempting to address this "tech vs business" conflict like DevOps, SRE and Data Mesh/Contracts have concerned themselves with bringing more technical/formal analysis to business process using business critical concepts to leverage it's relevance through a language more familiar to the "business" culture.
DevOps/SRE focusing on user satisfaction as a core indicator of reliability.
Data Mesh/Contracts (for the lack of better terms) focusing on shared ownership as a core indicator of coherence.
(I'm not pleased with how i'm expressing it.. please forgive the lack of better concepts)
I'd also add that the data industry accidentaly shot itself in the foot with the advent of mainstream AI, beacuse it has led to a market ideal that business leaders should be AI driven when they haven't yet become Analytics driven..
So instead of focusing on addressing the core issues you've pointed, the AI/Data hype has been adding fuel to the fire and just making business leaders expectations diverge even more from reality and the core needs.
Very nicely laid out. Though I must admit I'm not convinced that the root is data vs intuition. But I'm also not quite ready to refute it either.
This conclusion seems to find its ultimate focus on a tension internal to an individual and presumes that the individual has full agency and autonomy. That may hold in small organizations (where I'm often employed) but may not in larger organizations (where I often consult).
The DevOps example is a nice contrast because it easily roles up to the organizational level, even though several of its parameters can be applied at the individual level (work, time). So while data has the flexibility of application to both individual and organization, I'm not sure that intuition does.
As a final thought, I think there may be another dimension, or alternative expression of intuition. Perhaps it is rigor, or structure, of decision-making. The most structured expression of decision-making would be a coded algorithm which is completely deterministic in its results. The less rigorous end of the spectrum would be decisions based on emotional or other whims, nearly indistinguishable random dice roles or monkeys with keyboards.
I find that over time organizations tend to get more rigorous in their approach to decision-making, as they layer on processes and requirements in an effort to better manage risks and variations. Perhaps this is the basic contract with the concept of intuition.
I've battled with this for months and I'm also not 100% convinced it's intuition vs data analysis. I think it's part of it for sure but I haven't found anything else that comes this close. I keep wondering: "do organizations even want to make rigorous data driven decisions?" I know it will benefit them for sure but I don't know they don't do it.
I find it a reasonable simplification to make after the main argument of root cause analysis and how you've approached it.
Intuition and culture are somewhat interchangeable in this context, as both derive action from a set of inherited practices, symbols and social roles, rather than some logical or analytical effort.
As we've seen in the past decade or so, some of the most relevant paradigm shifts attempting to address this "tech vs business" conflict like DevOps, SRE and Data Mesh/Contracts have concerned themselves with bringing more technical/formal analysis to business process using business critical concepts to leverage it's relevance through a language more familiar to the "business" culture.
DevOps/SRE focusing on user satisfaction as a core indicator of reliability.
Data Mesh/Contracts (for the lack of better terms) focusing on shared ownership as a core indicator of coherence.
(I'm not pleased with how i'm expressing it.. please forgive the lack of better concepts)
I'd also add that the data industry accidentaly shot itself in the foot with the advent of mainstream AI, beacuse it has led to a market ideal that business leaders should be AI driven when they haven't yet become Analytics driven..
So instead of focusing on addressing the core issues you've pointed, the AI/Data hype has been adding fuel to the fire and just making business leaders expectations diverge even more from reality and the core needs.
I love this.
Geres my conflict cloud for data:
I want to succeed
I need to make good decisions rooted in reality
I should find the truth
I need to play well with others and look good
I should control the truth
You can't both look for the truth in earnest as a scientist may do because you must play along with others in the organization.
As Bezos put it : "we are not really truth-seeking in nature, we're social"
Problems persist because this conflict is inherent in decision-makers trying to collaborate.
Thanks for writing this.
"You need the methodology to be imposed from the top which means you have to get buy-in from the C-suite."
How do you recommend data teams get this buy-in? Or is it mostly out of their control?
You basically have to pitch the CEO or CFO on the problems and benefits
I like how thorough this analysis is
“Such a methodology does exist! Amazon has been using it for years. It’s made up of three components:“
Do you mean the Weekly Business Review by that?
Yes that’s the one