Something hit me recently. Across two decades of work in data, there’s a single thread that permeates through. We’ve called it “data cleaning”, “data preparation” or even “data modeling” but at the end of the day it’s still a form ot data transformation, aka changing the shape of data from A to B while adding or clarifying meaning to it.
Back in the olden days (like a decade ago 😁) we used to do all this by hand. Sample a few rows, look for outliers, make assumptions about the underlying processes, add checks, write code to convert strange formats, and so on.
It was tedious!
So tedious fn fact that we used to say that “80% of data work is preparation.”
Well, that hasn’t changed. It’s still a problem, still a limitation. And it’s more relevant these days. What makes it a tough problem is that you never quite know the underlying function that generated this data. So you often have to make assumptions that could break at any moment.
Data transformation shows up everywhere, even when you don’t think about it that way. For example an accountant taking messy transactional data and separating credit and debit line items in order to illuminate profit and loss for a company is still data transformation. In fact, I consider accountants just as much data professionals as anybody in our field.
I’ve been doing more AI work recently. Just a few Claude skills for now; but some interesting stuff queued up. But I did notice something quite interesting. At the heart of all of it still lies our old friend data transformation!
It doesn’t matter if you are scraping a website, extracting data based on a JSON schema and then rendering an HTML report or if you’re running a SQL query, looking for certain signals, composing an email and triggering a send or even if you’re working in Excel and you’re manipulating a large spreadsheet by cleaning things up, adding columns based on formulas, pivoting, and so on.
And so I’ve been wondering whether it’s possible to automate such manual work. Sure you might be able to automate the actual code writing part. You can tell the AI agent to “add a column that strips out all extraneous characters from customer_name and puts them in proper case” and more than likely you’ll get exactly what you asked. Parsing an HTML page into a structured format, which used to take hours, if not days to do manually, is now table stakes for most AI systems.
But that’s not true automation now is it? The AI system isn’t making any judgement calls about the data. I’m not saying that it can’t, and therefore your job is safe from automation. Au contraire….I’m asking what if we could do it? What if we could automate all that manual work?
Do I think this is possible? Should we even try?
Given that it’s such a fundamental constraint of our modern digital world, I believe we should try. But does that make us (data professionals) redundant? irrelevant? old rusty cogs to be replaced with fancy new ones?
NO! If we can reduce the amount of manual data transformations we do—even if we’re typing prompts instead of Pandas code—it will help us transition into far more interesting, more important, more impactful work.
We can now focus on system design, where the value of what we do is more easily realized. We can focus on understanding the business, finding key drivers of growth and work on far more impactful things. It may not be called “data engineering” or “data science” but it still has those things at its core. (for the record I prefer Data Solutions Architect)
The future is bright for data work!
(no AI was used in the writing of this essay, because I still want to have a brain in the future)