Lift and Shift Rarely Works

Rethink your data strategy instead

When migrating an existing platform it’s very tempting to simply “lift and shift” the existing data assets into the new system as quickly as possible. This means you’re copying data and queries “as is” and focusing on getting them working with as little modifications as possible.

I’ve seen his play out multiple times in my career but it rarely works long-term for two very important reasons:

1. Yes you move quickly but all the tech debt and inadequate architecture design is carried forward into the new system. When you lift and shift you “copy / paste” the old system into a new environment. Eventually cracks will appear.

2. You miss out on the opportunity to redesign the data architecture properly. Do you know the expression “never let a good crisis go to waste?” It applies here. A redesign gives you a chance to clean house and rethink your approach.

Take the time to analyze the existing architecture, figure out what’s working and what needs to change, then come up with a plan where you can both update the design and deliver value incrementally.

My good friend

Aaron Ormiston

calls this method “steel threads” whereby you take a horizontal slice of your existing architecture, from source system to end product (e.g dashboard) and deliver that in the new platform.

The beauty of this technique is that you not only deliver value incrementally and quickly, but you also validate your work by comparing key metrics in the two systems.

These steel threads make a lot of sense to the business than vertical slices which only make sense to engineers (build the ETL first, the data model next, the metrics next and finally all the dashboards)

Rethinking your data strategy is not for the faint of heart though. You need confidence, political capital and experience to convince stakeholders to give you a big budget with low initial ROI.

I know at least one person who is building a whole new data platform using open source technologies like Iceberg, Airbyte, etc. who pitched the full data strategy to their stakeholders and secured a nice budget. Usually this is a recipe for instant rejection.

The new platform they proposed will set the business up for near infinite data growth and easily swappable components without worrying about investing in proprietary technologies and vendor lock-in. Maybe I can get them to write something about it.

A strategy that builds for the future, especially one utilizing open source technologies will look like a bad investment early on. You’re spending additional capital on top of the current platform.

Hence why “steel threads” methodology is so powerful. You can compare the costs of delivering each dashboard with the new system vs the current one and show the potential for reducing costs or improving efficiency.

Thats it for this issue. I have at least a couple of in depth use cases coming up that show you how different companies are implementing and using metric trees. Both of them will have a visual demo.

Until then.