Learn Things That Don't Change

Professional development through Lindy technology and timeless principles

In the last issue of Data Patterns we talked about some of my favorite high level principles to help you build a long lasting career in data. Today I want to take one of those principles and break it down further so you can apply it to your career. Interested?

Let’s do it!

Focus on things that don’t change

Jeff Bezos is famous for saying that you should build your business on things that won’t change in the next decade. The same applies to resilient careers. Would you risk your career learning things that could become obsolete in the future? I wouldn’t. Technological progress will only get faster, tools and techniques will continue to evolve. If all you focus on is tools you’ll have to keep learning new ones just to keep up. But if you focus on evergreen patterns and concepts you’ll continue to thrive.

I used to think that advancing my career meant learning new technologies like Machine Learning, Snowflake, Databricks, etc. There’s a good reason for this. When looking to hire someone companies seem to prefer candidates that have experience with newer technologies. In fact your resume might not even make it through automated screening mechanisms if you don’t mention certain keywords. And hiring managers are biased towards them.

But focusing on new technologies is a flawed way to think about professional development. The biggest reason why is because they become obsolete at a very fast rate. You could end up spending thousands of dollars to learn a new technology only for it to become irrelevant in 3-5 years. Would you want to base your career on such shaky foundations?

Just look at Apache Hadoop. It was a fantastic idea (and all the rage) 10 years ago, at the peak of the “big data” boom. Sure, if you had learned enough of it to be useful and get a job you would have thrived, but it won’t help you today.

On new technologies

New technologies evolve to solve novel problems faced by tech giants. In Hadoop’s case it was developed by tech giant Yahoo to mimic another tech giant’s technology (Google’s GFS). These companies faced the problem of having to process gigabytes of data per second without using expensive hardware.

And that’s the central point of my argument. The problems of tech giants are not necessarily the same as the ones many companies deal with. Outside of big tech, who else needs to process that much data? Even big retailers can get by with traditional RDBMSes that can process millions of transactions per second.

But for some reason people love to jump on the hype bandwagon. When they do, they drag the rest of the unsuspecting masses with them. Often this is done on purpose to promote a company’s new technology so it will become commonplace. Keep that in mind next time you’re thinking about jumping on the latest trends.

---

Side note: For an illuminating read on the dangers of choosing a new technology check out this 3-part series about MongoDB. Many of those companies had to go through a painful period of transitioning their entire codebase to more traditional RDBMSes like PostgreSQL after failing to see meaningful benefits from MongoDB.

New technologies are unproven. They haven’t gone through the trials and failures of being implemented by thousands of companies. There are no communities with large knowledge bases that can help you if you got stuck. They just haven’t solved enough problems and come through triumphant.

On top of all that, new technologies are in constant competition with existing technologies and organizational inertia. The long tail of companies that still rely on old technology is longer than you think. For example I’ve worked in non-tech companies that still have functioning data warehouses built on top of Microsoft SQL Server and many have no plans of transitioning to the cloud. You can still have a good career working there.

On old (Lindy) technologies

Old technologies that are still around are Lindy. Uhm what’s a Lindy technology Ergest?

In an article published in 1964, Albert Goldman wrote about New York comedians that used to meet up every night at Lindy’s Deli and discuss running shows [1]. These discussions led to an observation which Goldman called “Lindy’s Law”, and which can be formulated as follows: the future lifetime of a running show is proportional, on average, to its present age—i.e., the duration it is already performing. Hence, the longer a show has been running, the longer it is expected to continue to run in the future. Source

Lindy’s Law applied to technology says that the longer a technology has been around, the longer it’s expected to be around.

Relational databases and SQL have been around for decades. Statistics has been around for centuries. They have been applied to hundreds of thousands of problems over the years, have a long track record of successes and a growing knowledge base. If you have a question or problem, it’s likely been faced before and solved.

They have stood the test of time and continue to thrive. Why? Because they work and they solve problems. As long as the problems they solve don’t change, the technologies won’t change. And the coolest part is that these Lindy technologies continue to evolve in order to solve new problems (like storing JSON data in RDBMS tables).

Let me illustrate with an example.

I started my career working with SQL Server 6.0. Yes it was old even back then, but the application was built in the 1990’s so what choice did I have? As I wrote production queries (stored procedures and views) I often needed to optimize them for speed. So I had to to learn about table indexes and performance tuning.

Later I got jobs at a couple of tech companies. Their infrastructure was in the cloud (Snowflake) and the specifics of table indexes no longer applied BUT the ideas behind indexes (like table statistics) and how they help optimize queries were still invaluable. These days I work with PostgreSQL. My knowledge of table indexes and performance tuning is still valuable a decade later.

Hopefully you get the point. New technologies are cool but the fundamentals rarely change. When you study them, you have a transferable set of skills that will continue to make you valuable in the job marketplace.

Here’s a list of Lindy ideas, patterns and technologies that will never go out of style:

• SQL

• Relational model

• Relational databases

• Query optimization patterns

• Data transformation patterns

• Extracting data from various sources

• Loading data in storage efficiently

• Moving data efficiently between locations

• Workflow orchestration

• Data quality and monitoring

• Data system architecture and design

• etc…

Are you interested in learning more about these ideas? I’m thinking about creating a library of timeless patterns from all of todays technologies that will help you advance your career and prepare for future interviews. Let me know by commenting on this post or replying to this email.

Until next time.