The Curse of the Data Lake Monster

A lot has happened since I last wrote. For one, I started at ThoughtWorks. This has given a bit of a change in direction and I am grappling with enterprise software development.

At ThoughtWorks I’ve been exposed quite a bit to companies asking for Data Lakes.

I co-wrote an article with Kiran Prakash, a Principal Engineer at ThoughtWorks, about common antipatterns we see in approaches to building data lakes. tl;dr they are often approached as an infrastructure problem, whereas we advocate treating them as products.

[…] we argue that software is best developed in thin, vertical slices that emphasize use cases and user outcomes, and data-intensive projects are no exception. When it comes to use cases that rely on rich, multi-format data, it can be tempting to start by creating a horizontal data platform layer, sometimes called a data lake. We are going to explore examples of how product thinking can apply to a project where a data lake is being considered as a solution.

Read the full article on ThoughtWorks Insights