When taking a new job in data, ask how they test their stuff. No testing is🚩, either not serious about data or you're joining a firehouse. Beware of the opposite too: don't want to spend days filling out an Excel spreadsheet to merge a single PR

Built a column-level lineage graph for a data platform with 3.1M tables and 1B+ columns. Metadata is new big data.

After checking out john kutay’s music, I am taking streaming data much more seriously

Good BI is more valuable than AI. Good CI is more valuable than BI.

It just hit me: Fortran > Fivetran

Is there a mature-enough open-source framework for managing analytical event schemas? Like iterative.ly / avo.app but one I can tinker with?

Data engineering is like you take all the frustrating parts of being a data analyst and combined them with all the frustrating parts of being a software engineer

This is both extremely sophisticated and beautifully simple. Ironically, they've shut down this market-beating system because it lacked product-market fit! principiamundi.com/posts/didact-a…

OK, besides forcing everyone to lowercase/uppercase SQL keywords, what do you use CI for in your data/analytics repo?

version control? oh yeah, I love version control

When you realize it's time to grow up and rewrite your cute ML pipeline from Pandas into Spark 🥶 🦸‍♀️ comes Fugue and magically does this for you ✨ github.com/fugue-project/… a really neat project from my alma mater – Lyft 🚀

But the real diff is in the data, not the code

A fellow staff data engineer at a public tech company says it takes a full week to ship any change to a SQL pipeline that powers core financial reporting. 1-2 days dev work, then 4-5 days QA 🤯. I guess the risk of blowing up reporting is worth the misery but we need to do better

SQL is life. Run SQL on CSV, Parquet, JSON, Arrow, Unix Pipes and Google Sheets. P.S. if you love SQL so much that you are running it on Unix Pipes – feel free to DM me your resume

Spending some time improving how you work is often more impactful than the same time spent on the work itself.

Datafold has been helping dbt developers prevent bad code deploys in CI. Now it can also help you during development by profiling and diffing your local dev data against prod to quickly audit your work to ship faster and confidently!

I've always struggled with QAing SQL code I was writing – super hard to trace the impact of code changes on the resulting data. Now the VS Code extension lets you preview the changes to SQL/dbt code right in the IDE by diffing the local data against production and ship faster 🚀

What are some of the best ways to approach testing in dbt? We've got you covered 🧵.

Data migrations are hard. I have a PTSD after spending 2 years migrating Lyft off Redshift and failing spectacularly. Fortunately, fearless Kira Furuichi is helping me recover by writing great guides on how to do migrations the right way 💡

There are major fires burning in and around the Los Angeles metro area this week, causing tragic loss of life and property. What are the primary drivers of these events and their consequences? What is the impact of climate change and fuel (vegetation and, in this case,