![]() All of these are more than tedious introspection. These aren’t just nice to have questions. That’s 11 tabs and 40+ clicks more than I want in my life, and in practice I’ll touch maybe 2-3 of these during active development. What’s a data diff compared to current production data? → Query Console tab and run a query against production and/or learn a whole new tool What are existing database permissions on this model? → similar as aboveĪnyone working on pull requests in real time that rely on my work? → Slack thread with 2-3 replies hurting? → go through a whole tutorial and babysit some SQL commands to copy and paste if I don’t want to model them in dbt How much does this cost to run in production and am I helping vs. How many scheduled data pipelines are tied to this model? → Airflow or dbt Cloud or random scheduler and click 10-20 different times across DAGs What’s a data preview based on my updates look like? → Query Console tab → 2-3 clicks, copy and paste my compiled dbt SQL, scrolling and eyeballing hurt? → Dig into BI configs → 5-10 clicks and pray my intuition guides me to the right bar chart Who uses this model and how often? Query history for the specific table → 2-3 clicks and typing out a filter, scrolling and eyeballing How often does this fail in production? Look at a tab in Airflow or dbt Cloud → 2-3 clicks and typing out a filter, scrolling and eyeballing What’s historical performance on this and am I beating it? → Query History → 2-3 clicks and typing out a filter Who else is touching this precious file of mine? I’m tired of pull request clashing → Github PR page → 2-3 clicks ![]() Let’s count ‘em out Blue’s Clues™️ style. Each of those questions is at least one tab and a couple clicks in my browser or IDE. I thought data engineering was supposed to be more fun than this.
0 Comments
Leave a Reply. |