Great writeup! I'm super curious to see a good setup of "answering business questions" with AI working at scale. There's a few ways around like snowflake cortex on top of their semantic layer. However I'd prefer something less opinionated and vendor locked
AI is both exciting and scary. But there is still so much that still need human input in the data engineering and modeling. Hopefully that stays that way.
Hey Zach, I would love you to write a foreword for my upcoming SQL book.
I think data modeling has been the core of any pipelines and that's something at least in the near future cannot be replaced by AI. Modeling using either top down or bottom up approach requires enormous amounts of human effort.
I'm having hard time in deciding between your DE vs AE program. When's your next AE batch?
I think the "The data engineer models the data correctly and the data documentation is stellar and accessible via AI." is a tricky one. No data engineer model things 100% correctly, and humans hate to document.
That's why we're spending so much time building an AI that can figure out the data without relying on human documentation.
Great writeup! I'm super curious to see a good setup of "answering business questions" with AI working at scale. There's a few ways around like snowflake cortex on top of their semantic layer. However I'd prefer something less opinionated and vendor locked
AI is both exciting and scary. But there is still so much that still need human input in the data engineering and modeling. Hopefully that stays that way.
Hey Zach, I would love you to write a foreword for my upcoming SQL book.
I think data modeling has been the core of any pipelines and that's something at least in the near future cannot be replaced by AI. Modeling using either top down or bottom up approach requires enormous amounts of human effort.
I'm having hard time in deciding between your DE vs AE program. When's your next AE batch?
The next AE batch will be in the fall most likely. September/October timeline
Thank You for that info.
Great post!
I think the "The data engineer models the data correctly and the data documentation is stellar and accessible via AI." is a tricky one. No data engineer model things 100% correctly, and humans hate to document.
That's why we're spending so much time building an AI that can figure out the data without relying on human documentation.