Posts on data engineering

Learn how to build an ETL pipeline that extracts AI-generated Slack huddle notes using Airflow, processes them with Snowflake Cortex, and loads them into a data warehouse for future AI agent consumption.

Discover how to construct a cost-effective and scalable reverse ETL pipeline using Apache Airflow and Snowflake to seamlessly synchronize data from your Snowflake warehouse to HubSpot. This guide covers essential aspects such as state management, dynamic batching, and API integration to ensure efficient and reliable data synchronization.

Explore the seamless integration of secrets management in Apache Airflow, allowing you to customize connections for different development environments. Illustrated through the perspective of our Data Engineer hero, Bob.

Explore the art of transforming JSON data in Amazon Redshift databases, mastering elegant techniques to split and extract data from nested structures. Uncover time-saving strategies for managing one-to-many relationships and enhancing data manipulation workflows.

Discover the distinctions between Data Analyst, Data Engineer, and Data Scientist roles and learn the essential skills required to transition from one to the other.

The Power of Pre-Commit and SQLFluff - Ensuring Consistent SQL Code Across Your Team's Projects