Blog

Jan 9, 2025

Learn how to build an ETL pipeline that extracts AI-generated Slack huddle notes using Airflow, processes them with Snowflake Cortex, and loads them into a data warehouse for future AI agent consumption. Airflow / Snowflake / ETL / Data Engineering

Dec 23, 2024

Discover how to construct a cost-effective and scalable reverse ETL pipeline using Apache Airflow and Snowflake to seamlessly synchronize data from your Snowflake warehouse to HubSpot. This guide covers essential aspects such as state management, dynamic batching, and API integration to ensure efficient and reliable data synchronization. Airflow / Snowflake / ETL / Data Engineering / Python / SQL

Oct 4, 2024

Explore how we built a Slackbot powered by Snowflake Cortex Analyst to handle ad-hoc analytics requests, from the foundational data pipeline to a user-facing Slack interface. Snowflake / Airflow

May 28, 2024

Explore the integration of Apache Airflow and Airbyte for efficient ETL, enhancing scalability and maintainability in data engineering workflows. Airflow / ETL

May 21, 2024

Explore how Artie helped us streamline our ETL pipelines by replicating our MySQL database into Snowflake. Discover how Artie's real-time database replication tool helped us save time and focus on more important tasks. ETL

Mar 3, 2024

After five years of using Apache Airflow, I finally contributed to the project. Here's how I overcame imposter syndrome and made my first open-source contribution. Airflow

Nov 19, 2023

Explore the power of Python and GitHub Workflows in enhancing data quality with our comprehensive guide. Learn how to analyze dbt model coverage, ensuring robust documentation and efficient testing in your data engineering projects. dbt / Python / CICD

Nov 12, 2023

Explore the seamless integration of secrets management in Apache Airflow, allowing you to customize connections for different development environments. Illustrated through the perspective of our Data Engineer hero, Bob. Airflow / Data Engineering

Nov 6, 2023

Discover a guide to developing Apache Airflow workloads locally with Docker and a template repository for efficient, consistent development. Airflow

Oct 30, 2023

Elevate your data management with DataHub, dbt Core in Apache Airflow, and Astronomer Cosmos. Learn how to automate metadata ingestion and streamline data quality checks for your data model in this comprehensive guide. dbt

Sep 23, 2023

Learn how to set up and run Astronomer Cosmos, an open-source package that allows you to run dbt core projects as Apache Airflow DAGs and Task Groups with ease. This blog post, provides a comprehensive guide to running Cosmos on MWAA (Amazon Managed Workflows for Apache Airflow). dbt / Airflow

Aug 30, 2023

Explore the art of transforming JSON data in Amazon Redshift databases, mastering elegant techniques to split and extract data from nested structures. Uncover time-saving strategies for managing one-to-many relationships and enhancing data manipulation workflows. Data Engineering / ETL

Jul 3, 2023

Discover the distinctions between Data Analyst, Data Engineer, and Data Scientist roles and learn the essential skills required to transition from one to the other. Data Engineering

Jun 10, 2023

The Power of Pre-Commit and SQLFluff - Ensuring Consistent SQL Code Across Your Team's Projects Data Engineering / SQL / CICD