Posts on airflow

Learn how to build an ETL pipeline that extracts AI-generated Slack huddle notes using Airflow, processes them with Snowflake Cortex, and loads them into a data warehouse for future AI agent consumption.

Discover how to construct a cost-effective and scalable reverse ETL pipeline using Apache Airflow and Snowflake to seamlessly synchronize data from your Snowflake warehouse to HubSpot. This guide covers essential aspects such as state management, dynamic batching, and API integration to ensure efficient and reliable data synchronization.

Explore how we built a Slackbot powered by Snowflake Cortex Analyst to handle ad-hoc analytics requests, from the foundational data pipeline to a user-facing Slack interface.

Explore the integration of Apache Airflow and Airbyte for efficient ETL, enhancing scalability and maintainability in data engineering workflows.

After five years of using Apache Airflow, I finally contributed to the project. Here's how I overcame imposter syndrome and made my first open-source contribution.

Explore the seamless integration of secrets management in Apache Airflow, allowing you to customize connections for different development environments. Illustrated through the perspective of our Data Engineer hero, Bob.

Discover a guide to developing Apache Airflow workloads locally with Docker and a template repository for efficient, consistent development.

Reduce your AWS Managed Workflow for Apache Airflow (MWAA) costs by 50% with Plural. Explore our journey of transitioning to self-hosted Apache Airflow on AWS Kubernetes (K8s) and gain cost savings and control over data workflows.

Learn how to set up and run Astronomer Cosmos, an open-source package that allows you to run dbt core projects as Apache Airflow DAGs and Task Groups with ease. This blog post, provides a comprehensive guide to running Cosmos on MWAA (Amazon Managed Workflows for Apache Airflow).