airflow

What is airflow

The airflow image packages Apache Airflow, a platform for programmatically authoring, scheduling, and monitoring workflows. Airflow is widely used for orchestrating complex data pipelines, especially in data engineering, ETL, and machine learning workflows.

The container image provides a preconfigured Airflow environment that includes the web server, scheduler, and optional executor services. It’s commonly deployed as part of a multi-container setup with supporting services like PostgreSQL (metadata DB) and Redis (for Celery executors), either in Docker Compose or Kubernetes environments.

This image is ideal for running Airflow in reproducible, containerized environments – whether for local development, CI pipelines, or production deployments in orchestration platforms like Kubernetes.

How to use this image

The airflow image is most often used as part of a multi-service deployment. It supports various executors (Sequential, Local, Celery, Kubernetes) and components (webserver, scheduler, worker).

Run a basic instance (for testing only):

Use with Docker Compose:

‍Airflow provides an official docker-compose.yaml that brings up the full stack (webserver, scheduler, metadata DB, etc.):

Build a custom image with DAGs and plugins:

Logging and ports:

‍Airflow logs to standard output by default and exposes the web UI on port8080. Persistent logs and metadata should be configured for production use.

Image variants

The Airflow image supports several variants based on Python versions and runtime flavors:

‍Standard image with the default executor (SequentialExecutor). Tagged by Airflow version (e.g.2.8.1). Use for basic setups and dev environments.

‍Specifies a Python version (e.g.apache/airflow:2.8.1-python3.9). Use to align with your system's Python dependencies.

‍Includes specific executor support like <span class='code-box xs'>celery</span>, <span class='code-box xs'>kubernetes</span>, or <span class='code-box xs'>local</span> (e.g. <span class='code-box xs'>apache/airflow:2.8.1-python3.9-kubernetes</span>). Use for production setups where a specific execution model is required.

In addition to these variants, the image supports versioned tags for full reproducibility (e.g. <span class='code-box xs'>apache/airflow:2.8.1-python3.10-celery)</span>.