Apache Airflow Pipeline

Airflow orchestrates the Marble Music rendering pipeline, coordinating Perforce sync, Kubernetes jobs, and Blender rendering tasks.

It is hosted using docker compose, following the documentation provided by Apache Airflow, with the Kubernetes executor enabled to allow dynamic task execution on the k8s cluster.

The Airflow Setup

These are the systems integrated with Airflow for the rendering pipeline:

• Perforce: Asset management and versioning
• Kubernetes: Container orchestration, where jobs are created to sync assets and create the animation .blend file.
• Storage: NFS/SMB shared storage
• Blender: 3D rendering engine (as a step of the pipeline)

DAGs and screenshots

These are screenshots of the Airflow setup and configuration used in this project.

Airflow DAGs directory structure

DAG configuration files

Pipeline configuration setup and the logs showing the animation script running successfully.

Improvements planned

• Better access control with more granular user roles.
• Automated backup of the Airflow metadata database
• Add local DNS names for easier access
• Monitoring and uptime alerts following the same approach used for other services in the homelab.