#Apache airflow python tutorial update#

This should be fairly simple as writing a plugin is mostly as simple as writing a Python package. Obviously you can write your own plugins as well. Similar to operators, there’s support for plugins as well, where you can integrate third party plugins to bring in more functionality to Airflow. Similarly, if you want to run a execute a bash command or run a bash script file, you can use the BashOperator. For example, in a DAG if you want to have an operation just to denote a virtual entity, you can use a DummyOperator. And of course, there are various operators. Airflow uses the same concept to chain various operations together in a workflow.Įach operation in Airflow is defined using an operator. Spark jobs are internally converted to DAGs as well, which you can see visually from the web UI whenever a Spark job is running. This shouldn’t be new to you if you’ve already worked with tools such as Apache Spark. Wait, what’s a DAG, you ask? DAG stands for Directed Acyclic Graph. Oh, by the way, Airflow DAGs are written in Python.

#Apache airflow python tutorial update#

As I already mentioned, you can have versioning on Airflow code and easily update or rollback the code at will. It can be programmatically authored, which means you can write some code to not just define the workflow, but also to schedule and update it.

Like most Apache projects, this is community driven and open source. That should give a high-level understanding of what Airflow is. The official definition of Airflow on it’s Apache homepage is:Īirflow is a platform created by the community to programmatically author, schedule and monitor workflows.