I have an airflow instance running inside a docker container. I might want to update Airflow in the future while ensuring that the database matches the current version of airflow.

I have an application that is essentially a web server (with a separate Dockerfile and dependencies) that includes a custom workflow engine. Additionally, I have around 20 Docker images with AI models, each with a FastAPI wrapper exposing the models’ APIs. When a user makes a request, the web server builds a DAG (not an Airflow DAG, but a DAG in this custom workflow engine), where each ‘component’ of the DAG calls the web API of a specific container with an AI model.

Is there a way to pause a task for 24 hours then run the same task again?

I’d like to have an Airflow DAG get some status information back after running an AzureContainerInstancesOperator task in Airflow. With the equivalent docker operator, DockerOperator, it is possible to get the last line of output by default inserted into XCOM. With the AzureContainerInstancesOperator, this seems less likely to be possible since the container is run remotely, possibly without network access.

I have a dag scheduled using EventsTimetable. It ran three time on schedule as expected, and then completely ignored two runs in a row- No failure/skipped in the UI, no alert email. Nothing in the logs either, as if the run was never scheduled at all. The “next run” showed up correctly in the UI before the runs that were skipped. Today, it magically started running on schedule again. We are running airflow on AKS, but I checked the pods and all seemed to be running normally when the outage occurred.

We have a Airflow job thats sends a lot of messages (>100k) to SQS. The job fails sometimes fails for no reason. When retriggered, it succeeds without any change in code.
Error message: airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.

I am starting to learn how to use Airflow for pipelines.
I have the simplest DAG in

I’m trying to build my first airflow DAG in k8s running on custom image.
I did the following example code:

Often times i need to use task catchup to fill some tables, and sometimes these dags are dependent on other ones. However, dependency dag might not have runs in all days on which the dependent one will catchup.
So far solution is to merge dag with commented sensor, run catchup, then uncomment sensor.
I wonder if there is a way to specify a date from which sensor will actually check for dagruns, and until that day to have it succeeding.

I use airflow to schedule my ETL jobs, and the project have version control.