Relative Content

Tag Archive for airflowgoogle-cloud-composer

Airflow to load data from GCS to BQ

I am learning airflow and trying to use GCSToBigQueryOperator to load data from GCS to BQ Table, it seems the task_id is not an accepted parameter and I am getting below error

Cloud Composer 3 pricing explanation

Cloud Composer 3 pricing is based on DCU (data compute unit), https://cloud.google.com/composer/pricing#composer-3 but I couldn’t find anywhere concrete explanation of what it represents (besides a vague explanation “Data Compute Unit is an abstract metering unit that represents computational resources allocated by a Cloud Composer environment at a point in time.”)

Why is my Airflow task failing initially then succeeding after a moment?

I’m noticing some unusual behaviour with an Airflow DAG of mine: One of the tasks (a PythonOperator) will fail. The downstream tasks will then show a status of “upstream failed”. Then, after another second or so, the original task that failed will suddenly be marked as “success”.

Strange Airflow behaviour: task fails but then succeeds after a moment

I’m noticing some unusual behaviour with an Airflow DAG of mine: One of the tasks (a PythonOperator) will fail. The downstream tasks will then show a status of “upstream failed”. Then, after another second or so, the original task that failed will suddenly be marked as “success”.