I am trying to dynamically generate a SQL statement with the SQLExecuteQueryOperator based on a previous task’s xcom value. The problem is that the xcom value is not being inserted into the SQL Statement. Instead, Airflow is attempting to execute the query select * from table where value = {{ti.xcom_pull(task_ids='return_bq_value')}} which obviously fails. Is it possible using this operator to have the xcom value render in the query so that select * from table where value = 1 is executed?

I am trying to use the Airflow operator BigQueryToMsSqlOperator to move a dataset from BigQuery to SQL Server (2019). The dataset is 9 columns of numeric, date, and integer data types. It currently transfers at about 600 rows per minute which will take hours since the entire data set is about 500k rows. Is there a way to speed this up? The target dataset is truncated before insert and is not clustered.