I’m running Spark 3.4.1 tasks scheduled by Airflow 2.6.1. with a SparkSubmit Operator. Spark is running on Cluster Mode and therefore I don’t have explicit logs from Spark Driver. Instead i have updates from spark_submit.py job which polls spark driver pod whether job is finished or not.