I am running a DAG which invokes an API multiple times and API gets data from a DB. After few invocations, the DAG returns the below error. I am running the dag in kubernetes environment.
Could someone redirect me where to check and from where is dag receiving SIGTERM?
Error Logs :
[2024-06-29, 23:46:39 UTC] {local_task_job_runner.py:115} ERROR – Received SIGTERM. Terminating subprocesses
[2024-06-29, 23:46:39 UTC] {process_utils.py:131} INFO – Sending 15 to group 28. PIDs of all processes in the group: [28]
[2024-06-29, 23:46:39 UTC] {process_utils.py:86} INFO – Sending the signal 15 to group 28
[2024-06-29, 23:46:39 UTC] {taskinstance.py:1630} ERROR – Received SIGTERM. Terminating subprocesses.
[2024-06-29, 23:46:39 UTC] {test_driver_airflow.py:648} WARNING – Retrying test
[2024-06-29, 23:47:39 UTC] {process_utils.py:149} WARNING – process psutil.Process(pid=28, name=’airflow task runner:test test manual__2024-06-21T11:04:44+00:00 11476′, status=’sleeping’, started=’18:36:37′) did not respond to SIGTERM. Trying SIGKILL
[2024-06-29, 23:47:39 UTC] {process_utils.py:86} INFO – Sending the signal 9 to group 28
[2024-06-29, 23:47:39 UTC] {process_utils.py:79} INFO – Process psutil.Process(pid=28, name=’airflow task runner: test test manual__2024-06-21T11:04:44+00:00 11476′, status=’terminated’, exitcode=<Negsignal.SIGKILL: -9>, started=’18:36:37′) (28) terminated with exit code -9
[2024-06-29, 23:47:39 UTC] {standard_task_runner.py:172} ERROR – Job 11476 was killed before it finished (likely due to running out of memory)
[2024-06-29, 23:47:39 UTC] {local_task_job_runner.py:228} INFO – Task exited with return code 143
I checked the logs of API and no errors there. CPU and Memory of pods and nodes are within limits as checked in grafana.