Airflow ERROR – Received SIGTERM. Terminating subprocesses

  Kiến thức lập trình

I am running a DAG which invokes an API multiple times and API gets data from a DB. After few invocations, the DAG returns the below error. I am running the dag in kubernetes environment.

Could someone redirect me where to check and from where is dag receiving SIGTERM?

Error Logs :

[2024-06-29, 23:46:39 UTC] {local_task_job_runner.py:115} ERROR – Received SIGTERM. Terminating subprocesses
[2024-06-29, 23:46:39 UTC] {process_utils.py:131} INFO – Sending 15 to group 28. PIDs of all processes in the group: [28]
[2024-06-29, 23:46:39 UTC] {process_utils.py:86} INFO – Sending the signal 15 to group 28
[2024-06-29, 23:46:39 UTC] {taskinstance.py:1630} ERROR – Received SIGTERM. Terminating subprocesses.
[2024-06-29, 23:46:39 UTC] {test_driver_airflow.py:648} WARNING – Retrying test
[2024-06-29, 23:47:39 UTC] {process_utils.py:149} WARNING – process psutil.Process(pid=28, name=’airflow task runner:test test manual__2024-06-21T11:04:44+00:00 11476′, status=’sleeping’, started=’18:36:37′) did not respond to SIGTERM. Trying SIGKILL
[2024-06-29, 23:47:39 UTC] {process_utils.py:86} INFO – Sending the signal 9 to group 28
[2024-06-29, 23:47:39 UTC] {process_utils.py:79} INFO – Process psutil.Process(pid=28, name=’airflow task runner: test test manual__2024-06-21T11:04:44+00:00 11476′, status=’terminated’, exitcode=<Negsignal.SIGKILL: -9>, started=’18:36:37′) (28) terminated with exit code -9
[2024-06-29, 23:47:39 UTC] {standard_task_runner.py:172} ERROR – Job 11476 was killed before it finished (likely due to running out of memory)
[2024-06-29, 23:47:39 UTC] {local_task_job_runner.py:228} INFO – Task exited with return code 143

I checked the logs of API and no errors there. CPU and Memory of pods and nodes are within limits as checked in grafana.

New contributor

deepak is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

Theme wordpress giá rẻ Theme wordpress giá rẻ Thiết kế website

LEAVE A COMMENT