Relative Content

Tag Archive for pythonconcurrency

ProcessPoolExecutor() with the workers set to default crashes on Rocks 7.0 (Manzanita)

I’m using Python 3.9.19 on Rocks 7.0 (Manzanita) , and when I do not set the number of threads to some value and let it default , it crashes. I have 64 cpu cores. Could the issue be that it is just using too much memory and then crashing? While debugging I set the number_workers parameter to 2 and it was working fine, just too slow, so using only 2 is not practically feasible for me.
I am currently running it with 8 workers and it is running fine for now.