Protection against that spark driver creates infinite amount of executors if executors fail
I use spark 3.4.1 in kubernetes in cluster mode. The problem I experienced was that I had a syntax error in log4j2 settings for executor and therefore executor pod fails as sonn as it is started. Then the driver will create another executor which also fails around 10-20sec later and so forth…
How to make existing pyspark visible for pip
I am using python binding of spark docker image for kubernetes, this image already contains pyspark lib. I need to use conda in this image. Is there the way to make existing pyspark visible for conda pip?
Spark client application hangs even after remote job completion
I am having a spark remote server setup in k8s , i am having a dataset of 70 million in which i perform some joins and then write them to db , all my data is successfully written in the db and there is no activity in the spark UI however my spark client applications hangs and do not exit most of the time.