Relative Content

Tag Archive for pyspark

Getting an error while runing a pyspark job

spark3-submit –properties-file ${CONFIG_DIR}/${PROCESS_NM}.conf –queue ${YARN_POOL_NAME} –master yarn –deploy-mode cluster –num-executors 20 –executor-cores 2 –driver-memory 60g –executor-memory 4g –conf spark.sql.shuffle.partitions=400 /apps/edh/prod/adhocjobs/data/rubin/${PROCESS_NM}.py > ${LOG_DIR}/${PROCESS_NM}_${LOG_FILE_TS}.log produces the error File “wa907j00.py”, line 89, in <module> h_perf_evnt_trna_smry_t_2.write.options(delimiter=params[‘DEF_DELIM’], emptyValue=”,mode=’overwrite’).csv(params[‘ETAT_RAW_DIR’] + ‘/h_perf_evnt_trna_smry_t’) File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/pyspark.zip/pyspark/sql/readwriter.py”, line 1799, in csv File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py”, line 1322, in __call__ File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py”, line 169, in deco File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py”, […]

Getting an error while runing a pyspark job

spark3-submit –properties-file ${CONFIG_DIR}/${PROCESS_NM}.conf –queue ${YARN_POOL_NAME} –master yarn –deploy-mode cluster –num-executors 20 –executor-cores 2 –driver-memory 60g –executor-memory 4g –conf spark.sql.shuffle.partitions=400 /apps/edh/prod/adhocjobs/data/rubin/${PROCESS_NM}.py > ${LOG_DIR}/${PROCESS_NM}_${LOG_FILE_TS}.log produces the error File “wa907j00.py”, line 89, in <module> h_perf_evnt_trna_smry_t_2.write.options(delimiter=params[‘DEF_DELIM’], emptyValue=”,mode=’overwrite’).csv(params[‘ETAT_RAW_DIR’] + ‘/h_perf_evnt_trna_smry_t’) File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/pyspark.zip/pyspark/sql/readwriter.py”, line 1799, in csv File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py”, line 1322, in __call__ File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py”, line 169, in deco File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py”, […]

Getting an error while runing a pyspark job

spark3-submit –properties-file ${CONFIG_DIR}/${PROCESS_NM}.conf –queue ${YARN_POOL_NAME} –master yarn –deploy-mode cluster –num-executors 20 –executor-cores 2 –driver-memory 60g –executor-memory 4g –conf spark.sql.shuffle.partitions=400 /apps/edh/prod/adhocjobs/data/rubin/${PROCESS_NM}.py > ${LOG_DIR}/${PROCESS_NM}_${LOG_FILE_TS}.log produces the error File “wa907j00.py”, line 89, in <module> h_perf_evnt_trna_smry_t_2.write.options(delimiter=params[‘DEF_DELIM’], emptyValue=”,mode=’overwrite’).csv(params[‘ETAT_RAW_DIR’] + ‘/h_perf_evnt_trna_smry_t’) File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/pyspark.zip/pyspark/sql/readwriter.py”, line 1799, in csv File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py”, line 1322, in __call__ File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py”, line 169, in deco File “/opt/cloudera/parcels/CDH-7.2.18-1.cdh7.2.18.p200.54625612/lib/spark3/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py”, […]