Relative Content

Tag Archive for pythonpysparkpicklerddazure-synapse

Save text files as binary format using saveAsPickleFile with pyspark

I have around 613 text files stored in azure data lake gen 2 at this path for eg ‘/rawdata/no=/.txt’. I want to read all the text files and unbase 64 all text files as they are base64 encoded. But when i run my code the code reads 600 text files but gives output as 20 files only. I am not able to understand why am i getting only 20 output files?