What’s difference between pyspark.DataFrame.checkpoint() and pyspark.RDD.checkpoint()?
I’m currently struggling with spark checkpoints and trying to understand what’s the difference between DataFrame and RDD checkpoints.
How can I read large file CSV file up 500 GB in Apache Spark and perform aggregation on it?
How can I read large file CSV file up 500 GB in Apache Spark and perform calculation and transformation on one of its Column. I have been given a large file to perform ETL and calculation on it. I am newbie in Python / Spark. Any help will be appreciated
pyspark .display() works but .collect(), .distinct() and show() don’t
I’m working with a pyspark dataframe in Azure Databricks and I’m trying to count how many unique (distinct) values a particular column has.
spark.write.saveAsTable not writing all the rows
Yesterday, I ran a simple spark code on ingesting a large table. The code was simple in that it did a
Pyspark SQL not spliting column
I was trying to split my column using pyspark sql based on the values that are stored in another column, I saw that it worked for some specific values but for some other this is not working.
Get aggregates for a dataframe with different combinations
Total pyspark noob here. I have a dataframe similar to this:
PySpark basic question – who runs the Python code and how after all?
I am following a course on Spark. I installed and so I now am running Spark on Windows.
pyspark How ready folder with binary files continuously – on new files
I created a pyspark pipeline that begins on reading binary files:
PySpark NOT_COLUMN_OR_STR Exception on Disconnected List
I am getting an odd pyspark exception when attempting to use filter and lambda functions on a list of ints I’ve collected from a pyspark dataframe, which makes no sense as the data exists in memory as a list and should be completely disconnected from pyspark. Here is the scenario.
Running Pyspark
Pyspark functions not working