I have a complex join that I’m trying to optimize
df1 has cols id,main_key,col1,col1_isnull,col2,col2_isnull…col30
df2 has cols id,main_key,col1,col2..col_30

I am using spark sql to run sql jobs using 10G executor memory.
When I am monitoring using Spark UI, I can see that data is being spilled to Disk and Memory (expected doing some explode operations ).

I am using spark sql to run sql jobs using 10G executor memory.
When I am monitoring using Spark UI, I can see that data is being spilled to Disk and Memory (expected doing some explode operations ).

I am using spark sql to run sql jobs using 10G executor memory.
When I am monitoring using Spark UI, I can see that data is being spilled to Disk and Memory (expected doing some explode operations ).

I am working through this example of aggregation functions for PySpark Window.

I have a quote from a blog on the physical planning in spark. I want to understand what it means

Suppose I have a table that contains orders of customers, partitioned (only) by date_transaction. I want to find the maximum value of date_transaction.

I have a timeseries dataset and am looking to make a new column that represents the last reported values (not null). I think I have this part figured out, using a combination of lag and last

I came across a very weird discovery today.

I am encountering a ShutdownHookManager error when running Spark with a custom temporary directory configuration. My directory structure and configuration details are as follows: