I created a spark session like that:
from pyspark.sql import SparkSession

I want to load .dat(without headers) file to hive external table.
But in hive table there are extra columns like cob_date , region, file_name
which are not present in .dat file.
cob_date will be the date present in feed name like customer_data_20240708.dat
region and file_name cab be hard coded.

I tried running a script like this: