Why do I get a org.apache.spark.sql.connector does not allow create table as select. error when trying to write to a kinesis sink in glue
I’m trying to take some data from a few sources do some transformations on it and load it into Kinesis using AWS glue and scala. The data is coming from static sources like tables and s3 buckets so it’s not a streaming ETL job. Currently I’m using a dynamicFrame and trying to take my data sink and simply do a writeDynamicFrmae like so
Why do I get a org.apache.spark.sql.connector does not allow create table as select. error when trying to write to a kinesis sink in glue
I’m trying to take some data from a few sources do some transformations on it and load it into Kinesis using AWS glue and scala. The data is coming from static sources like tables and s3 buckets so it’s not a streaming ETL job. Currently I’m using a dynamicFrame and trying to take my data sink and simply do a writeDynamicFrmae like so
Why do I get a org.apache.spark.sql.connector does not allow create table as select. error when trying to write to a kinesis sink in glue
I’m trying to take some data from a few sources do some transformations on it and load it into Kinesis using AWS glue and scala. The data is coming from static sources like tables and s3 buckets so it’s not a streaming ETL job. Currently I’m using a dynamicFrame and trying to take my data sink and simply do a writeDynamicFrmae like so
Why do I get a org.apache.spark.sql.connector does not allow create table as select. error when trying to write to a kinesis sink in glue
I’m trying to take some data from a few sources do some transformations on it and load it into Kinesis using AWS glue and scala. The data is coming from static sources like tables and s3 buckets so it’s not a streaming ETL job. Currently I’m using a dynamicFrame and trying to take my data sink and simply do a writeDynamicFrmae like so
Why do I get a org.apache.spark.sql.connector does not allow create table as select. error when trying to write to a kinesis sink in glue
I’m trying to take some data from a few sources do some transformations on it and load it into Kinesis using AWS glue and scala. The data is coming from static sources like tables and s3 buckets so it’s not a streaming ETL job. Currently I’m using a dynamicFrame and trying to take my data sink and simply do a writeDynamicFrmae like so
Why am I getting a Failed to find data source error when writing to kinesis from AWS Glue (Spark)
I’m trying to take some data from a few sources do some transformations on it and load it into Kinesis using AWS glue and scala. The data is coming from static sources like tables and s3 buckets so it’s not a streaming ETL job. Currently I’m using a dynamicFrame and trying to take my data sink and simply do a writeDynamicFrmae like so
Why am I getting a Failed to find data source error when writing to kinesis from AWS Glue (Spark)
I’m trying to take some data from a few sources do some transformations on it and load it into Kinesis using AWS glue and scala. The data is coming from static sources like tables and s3 buckets so it’s not a streaming ETL job. Currently I’m using a dynamicFrame and trying to take my data sink and simply do a writeDynamicFrmae like so
Why am I getting a Failed to find data source error when writing to kinesis from AWS Glue (Spark)
I’m trying to take some data from a few sources do some transformations on it and load it into Kinesis using AWS glue and scala. The data is coming from static sources like tables and s3 buckets so it’s not a streaming ETL job. Currently I’m using a dynamicFrame and trying to take my data sink and simply do a writeDynamicFrmae like so
Why does using getField() return a none type in AWS glue for scala for an Array of objects despite confirming it is there
When using getField() I am able to return simple values but when I try and use it on an array of objects it returns None. So I’ve tried to create a simple version of the Scala code I’m having problems with below I’m trying to run a Glue job and as part of that do some mapping on a dynamicFrame. I’m aware I could just switch to a data frame to do this but I’m curious why the dynamic frame doesn’t work as expected.