I am trying to persist data into a delta table using a scheduled notebook which reads daily from an Postgres DB. I want to make sure no duplicates are stored so I perform a merge operation with new data:

I was reading the architecture of Databricks from the following video. And I came through this

I am attempting to connect to an Oracle server. I tried to install the ojdbc8.jar library, but I encountered an error: “Library installation attempted on the driver node of cluster 0718-101257-h5k9c5ud failed. Please refer to the following error message to resolve the issue or contact Databricks support. Error Code: DRIVER_LIBRARY_INSTALLATION_FAILURE. Error Message: com.databricks.api.base.DatabricksServiceException: BAD_REQUEST: DBFS library installations are not supported on DBR 15 or above.”

Scenario (in Databricks):

It may sound naive, recently I was exploring databricks and I got to know that Azure Databricks is a thing. I’m a little bit confused about what’s the difference between Azure databricks vs the databricks account we create by choosing cloud service, is it same or anything I’m missing here. If it is not the same are there such services for AWS/GCP.

I defined a couple of libraries within a policy.

I was at a databricks conference where they talked about computational power and how that should be a key factor when choosing a platform and not just be guided by price because if the same task could be done by databricks with photon in half the time even if it cost a bit more then it was worth it.

This should be so basic and obvious that a child could do it, yet I’ve spent several hours trying to figure it out. Every resource I’ve found (and there have been many) says to “Click Data in the left sidebar and then select Add data.” Even Databricks’ own chatbot says this. There is no Data option to click in the left sidebar. It doesn’t exist.

Found an old question for this problem in pandas:

I used below code to save ms sql table to databricks table.