DBT External Table Databricks

  Kiến thức lập trình

I get csv files dropped into Azure Blob storage daily and set up a Databricks volume to access these files. I ingest these files with dbt using:

select *
from read_files('<path/to/volume/folder>')

Is there a way to create this path as a source so I can use a command similar to:

select *
from {{ source('source_name', 'table_name') }}

I set up a source with the following data:

version: 2

sources:
  - name: source_name
    tables:
      - name: table_name
        external:
          location: '<path/to/volume/folder>'
          file_format: csv'

When I run:

select *
from {{ source('source_name', 'table_name') }}

The compiled query is:

select *
from catalog_name.schema_name.table_name

I get an error because this table does not exist in the Databricks catalog as it is an external blob storage folder.

LEAVE A COMMENT