Databricks Reference
Authentication options and data type handling for Databricks connections.
Authentication
| Variable | Example | Description |
|---|---|---|
DATACONTRACT_DATABRICKS_SERVER_HOSTNAME | dbc-abcdefgh-1234.cloud.databricks.com | Host of the SQL warehouse or compute cluster |
DATACONTRACT_DATABRICKS_HTTP_PATH | /sql/1.0/warehouses/b053a3ff... | HTTP path to the SQL warehouse or compute cluster |
DATACONTRACT_DATABRICKS_TOKEN | dapia0000... | A personal access token (PAT) |
DATACONTRACT_DATABRICKS_CLIENT_ID | 00000000-... | Service principal client ID for OAuth M2M auth |
DATACONTRACT_DATABRICKS_CLIENT_SECRET | dose0000... | Service principal OAuth secret (used with the client ID) |
DATACONTRACT_DATABRICKS_PROFILE | my-profile | A profile from ~/.databrickscfg (Databricks SDK unified auth) |
DATACONTRACT_DATABRICKS_AUTH_TYPE | databricks-oauth | Explicit connector auth type, e.g. for interactive U2M browser login |
The authentication method is selected from the variables you set, in this order: PAT → OAuth service principal (CLIENT_ID + CLIENT_SECRET) → config profile → explicit AUTH_TYPE. catalog and schema come from the contract's servers block.
datacontract import unity needs only the hostname plus a PAT or profile; datacontract test additionally needs DATACONTRACT_DATABRICKS_HTTP_PATH. When a spark session is passed programmatically (notebook/pipeline), no credentials are needed at all.
Data types
Importing
datacontract import unity keeps the full Unity Catalog type text as physicalType (e.g. decimal(10,2), struct<x:int>, map<string,int>) — also duplicated in the databricksType custom property — and maps to logicalType:
| Databricks type | logicalType |
|---|---|
STRING, VARCHAR, CHAR | string |
INT, BIGINT, SMALLINT, TINYINT | integer |
FLOAT, DOUBLE, DECIMAL | number |
BOOLEAN | boolean |
DATE | date |
TIMESTAMP | timestamp |
BINARY | string (format binary) |
ARRAY<...> | array (nested items when pyspark is available) |
STRUCT<...> | object (nested properties when pyspark is available) |
MAP<...>, INTERVAL, VARIANT | (unset) — physicalType is still written |
datacontract import spark maps Spark session types the same way, with two caveats: TimestampType and TimestampNTZType are currently imported as date, and BinaryType as array.
Testing
Databricks supports native type introspection: the declared physicalType is checked against the actual information_schema column type. Timezone variants of timestamps are interchangeable; parameters are only enforced when declared. A physicalType that isn't valid Databricks SQL falls back to the logical type category comparison. When testing a Spark DataFrame (type: dataframe), only the logical type category is checked.
Logical type mapping
When no physicalType is declared, the CLI derives the native type from the logicalType — for example in datacontract export sql and the dbt exports. This table is generated from the converter in the CLI's code:
logicalType | Databricks type |
|---|---|
string | STRING |
integer | INT |
number | DECIMAL(38,0) |
boolean | BOOLEAN |
date | DATE |
timestamp | TIMESTAMP |
time | STRING |
object | STRUCT<> |
array | ARRAY<STRING> |