Azure Blob / ADLS

Required extra

This connection requires the azure and duckdb extras. See Installation.

Test data stored in Azure Blob storage or Azure Data Lake Storage Gen2 (ADLS) in various formats.

Server

servers:
  - server: production
    type: azure
    location: abfss://datameshdatabricksdemo.dfs.core.windows.net/inventory_events/*.parquet
    format: parquet

Environment variables

Authentication uses an Azure Service Principal (App Registration) with a secret.

Variable	Example	Description
DATACONTRACT_AZURE_TENANT_ID	79f5b80f-...	The Azure Tenant ID
DATACONTRACT_AZURE_CLIENT_ID	3cf7ce49-...	The Application/Client ID of the app registration
DATACONTRACT_AZURE_CLIENT_SECRET	yZK8Q~GWO1M...	The client secret value

Metadata checks

Instead of reading the file contents, you can validate the metadata of the blobs themselves (size, content type, last modified, file count, …). Each ODCS schema object with physicalType: file represents a unique folder/prefix in Azure Blob storage or ADLS Gen2. Its properties map directly to BlobProperties attributes from the Azure SDK: for every declared property the engine extracts the corresponding attribute from each blob and validates it against the quality constraints declared on that property.

Property name	BlobProperties attribute
name	blob.name
size	blob.size
lastModified	blob.last_modified (UTC datetime)
creationTime	blob.creation_time (UTC datetime)
lastAccessedOn	blob.last_accessed_on (UTC datetime)
contentType	blob.content_settings.content_type
contentEncoding	blob.content_settings.content_encoding
contentLanguage	blob.content_settings.content_language
contentDisposition	blob.content_settings.content_disposition
cacheControl	blob.content_settings.cache_control
contentMd5	blob.content_settings.content_md5
etag	blob.etag
blobType	blob.blob_type.value (e.g. BlockBlob)
blobTier	blob.blob_tier.value (e.g. Hot)
archiveStatus	blob.archive_status
serverEncrypted	blob.server_encrypted
deleted	blob.deleted
snapshotId	blob.snapshot
versionId	blob.version_id
tagCount	blob.tag_count

Schema-level quality checks (schema.quality) on the rowCount metric are evaluated as file-count thresholds against the total number of blobs found under the prefix.

contentType is normalised before comparison: MIME parameters are stripped, so application/json; charset=utf-8 matches application/json.

Supported location URL formats (on the server block):

• https://<account>.blob.core.windows.net/<container>/<prefix>
• abfss://<container>@<account>.dfs.core.windows.net/<prefix>
• azure://<container>@<account>.blob.core.windows.net/<prefix>
• wasbs://<container>@<account>.blob.core.windows.net/<prefix>