Skip to main content

Azure Blob / ADLS

Test data stored in Azure Blob storage or Azure Data Lake Storage Gen2 (ADLS) in various formats.

1. Install​

uv tool install --python python3.11 --upgrade 'datacontract-cli[azure,duckdb]'

See Installation for pip, pipx, and Docker.

2. Set credentials​

Authentication uses an Azure Service Principal (App Registration) with a secret. Create a .env file in your working directory (or export the variables):

# .env
DATACONTRACT_AZURE_TENANT_ID=79f5b80f-10ff-40b9-9d1f-774b42d605fc
DATACONTRACT_AZURE_CLIENT_ID=3cf7ce49-e2e9-4cbc-a922-4328d4a58622
DATACONTRACT_AZURE_CLIENT_SECRET=yZK8Q~GWO1MMXXXXXXXXXXXXX

3. Create a contract from your files​

Download one blob and import its schema, then point the generated servers block at the storage account:

az storage blob download --account-name myaccount --container-name inventory \
  --name inventory_events/part-000.parquet --file part-000.parquet
datacontract import parquet --source part-000.parquet --output datacontract.yaml

The import generates a servers entry of type: local. Replace it with your Azure location:

servers:
  - server: production
    type: azure
    location: abfss://inventory@myaccount.dfs.core.windows.net/inventory_events/*.parquet
    format: parquet

4. Test the actual data​

datacontract test datacontract.yaml
🟢 data contract is valid. Run 17 checks. Took 4.1 seconds.

5. Let it catch a violation​

The contract becomes valuable when it detects drift. Tighten an expectation — for example, mark a field as required: true that occasionally arrives empty, or add a quality rule:

schema:
  - name: inventory_events
    # ...
    quality:
      - type: sql
        description: No event has a negative quantity
        query: SELECT COUNT(*) FROM inventory_events WHERE quantity < 0
        mustBe: 0

Run datacontract test datacontract.yaml again: every violation is listed as an error, and the command exits with code 1 — ready for CI/CD scheduling so you catch drift before your consumers do.

Server reference​

servers:
  - server: production
    type: azure
    location: abfss://datameshdatabricksdemo.dfs.core.windows.net/inventory_events/*.parquet
    format: parquet

Reference​

All authentication options (service principal, connection string, account key), supported location URL formats, and the data type handling per file format: Azure Reference.

Troubleshooting​

  • AuthorizationPermissionMismatch / 403 — the service principal needs the Storage Blob Data Reader role on the container or storage account (IAM role assignment, not just API permissions).
  • No files found that match the pattern — check the location URL format (see below) and the glob; it matches blob names under the prefix.

Metadata checks​

Instead of reading the file contents, you can validate the metadata of the blobs themselves (size, content type, last modified, file count, …). Each ODCS schema object with physicalType: file represents a unique folder/prefix in Azure Blob storage or ADLS Gen2. Its properties map directly to BlobProperties attributes from the Azure SDK: for every declared property the engine extracts the corresponding attribute from each blob and validates it against the quality constraints declared on that property.

Property nameBlobProperties attribute
nameblob.name
sizeblob.size
lastModifiedblob.last_modified (UTC datetime)
creationTimeblob.creation_time (UTC datetime)
lastAccessedOnblob.last_accessed_on (UTC datetime)
contentTypeblob.content_settings.content_type
contentEncodingblob.content_settings.content_encoding
contentLanguageblob.content_settings.content_language
contentDispositionblob.content_settings.content_disposition
cacheControlblob.content_settings.cache_control
contentMd5blob.content_settings.content_md5
etagblob.etag
blobTypeblob.blob_type.value (e.g. BlockBlob)
blobTierblob.blob_tier.value (e.g. Hot)
archiveStatusblob.archive_status
serverEncryptedblob.server_encrypted
deletedblob.deleted
snapshotIdblob.snapshot
versionIdblob.version_id
tagCountblob.tag_count

Schema-level quality checks (schema.quality) on the rowCount metric are evaluated as file-count thresholds against the total number of blobs found under the prefix.

contentType is normalised before comparison: MIME parameters are stripped, so application/json; charset=utf-8 matches application/json.

Supported location URL formats (on the server block):

  • https://<account>.blob.core.windows.net/<container>/<prefix>
  • abfss://<container>@<account>.dfs.core.windows.net/<prefix>
  • azure://<container>@<account>.blob.core.windows.net/<prefix>
  • wasbs://<container>@<account>.blob.core.windows.net/<prefix>

The metadata-check path also accepts DATACONTRACT_AZURE_CONNECTION_STRING or DATACONTRACT_AZURE_STORAGE_ACCOUNT_KEY instead of the service principal variables.