Skip to main content

Google Cloud Storage (GCS)

The Amazon S3 integration also works with files on Google Cloud Storage through its interoperability. Use https://storage.googleapis.com as the endpoint URL and the s3:// scheme for the location.

1. Install​

uv tool install --python python3.11 --upgrade 'datacontract-cli[gcs]'

See Installation for pip, pipx, and Docker.

2. Set credentials​

Create an HMAC key for your user or service account, then create a .env file in your working directory (or export the variables):

# .env
DATACONTRACT_S3_ACCESS_KEY_ID=GOOG1EZZZXXXXXXXXXXXXX
DATACONTRACT_S3_SECRET_ACCESS_KEY=PDWWpbXXXXXXXXXXXXX

3. Create a contract from your files​

Download one object and import its schema, then point the generated servers block at the bucket:

gcloud storage cp gs://my-bucket/orders/orders-2024-01.json .
datacontract import json --source orders-2024-01.json --output datacontract.yaml

The import generates a servers entry of type: local. Replace it with your GCS location:

servers:
  - server: production
    type: s3
    endpointUrl: https://storage.googleapis.com
    location: s3://my-bucket/orders/*.json # use s3:// instead of gs://
    format: json
    delimiter: new_line # new_line, array, or none

4. Test the actual data​

datacontract test datacontract.yaml
🟢 data contract is valid. Run 17 checks. Took 3.9 seconds.

5. Let it catch a violation​

The contract becomes valuable when it detects drift. Tighten an expectation — for example, mark a field as required: true that occasionally arrives empty, or add a quality rule:

schema:
  - name: orders
    # ...
    quality:
      - type: sql
        description: No order has a negative total
        query: SELECT COUNT(*) FROM orders WHERE order_total < 0
        mustBe: 0

Run datacontract test datacontract.yaml again: every violation is listed as an error, and the command exits with code 1 — ready for CI/CD scheduling so you catch drift before your consumers do.

Reference​

All authentication options and the data type handling per file format: GCS Reference.

Troubleshooting​

  • 403 Forbidden — the HMAC key's principal lacks storage.objects.get/storage.objects.list on the bucket, or the key was deactivated.
  • No files found that match the pattern — the location must use the s3:// scheme (not gs://), and endpointUrl must be https://storage.googleapis.com.