Skip to main content

Amazon S3

Go from files in a bucket to a tested data contract in about five minutes. Works with S3 and any S3-compatible endpoint (e.g. MinIO), for data in CSV, JSON, Delta, or Parquet format.

1. Install​

uv tool install --python python3.11 --upgrade 'datacontract-cli[s3]'

See Installation for pip, pipx, and Docker.

2. Authenticate​

The easiest way is to sign in to AWS once — the CLI picks the session up, so no key is stored anywhere:

aws sso login   # or any other way of getting an AWS session

Prefer static keys? Set them directly and they are used instead:

# .env
DATACONTRACT_S3_REGION=eu-central-1
DATACONTRACT_S3_ACCESS_KEY_ID=AKIAXV5Q5QABCDEFGH
DATACONTRACT_S3_SECRET_ACCESS_KEY=93S7LRrJcqLaaaa/XXXXXXXXXXXXX

Public buckets need neither: with nothing configured the objects are read unsigned.

3. Create a contract from your files​

Import the schema straight from the bucket. This also generates a ready-to-test servers block:

datacontract import s3 \
  --source s3://my-bucket/orders/*.json \
  --output datacontract.yaml

The format is taken from the file suffix; pass --format for Delta tables, which have none. Add --endpoint-url for an S3-compatible store such as MinIO.

4. Test the actual data​

datacontract test datacontract.yaml
Testing datacontract.yaml
Server: production (type=s3, format=json, location=s3://my-bucket/orders/*.json)
╭────────┬─────────────────────────────────────────────────┬─────────────────┬─────────╮
│ Result │ Check                                           │ Field           │ Details │
├────────┼─────────────────────────────────────────────────┼─────────────────┼─────────┤
│ passed │ Check that field 'order_id' is present          │ orders.order_id │         │
│ passed │ Check that field order_id has no missing values │ orders.order_id │         │
│  ...   │                                                 │                 │         │
╰────────┴─────────────────────────────────────────────────┴─────────────────┴─────────╯
🟢 data contract is valid. Run 17 checks. Took 3.6 seconds.

5. Let it catch a violation​

The contract becomes valuable when it detects drift. Tighten an expectation — for example, mark a field as required: true that occasionally arrives empty, or add a quality rule:

schema:
  - name: orders
    # ...
    quality:
      - type: sql
        description: No order has a negative total
        query: SELECT COUNT(*) FROM orders WHERE order_total < 0
        mustBe: 0

Run datacontract test datacontract.yaml again: every violation is listed as an error, and the command exits with code 1 — ready for CI/CD and scheduled runs so you catch drift before your consumers do.

Reference​

All authentication options and the data type handling per file format: S3 Reference.

Troubleshooting​

  • 403 Forbidden / Access Denied — the key pair lacks s3:GetObject/s3:ListBucket on the location, or DATACONTRACT_S3_REGION doesn't match the bucket's region.
  • No files found that match the pattern — check the location glob; it matches object keys, not directories.
  • MinIO / S3-compatible storage fails to connect — set endpointUrl in the servers block; the CLI then switches to path-style addressing.