# Amazon S3

> Create a data contract from files in S3 and test them against it — in about 5 minutes.

#  Amazon S3

Go from files in a bucket to a tested data contract in about five minutes. Works with S3 and any S3-compatible endpoint (e.g. MinIO), for data in CSV, JSON, Delta, or Parquet format.

## 1. Install

```bash
uv tool install --python python3.11 --upgrade 'datacontract-cli[s3,duckdb]'
```

See [Installation](../installation.md) for pip, pipx, and Docker.

## 2. Set credentials

Create a `.env` file in your working directory (or export the variables):

```bash
# .env
DATACONTRACT_S3_REGION=eu-central-1
DATACONTRACT_S3_ACCESS_KEY_ID=AKIAXV5Q5QABCDEFGH
DATACONTRACT_S3_SECRET_ACCESS_KEY=93S7LRrJcqLaaaa/XXXXXXXXXXXXX
```

For public buckets, or when ambient AWS credentials are available, you can skip this step.

## 3. Create a contract from your files

Download one object and import its schema, then point the generated `servers` block at the bucket:

```bash
aws s3 cp s3://my-bucket/orders/orders-2024-01.json .
datacontract import json --source orders-2024-01.json --output datacontract.yaml
```

The import generates a `servers` entry of `type: local`. Replace it with your S3 location:

```yaml
servers:
  - server: production
    type: s3
    location: s3://my-bucket/orders/*.json
    format: json
    delimiter: new_line # new_line, array, or none
```

## 4. Test the actual data

```bash
datacontract test datacontract.yaml
```

```
Testing datacontract.yaml
Server: production (type=s3, format=json, location=s3://my-bucket/orders/*.json)
╭────────┬─────────────────────────────────────────────────┬─────────────────┬─────────╮
│ Result │ Check                                           │ Field           │ Details │
├────────┼─────────────────────────────────────────────────┼─────────────────┼─────────┤
│ passed │ Check that field 'order_id' is present          │ orders.order_id │         │
│ passed │ Check that field order_id has no missing values │ orders.order_id │         │
│  ...   │                                                 │                 │         │
╰────────┴─────────────────────────────────────────────────┴─────────────────┴─────────╯
🟢 data contract is valid. Run 17 checks. Took 3.6 seconds.
```

## 5. Let it catch a violation

The contract becomes valuable when it detects drift. Tighten an expectation — for example, mark a field as `required: true` that occasionally arrives empty, or add a quality rule:

```yaml
schema:
  - name: orders
    # ...
    quality:
      - type: sql
        description: No order has a negative total
        query: SELECT COUNT(*) FROM orders WHERE order_total < 0
        mustBe: 0
```

Run `datacontract test datacontract.yaml` again: every violation is listed as an error, and the command exits with code `1` — ready for [CI/CD scheduling](../ci-cd.md) so you catch drift before your consumers do.

## Server reference

### JSON

```yaml
servers:
  - server: production
    type: s3
    endpointUrl: https://minio.example.com # not needed with AWS S3
    location: s3://bucket-name/path/*/*.json
    format: json
    delimiter: new_line # new_line, array, or none
```

### Delta

```yaml
servers:
  - server: production
    type: s3
    endpointUrl: https://minio.example.com # not needed with AWS S3
    location: s3://bucket-name/path/table.delta # folder with parquet files and _delta_log
    format: delta
```

## Reference

All authentication options and the data type handling per file format: **[S3 Reference](../reference/s3.md)**.

## Troubleshooting

- **`403 Forbidden` / `Access Denied`** — the key pair lacks `s3:GetObject`/`s3:ListBucket` on the location, or `DATACONTRACT_S3_REGION` doesn't match the bucket's region.
- **`No files found that match the pattern`** — check the `location` glob; it matches object keys, not directories.
- **MinIO / S3-compatible storage fails to connect** — set `endpointUrl` in the `servers` block; the CLI then switches to path-style addressing.