Skip to main content

Spark DataFrame

Test Spark DataFrames in a pipeline before writing them to a data source. DataFrames are registered as named temporary views; multiple views are supported if the contract has multiple schemas. This connection is used programmatically from Python — no credentials are needed, the existing Spark session is reused.

1. Install​

No extra is required (the Spark session comes from your environment):

uv tool install --python python3.11 --upgrade datacontract-cli

In a notebook or pipeline environment, install datacontract-cli as a library instead. See Installation for pip, pipx, and Docker.

2. Create a contract from your DataFrames​

Inside an active Spark session, import the schema of registered tables or views:

datacontract import spark --tables my_table --output datacontract.yaml

The generated contract includes a servers entry of type dataframe:

servers:
  - server: production
    type: dataframe

3. Test the DataFrame​

Register the DataFrame as a temporary view named like the schema, then run the test with the Spark session:

from datacontract.data_contract import DataContract

df.createOrReplaceTempView("my_table")

data_contract = DataContract(
  data_contract_file="datacontract.yaml",
  spark=spark,
)
run = data_contract.test()
assert run.result == "passed"

4. Let it catch a violation​

The contract becomes valuable when it detects drift. Tighten an expectation — for example, mark a field as required: true or add a quality rule — and run the test again: run.result becomes "failed" and run.checks lists each violation, so the assert stops your pipeline before bad data is written.

Reference​

The Spark data type mappings: Spark DataFrame Reference.