Skip to main content

Define your Quality Rules

Beyond the schema checks (presence, type, required, unique, primary keys, value ranges), a data contract can declare quality rules. When you run datacontract test, executable rules run against the data source and are reported alongside the schema checks.

Quality rules use the ODCS quality attribute, which can be attached either to a schema (table/object level) or to an individual property (column/field level):

schema:
  - name: orders
    quality:                       # schema-level rule
      - type: library
        metric: rowCount
        mustBeGreaterThan: 0
    properties:
      - name: order_total
        quality:                   # property-level rule
          - type: library
            metric: nullValues
            mustBe: 0

Quality rule types​

ODCS defines four types of quality rule. Each has its own page:

SQLRun a custom SQL query and compare the result to a threshold.LibraryPredefined, portable checks such as rowCount, nullValues, and duplicateValues.TextA human-readable expectation, for documentation only.CustomEngine-specific checks (e.g. DQX, SodaCL, Great Expectations).

Quality dimensions​

Any rule can be classified with a dimension — the aspect of data quality it protects. It is optional and works with every rule type. It documents what the contract's quality coverage adds up to, and datacontract test --dimension runs one aspect at a time.

schema:
  - name: orders
    quality:
      - type: library
        metric: rowCount
        mustBeGreaterThan: 0
        dimension: completeness
    properties:
      - name: order_id
        quality:
          - type: library
            metric: duplicateValues
            mustBe: 0
            dimension: uniqueness

ODCS defines seven dimensions. datacontract lint rejects any other value:

DimensionThe data…
accuracycorrectly describes the real-world entity or event it represents
completenesscontains all the records and values that are expected
conformityfollows the agreed format, type, and set of allowed values
consistencyagrees with itself and with related data elsewhere
coverageincludes the full population it claims to describe
timelinessis available and current when it is needed
uniquenesscontains no unintended duplicates

The HTML export renders the dimension as a badge next to schema-level rules.

Identifying rules​

Any rule can carry an id and a list of tags. Both are optional and work with every rule type. They name a rule so that a pipeline can run it on its own: an id addresses exactly one rule, tags group rules that belong together — by cost, criticality, or the pipeline step they guard.

schema:
  - name: orders
    quality:
      - id: orders_not_empty
        type: library
        metric: rowCount
        mustBeGreaterThan: 0
        tags: ["critical", "cheap"]
    properties:
      - name: order_id
        quality:
          - id: order_id_is_unique
            type: library
            metric: duplicateValues
            mustBe: 0
            tags: ["critical", "expensive"]

An id must be unique within the contract, so that --quality-id selects a single rule. Test results report the id and tags of the rule each check comes from.

Running only quality checks​

Use --checks to restrict a run to quality rules:

datacontract test --checks quality datacontract.yaml

Use --dimension to run one aspect of quality across the whole contract:

datacontract test --dimension uniqueness datacontract.yaml
datacontract test --dimension completeness,accuracy datacontract.yaml

This selects quality rules by their declared dimension and the built-in checks that measure the same thing — --dimension uniqueness runs your duplicateValues rules alongside the unique and primary-key checks derived from the schema.

Quality rules that declare no dimension are never selected: only the author can say what a custom rule measures. If nothing matches, the run executes nothing and reports no checks.

Use --quality-id to run a single rule, and --tag to run a group of them:

datacontract test --quality-id order_id_is_unique datacontract.yaml
datacontract test --tag critical datacontract.yaml
datacontract test --tag critical,cheap datacontract.yaml

Both select quality rules only — no schema or service level checks run alongside them. This makes it possible to spread the rules of one contract over a data pipeline: run the cheap rules on ingest, the expensive ones after the nightly load, without splitting the contract into several files.

--quality-id matches the id of the rule, and fails the run if no rule in the contract declares it — a mistyped id must not pass by testing nothing. --tag matches the tags of the rule (not the tags of a schema or property), and reports no checks when nothing matches. Options combine: --tag critical --checks quality --server production runs the critical rules on production.

Dimensions of the built-in checks​

Schema checks and service levels have no dimension field to set, so the CLI assigns each the dimension it measures:

DimensionBuilt-in checks
completenessrequired fields, primary key not-null
uniquenessunique fields, primary key uniqueness (including composite keys)
conformityfield presence, logical and physical types, nested types, pattern, enum, length and value bounds, JSON Schema validation, slaProperties retention
timelinessslaProperties freshness

Where quality rules are used​

Executable rules (sql, library) run during test. All rule types are also translated when you export to a quality engine:

  • dbt — rules become native dbt tests, with dbt_utils tests and singular SQL tests for rules that cannot be expressed natively.
  • SodaCL — rules become Soda checks.
  • Great Expectations — rules become expectations in a suite.
  • DQX — rules become Databricks DQX checks.

Inspecting failing rows​

When a quality (or schema) check fails, add --include-failed-samples to test to collect a small sample of the offending rows (identifier + offending columns; sensitive columns are omitted). This is off by default.

datacontract test --include-failed-samples datacontract.yaml

Learn more​