# Spark DataFrame Reference

> Data type mappings for in-memory Spark DataFrames.

#  Spark DataFrame Reference

Data type handling for [Spark DataFrame connections](../connect/dataframe.md). No environment variables are needed — the existing Spark session is passed programmatically.

## Data types

### Importing

`datacontract import spark --tables <name>` maps Spark types as follows; `physicalType` is the Spark simple string (e.g. `bigint`, `decimal(10,2)`, `array<int>`).

| Spark type | `logicalType` |
|---|---|
| `StringType`, `VarcharType` | `string` |
| `IntegerType`, `ShortType`, `LongType` | `integer` |
| `FloatType`, `DoubleType`, `DecimalType` | `number` |
| `BooleanType` | `boolean` |
| `DateType` | `date` |
| `TimestampType`, `TimestampNTZType` | `date` |
| `StructType` | `object` with nested `properties` |
| `ArrayType` | `array` with typed `items` |
| `MapType`, `VariantType` | `object` |
| `BinaryType` | `array` |

### Testing

DataFrames are registered as temporary views named after the contract's schema objects. There is no catalog to introspect, so the `logicalType` is checked by category (`Check that field x has type y`) against the DataFrame's Spark types, with `integer` and `number` treated as compatible.

{/* AUTOGENERATED TYPE MAPPING: do not edit by hand; regenerate with update_reference_types.py */}

### Logical type mapping

When no `physicalType` is declared, the CLI derives the native type from the `logicalType` — for example in `datacontract export sql` and the dbt exports. This table is generated from the converter in the CLI's code:

| `logicalType` | Spark type |
|---|---|
| `string` | `STRING` |
| `integer` | `INT` |
| `number` | `DECIMAL(38,0)` |
| `boolean` | `BOOLEAN` |
| `date` | `DATE` |
| `timestamp` | `TIMESTAMP` |
| `time` | `STRING` |
| `object` | `STRUCT<>` |
| `array` | `ARRAY<STRING>` |

{/* END AUTOGENERATED TYPE MAPPING */}