Spark DataFrame Reference

Data type handling for Spark DataFrame connections. No environment variables are needed — the existing Spark session is passed programmatically.

Data types

Importing

datacontract import spark --tables <name> maps Spark types as follows; physicalType is the Spark simple string (e.g. bigint, decimal(10,2), array<int>).

Spark type	logicalType
StringType, VarcharType	string
IntegerType, ShortType, LongType	integer
FloatType, DoubleType, DecimalType	number
BooleanType	boolean
DateType	date
TimestampType, TimestampNTZType	date
StructType	object with nested properties
ArrayType	array with typed items
MapType, VariantType	object
BinaryType	array

Testing

DataFrames are registered as temporary views named after the contract's schema objects. There is no catalog to introspect, so the logicalType is checked by category (Check that field x has type y) against the DataFrame's Spark types, with integer and number treated as compatible.

Logical type mapping

When no physicalType is declared, the CLI derives the native type from the logicalType — for example in datacontract export sql and the dbt exports. This table is generated from the converter in the CLI's code:

logicalType	Spark type
string	STRING
integer	INT
number	DECIMAL(38,0)
boolean	BOOLEAN
date	DATE
timestamp	TIMESTAMP
time	STRING
object	STRUCT<>
array	ARRAY<STRING>
map	MAP<STRING,STRING>
vector	ARRAY<FLOAT>