Files
Gravity can write data files to several destinations. The same principles apply to all file destination types.
Job Modes
The available job modes for file destinations are:
Delete and Replace - extracts the full source data on each run and replaces the target file
Append - extracts the source data incrementally and creates a new file with the data from each run
File Format
The destination file format can be set for each job. Gravity supports the following file formats:
CSV: Gravity generates semi-colon (;) delimited files. Values are quoted (") to handle special characters
JSON: Gravity generates newline delimited JSON, which is the same format as the JSON Lines format.
Parquet: Gravity generates Parquet standard Parquet files. The block size, page size, group size and row count can be customised for each job if required.
Partitioning
File partitioning is available for Append jobs. Gravity supports partitioning files using the following formats.
gravity_id=<runid>
gravity_inserted_year=<yyyy>/gravity_inserted_month=<MM>/gravity_inserted_day=<dd>
gravity_inserted_year=<yyyy>/gravity_inserted_month=<MMMM>/gravity_inserted_year=<dd>
gravity_inserted=<yyyy>-<MM>-<dd>
gravity_inserted=<yyyy>-<MMMM>-<dd>
Files cannot be partitioned by a source column at this time.
The above partitions are not applicable for Delete and Replace job mode since the file is replace every run.
Compression
Destination files can be compressed. The compression type depends on the file format selected. The table below describes which compression types are available for each file format.
CSV
zip, gzip
JSON
zip, gzip
Parquet
snappy, gzip
Last updated