Files
Gravity can write data files to several destinations. The same principles apply to all file destination types.
The destination file format can be set for each job. Gravity supports the following file formats:
- CSV: Gravity generates semi-colon (;) delimited files. Values are quoted (") to handle special characters
- Parquet: Gravity generates Parquet standard Parquet files. The block size, page size, group size and row count can be customised for each job if required.
File partitioning is available for Append jobs. Gravity supports partitioning files using the following formats.
- gravity_id=<runid>
- gravity_inserted_year=<yyyy>/gravity_inserted_month=<MM>/gravity_inserted_day=<dd>
- gravity_inserted_year=<yyyy>/gravity_inserted_month=<MMMM>/gravity_inserted_year=<dd>
- gravity_inserted=<yyyy>-<MM>-<dd>
- gravity_inserted=<yyyy>-<MMMM>-<dd>
Files cannot be partitioned by a source column at this time.
The above partitions are not applicable for Delete and Replace job mode since the file is replace every run.
Destination files can be compressed. The compression type depends on the file format selected. The table below describes which compression types are available for each file format.
File Format | Compression Type |
---|---|
CSV | zip, gzip |
JSON | zip, gzip |
Parquet | snappy, gzip |
Last modified 1yr ago