Files

Gravity can write data files to several destinations. The same principles apply to all file destination types.

Job Modes

The available job modes for file destinations are:

  • Delete and Replace - extracts the full source data on each run and replaces the target file

  • Append - extracts the source data incrementally and creates a new file with the data from each run

File Format

The destination file format can be set for each job. Gravity supports the following file formats:

  • CSV: Gravity generates semi-colon (;) delimited files. Values are quoted (") to handle special characters

  • JSON: Gravity generates newline delimited JSON, which is the same format as the JSON Lines format.

  • Parquet: Gravity generates Parquet standard Parquet files. The block size, page size, group size and row count can be customised for each job if required.

Partitioning

File partitioning is available for Append jobs. Gravity supports partitioning files using the following formats.

  • gravity_id=<runid>

  • gravity_inserted_year=<yyyy>/gravity_inserted_month=<MM>/gravity_inserted_day=<dd>

  • gravity_inserted_year=<yyyy>/gravity_inserted_month=<MMMM>/gravity_inserted_year=<dd>

  • gravity_inserted=<yyyy>-<MM>-<dd>

  • gravity_inserted=<yyyy>-<MMMM>-<dd>

Files cannot be partitioned by a source column at this time.

The above partitions are not applicable for Delete and Replace job mode since the file is replace every run.

Compression

Destination files can be compressed. The compression type depends on the file format selected. The table below describes which compression types are available for each file format.

File FormatCompression Type

CSV

zip, gzip

JSON

zip, gzip

Parquet

snappy, gzip

PreviousAbout Google BigQueryNextAmazon S3

Last updated