Core Concepts

Default Name for Schema (Database Destination)
Users can change the Target DB schema name as per their needs. The schema will be created if does not exist.
For any Database source (e.g. MySQL) Source Schema name will be used as Target DB schema name
For any FlatFile source (e.g. SFTP) Folder name will be used as Target DB schema name
For any API source (e.g. Facebook) Source name will be used as Target DB schema name
Default Name for Table (Database Destination)
Users can change the Target DB table name as per their needs.
For any Database source (e.g. MySQL) Source Table name will be used as Target DB table name
For any API source (e.g. Facebook) Source Table/API Endpoint/Resource will be used as Target DB table name
Note: For report type of source(e.g. SearchAds360) or flat file source the Target DB table name will be blank. Users need to specify the Target DB table name manually.
Default Folder Path (Flat File Destination)
Users can change the Folder Path as per their needs. The child folders will be created if they do not exist.
For any Database source (e.g. MySQL) Source Schema name will be used as Folder Path
For any API source (e.g. Facebook) Source name will be used as Target Folder Path
Note: User can specify Date partition.
Default File Name (Flat File Destination)
For any Database source (e.g. MySQL) Source Table name will be used as File name
For any API source (e.g. Facebook) Source Table/API Endpoint/Resource will be used as File name
Note: For report type of source(e.g. SearchAds360) or flat file source the File name will be blank. Users need to specify the File name manually.
Write Data into Database Destination
From the temporary data storage, Gravity copies the file into the staging table.
We then merge the data from the staging tables with the existing data present in the target table.
Finally, we delete the staging table.
Backfill or Historical Data
Terminology
Start Date for data content - It means from the date you want to pull the data. If you want to pull the data from 1st January 2021 you need to set that date as the Start date.
Look Ahead Days - It means in one job run how many days of data you want to fetch. 30 means it will fetch data of 30 days. Gravity fetches 30 days of data in one run for most of the sources.
How it works
e.g. Start Date for data content - 01/01/2021, Look Ahead Days - 30, you want to pull the Data till 01/01/2022
For the above example, when you run the first time it will pull the data of 30 days and will set the Start Date for data content to 1st Feb, on the 2nd run it will fetch data from 1st Feb and to next 30 days and set Start Date for data content to 2nd March and same goes on and on.
For the above example, you need to run the job 13 times to get the data till 01/01/2022.
Gravity doesn't prefer to set higher Look ahead days as we might see rate limits error. Instead, we suggest you set schedule intervals frequently (e.g. hourly rather than daily) until the latest data is pulled. Once you reach the latest data you can set the schedule to daily if you want.

Orchestration - Previous

Allowing Gravity access to data behind a firewall

Next - Sources

Files Sources

Last modified 2mo ago

Copy link

Outline

Default Name for Schema (Database Destination)

Default Name for Table (Database Destination)

Default Folder Path (Flat File Destination)

Default File Name (Flat File Destination)

Write Data into Database Destination

Backfill or Historical Data