Synchronize data from external sources ​

Prerequisite: Create a partitioned table ​

By using the new keyword PARTITIONED BY, you will be able to synchronize your external database with Indexima at minimum cost.

You can create a partitioned external table like this :

text
CREATE EXTERNAL TABLE nyc_yellow FROM my_source_table IN 'jdbc:...'() PARTITIONED BY (year, month);

Synchronize an external table ​

By using the SYNCHRONIZE keyword on the desired partition, Indexima will apply all retrieve and all updates from the external source.

You can use it like this :

text
SYNCHRONIZE nyc_yellow PARTITION (year = 2016, month = 11);

Automatically synchronize external tables ​

INFO

Only available on Snowflake

Instead of running the SYNCHRONIZE command, Indexima will periodically ask Snowflake what are the last changes. By enabling this feature, Indexima will always be up-to-date with your external database.

For this purpose, you have 3 parameters to play with :

  • external.synchronize.check.rate : Number of seconds between external table synchronization check.
  • external.synchronize.check.cron : Cron expression for table synchronization check, more precise than external.synchronize.check.rate.
  • external.synchronize.check.user : The user who will run the SYNCHRONIZE during the automatic update.