Exporting to Snowflake

Owned by Juliane Wetzel
Apr 30, 2020

Export data to Snowflake through S3.

All records are exported as Avro files to a temporary (and configurable) location on S3. The Avro files are then added to Snowflake using a single Snowflake COPY INTO <table> command allowing for high performance on the Snowflake side. (See https://docs.snowflake.net/manuals/sql-reference/sql/copy-into-table.html). The temporary files and the Snowflake STAGE needed for the import to S3 are deleted once the export job has completed.

Configuring Snowflake as a Connection

A Snowflake connection with Datameer X must be created before exporting data.

Learn about setting up a connection between Datameer X and Snowflake.

Configuring Export Jobs with an Snowflake Connection

To export a worksheet using the Snowflake connector:

  1. Select the + (plus) at the top left of the File Browser and select Export Job or right-click on the workbook to export from and select Add to New Export Job.

  2. Select the workbook to export from in the browser.

  3. Select the worksheet to export from the drop down menu.

  4. Select the Snowflake connector.
  5. The Snowflake database name is that set in the connector. If not previously set from the connector, enter the Snowflake Warehouse name.
    To export to an existing table on Snowflake, select the radio button Use existing. Choose a schema and table name from the drop-down lists. 
    To export to a new table or replace an existing table on Snowflake, select the radio button Create new or replace existing table. Choose a schema from the drop-down list and enter a new table name.
     
    Select a record replacement policy. Learn more about database record replacement.  
    • Replace records - All records from the existing table are deleted and the new records are written to the table. No table properties are modified.
    • Append records - The records append (added in addition) to the existing table.
  6. Preview how the data to is being mapped to the Snowflake table and select how to handle invalid data.
  7. Select when to run the export job. This can be done manually by a user, run each time the workbook in Datameer X is calculated, or at specifically scheduled times.
    Custom properties can be added in the advanced section as well as logging specifications. 
  8. Enter a description and click Save.