Snowflake Limitations
Input Table Characteristics (Especially for Snowflake Dynamic Table Integration):
- Physical Tables: When S2V is used in conjunction with features like Snowflake Dynamic Tables for downstream processing (e.g., building a Standardized Zone from a Raw Zone), the input tables S2V reads from (e.g., Raw Zone tables) must be actual Snowflake tables. External tables or views may not be suitable as Dynamic Tables rely on change tracking capabilities of base tables. Please refer to the Snowflake documentation regarding limitations on source objects for Dynamic Tables
- Change Tracking: For optimal performance and to enable incremental processing by downstream Dynamic Tables, change tracking should be enabled on the underlying Snowflake objects
- Source Table History: These source tables should contain all relevant data which is required for downstream processing. Dynamic Tables in Snowflake act on the change tracking of underlying tables and if data get's deleted, it will also be taken out of the target Dynamic Table. Therfore we recommend setting up a Persistent Staging Area (PSA) that takes care of archiving required data. Having this history available also makes complicated migration scripts obsolete as the data can simply be reprocessed. Please visit Snowflake Documentation of Dynamic Tables for further details