Skip to main content

Deployment via Makefile

Data Vault Deployment Process​

This document describes the deployment process for the Data Vault objects, which is orchestrated by Makefiles and executed using SnowSQL, Snowflake's command-line client. Configuration, including connection parameters and script variables, is managed through an .ini file (e.g., conf.ini).

Deployment Flow Summary

  • Invocation: You run a target (e.g., make deploy_all) on the top-level Makefile.
  • Setup: The top-level Makefile configures the SNOWSQL command, specifying the .ini file and command-line variables like the release tag.
  • Navigation: It changes the current directory to $(OUTPUT_DIR), where the generated SQL scripts and the deployment-specific Makefile are located.
  • Delegation: It executes the same target (e.g., deploy_all) on the Makefile within $(OUTPUT_DIR).
  • Execution: The Makefile inside $(OUTPUT_DIR) (generated by the tool) then:
    • Runs initialization scripts (e.g., creating schemas, tags).
    • Deploys Data Vault objects (Hubs, Links, Satellites) in the correct sequence, often layer by layer (PREP_LAYER, MAIN_LAYER).
    • Each SQL execution uses the snowsql command. SnowSQL reads the .ini file, connects to Snowflake, and substitutes all &{variable_name} placeholders in the SQL scripts with values from the .ini file before running them.
  • Error Handling: If exit_on_error = True is active, any SQL error will stop the deployment for that target.

This deployment architecture offers a flexible and configurable method for managing your Data Vault. The clear separation of responsibilities—top-level Makefile for orchestration and environment setup, generated Makefile for detailed deployment steps, and .ini files for configuration—enhances manageability across various environments.

Partial Deployments

There's no special deployment process required for partial model deployments. The s2v generate command, with its --include-objects or --exclude-objects parameters, allows you to generate SQL scripts for only a subset of your Data Vault model. Once the partial model is generated, you can use the standard deployment targets like make deploy_all to deploy only the selected objects. The generated Makefile in the $(OUTPUT_DIR) will only contain the code for these included objects.

Core Components​

1. Top-Level Makefile​

The main Makefile (typically located at the root of the project) serves as the primary entry point for all deployment operations. Its main role is to delegate tasks to another Makefile that resides within the directory where the SQL scripts are generated by the code generation tool.

Key Features:

  • OUTPUT_DIR Variable: This variable specifies the path to the generated code directory. This directory contains all the SQL scripts and a dedicated Makefile for their deployment.
  • Delegation of Tasks: When a target like make deploy_all is executed from this top-level Makefile, it doesn't directly run the SQL files. Instead, it navigates to the $(OUTPUT_DIR) and executes the make deploy_all command on the Makefile found within that directory. This pattern is consistent for other targets such as deploy_dynamic_tables, deploy_tables, and refresh_dynamic_tables.
  • SNOWSQL Command Configuration: The Makefile defines how snowsql commands are constructed. This includes specifying the configuration file (derived from an $(ENV) variable, like dev for conf.ini) and passing variables such as a release identifier ($(TAG)) directly to SnowSQL. This tag is useful for versioning the deployed Data Vault objects.

Minimal Makefile example:

# Note: Replace <MODEL_DIRECTORY> and <GENERATED_CODE_DIRECTORY> with your actual project paths.

# Always execute targets in parallel, defaults to 20 parellel jobs
MAKEFLAGS += --jobs 20 --no-builtin-rules --no-keep-going

MODEL_DIR	?= <MODEL_DIRECTORY>
OUTPUT_DIR ?= $(MODEL_DIR)/<GENERATED_CODE_DIRECTORY>
CURRENT_DIR := $(shell pwd)

# ╔════════════════════════════════════════════════════════════════════════════╗
# ║                                    S2V                                     ║
# ╚════════════════════════════════════════════════════════════════════════════╝

TAG := $(shell git describe --tags --dirty --always)

# ╔════════════════════════════════════════════════════════════════════════════╗
# ║                  Deployment with SnowSQL and Azure CLI example             ║
# ║               - requires privileges to access Key Vault                    ║
# ╚════════════════════════════════════════════════════════════════════════════╝

# SNOWSQL example
export SNOWSQL := snowsql --config $(PWD)/conf/conf.ini --variable S2V_RELEASE_ID_DYNAMIC_TABLE=$(TAG) --variable S2V_RELEASE_ID_TABLE=$(TAG)

# ╔════════════════════════════════════════════════════════════════════════════╗
# ║           Default Makefile targets                                         ║
# ╚════════════════════════════════════════════════════════════════════════════╝

.PHONY: deploy_all
deploy_all:
	$(MAKE) --directory $(OUTPUT_DIR) deploy_all

.PHONY: deploy_dynamic_tables
deploy_dynamic_tables:
	$(MAKE) --directory $(OUTPUT_DIR) deploy_dynamic_tables

.PHONY: deploy_tables
deploy_tables:
	$(MAKE) --directory $(OUTPUT_DIR) deploy_tables

.PHONY: refresh_dynamic_tables
refresh_dynamic_tables:
	$(MAKE) --directory $(OUTPUT_DIR) refresh_dynamic_tables

tip

Leverage your top-level Makefile to streamline your S2V workflow:

  • Environment Management: Easily switch between deployment environments (dev, test, prod) by changing the ENV variable, which can point to different conf/<ENV>.ini files.
  • S2V Command Orchestration: Add targets to run S2V commands like s2v validate -i $(MODEL_DIR) or s2v generate -i $(MODEL_DIR) -o $(OUTPUT_DIR) before deployment.
  • Targeted Deployments: Create custom Make targets for deploying specific subsets of your Data Vault model (e.g., only Hubs, or objects related to a particular source system) by invoking specific targets in the $(OUTPUT_DIR)/Makefile.
  • Pre/Post Deployment Hooks: Add custom shell commands or scripts to execute before or after deployment steps for tasks like information schema refresh or warehouse scaling.

2. The .ini Configuration File (e.g., dev.ini)​

This file is essential for SnowSQL, providing connection details, operational options, and variables for substitution within the SQL scripts.

Example conf.ini Content:

[connections]
accountname = <YOUR_SNOWFLAKE_ACCOUNT>
...

[options]
exit_on_error = <TRUE/FALSE>
...

[variables]
TARGET_DATABASE=<YOUR_TARGET_DATABASE>
TARGET_WAREHOUSE=<YOUR_TARGET_WAREHOUSE>
...

info

Running Make on Windows may require additional configuration. See Run Make on Windows.