Skip to main content

Architecture

Overview​

The following diagram illustrates the architecture of the Stream2Vault (S2V) platform, showcasing its integration with enterprise services and its interaction with users. Here’s a breakdown of the key components and their relationships:

Stream2Vault Architecture Diagram

Stream2Vault Service​

This is the core backend service of the Stream2Vault platform, responsible for validating user-defined data-vault models and generating the SQL scripts and deployment artifacts for Data Vault automation.

  • It communicates directly with the S2V Client for validating and generating outputs.
  • It integrates with the Authentication Service to validate user credentials and permissions.

Organizational Enterprise Services​

Authentication Service​

Handles user authentication for both the S2V Client and the S2V Service. Ensures secure access to enterprise resources, such as Git repositories and databases.

Git Service​

Manages code generated by the S2V Service, enabling version control and collaboration. Acts as a bridge between the generated code and the organization's CI/CD pipelines for automated deployment.

Database Service​

Receives and stores the deployed Data Vault structures. Works as the target system for Data Vault objects generated and validated by the Stream2Vault platform. Deployments are managed through the CI/CD pipeline, ensuring automated, seamless integration into the database environment.

Stream2Vault Client​

The S2V Client is the command-line interface used by data modelers, data engineers, and other stakeholders. It enables data modeling, where users design or modify the metadata and Data Vault objects. It interacts with the S2V Service to validate and generate configurations for deployment. Authenticates with the Authentication Service to ensure secure operations within the organizational environment.

Workflow​

The typical workflow involving Stream2Vault includes the following stages:

Authentication​

The user authenticates via the S2V Client, which in turn interacts with the enterprise Authentication Service. The S2V Service also verifies its connection with the Authentication Service.

Validation and Generation​

The user defines their Data Vault model using YAML files and provides configuration files (e.g., data_vault_settings.yaml). These inputs are passed via the S2V Client to the S2V Service, which validates the model and generates SQL deployable scripts and other artifacts (like Makefiles).

Code Management​

The generated SQL scripts and deployment artifacts are typically committed to a Git repository, enabling version control and integration into enterprise CI/CD pipelines.

Deployment​

The CI/CD pipeline consumes the artifacts from Git (e.g., using the generated Makefile) to handle the deployment of the Data Vault structures to the target Database Service, ensuring efficient and automated deployment.

Summary​

This architecture ensures seamless collaboration between the Stream2Vault platform and organizational enterprise services. It integrates secure authentication, robust code management, and automated deployments, providing a streamlined and model-driven experience for data modelers and technical teams.