Data Infrastructure as Code

There are two important paradigms that significantly shaped the Data Infrastructure landscape:

• Infrastructure as Code
• Workflow as Code

Combining the two helps implement a reliable data engineering lifecycle with:

• reliable infrastructure management
• data pipelines built as code
• version control of both, data workflows and the underlying infrastructure.

This repository includes examples helping you adopt both of these paradigms at the same time.

Install Kestra

You can start Kestra using Docker-Compose:

curl -o docker-compose.yml https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml

docker-compose up

Before starting Kestra, make sure to create the .env file as shown in the .envexample file. Add any secrets there as environment variables so that you can use them in your workflows in a secure way.

Install Terraform

You can install Terraform on your local machine using Homebrew (for detailed instructions of your OS, check the Terraform CLI install guide Terraform):

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Write in code, deploy in one command

Navigate to the relevant project e.g. awss3tf and initialize the relevant Terraform providers:

cd aws_s3/
terraform init

Then, deploy the workflow and the underlying infrastructure using the command:

terraform apply -auto-approve